JP6859763B2 - Program, information processing device - Google Patents

Description

The present invention relates to a program and an information processing device.

Spherical cameras that can capture 360-degree landscapes are widespread. The image data (omnidirectional image) captured by the spherical camera has a three-dimensional shape attached to the sphere, and a method for effectively utilizing the three-dimensional spherical image is being studied. For example, in order to view a spherical image, it is necessary to project it on a two-dimensional plane, but distortion is likely to occur depending on the display range, and a technique for reducing this distortion has been devised (see, for example, Patent Document 1). .).

In Patent Document 1, an omnidirectional image is pasted on a three-dimensional model, and an image process for switching between preferentially changing the viewing angle and preferentially changing the viewpoint position according to a specified output range. The system is disclosed.

However, in the conventional technique, even if the spherical image has a three-dimensional shape, there is a problem that it is not easy to utilize it in computer graphics. In the development of application software for displaying three-dimensional objects on an information processing device, it is common to use a drawing library in which frequently used functions such as rotation, movement, and scaling are open to the public. Developers can use these drawing libraries to efficiently develop application software (ie, programs).

However, since the conventional drawing library cannot be directly linked with the spherical image, it requires man-hours for the developer to develop the application software in which the three-dimensional object and the spherical image are linked. For example, when creating application software that hits a ball (object) against a wall when the spherical image shows a wall, the developer separates the spherical image, the wall, and the object corresponding to the ball. It is necessary to generate in and describe these interactions.

In view of the above problems, an object of the present invention is to provide a program in which a three-dimensional object and a wide-angle image are linked.

The present invention uses an information processing device that acquires at least one of a wide-angle image and distance measurement data in the ceiling direction, the ground direction, or the surrounding direction to generate a screen object to which the wide-angle image is attached, and in the ceiling direction. An imaging object generating means that generates at least one of a first object based on distance measurement data, a second object based on the distance measurement data in the ground direction, or a third object based on the distance measurement data in the peripheral direction. When,
A camera object generating means for generating a camera object as a viewpoint of the wide-angle image, a user object generating means for generating a user-defined three-dimensional object, the screen object, the first object, and the second object. The third object, the three-dimensional object, and the wide-angle image attached to the screen object are made to function as drawing means to be sent to the drawing processing unit, and the imaging object generating means has a physical action of each object. Provides a program characterized by updating each object based on the attributes related to.

It is possible to provide a program in which a three-dimensional object and a wide-angle image are linked.

This is an example of an external view of an image pickup device that captures a spherical image. This is an example of the hardware configuration diagram of the image pickup device. This is an example of a schematic perspective view of an information processing device. This is an example of a hardware configuration diagram of an information processing device. This is an example of a functional block diagram showing the functions of the image pickup device and the information processing device in a block shape. It is a figure which shows an example of the object generated by the spherical image 3D library. It is a figure which shows an example of the screen object which is not a sphere. It is a figure which shows the side object schematically. This is an example of a diagram for explaining the drawing of a ball based on the drawing priority. This is an example of a sequence diagram for explaining the operation procedure of the application software created by the spherical image three-dimensional library. This is an example of a sequence diagram for explaining the operation procedure of the application software created by the spherical image three-dimensional library. It is a figure which shows an example of the object generated by the spherical image three-dimensional library (Example 2). This is an example of a sequence diagram for explaining the operation procedure of the application software created by the spherical image three-dimensional library (Example 2). This is an example of a sequence diagram for explaining the operation procedure of the application software created by the spherical image three-dimensional library (Example 3). This is an example of the hardware configuration diagram of the image pickup apparatus (Example 4). This is an example of a functional block diagram showing the functions of the image pickup device and the information processing device in a block shape (Example 4). This is an example of a sequence diagram for explaining the operation procedure of the application software created by the spherical image three-dimensional library (Example 4). This is an example of the hardware configuration diagram of the image pickup apparatus (Example 5). This is an example of a functional block diagram showing the functions of the image pickup apparatus and the information processing apparatus in a block shape (Example 5). This is an example of a sequence diagram for explaining the operation procedure of the application software created by the spherical image three-dimensional library (Example 5). This is an example of a diagram for explaining an imaging scene. This is an example of an external view of an imaging device that captures spherical images (Example 6). It is a figure which shows an example of the object generated by the spherical image 3D library.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

In the present embodiment, height information from the viewpoint position to the "ceiling" and "ground" and distance information from the viewpoint position to the surrounding direction are added to the spherical image in advance. The application software of the present embodiment projects an all-sky image onto a virtual screen object in a three-dimensional space, while the ceiling object and the ground object are based on the height information of the ceiling and the ground and the distance information to the surroundings, respectively. Place objects such as side objects and side objects in 3D space, and calculate the physical action with any 3D object.

Since object creation and physical actions are supplied by the library, it is easy to develop application software that links 3D objects and spherical images.

<Terminology>
Coordination means that one object acts on the other object and the attributes of at least one object change. For example, it means colliding, and the state of motion changes due to the collision.

An attribute is a feature or property of an object. It can also be said that it is a set value that affects the display mode and behavior of the object. Specific examples will be described in this embodiment.

<Configuration example>
FIG. 1 is an example of an external view of an image pickup device 100 that captures a spherical image. The housing 101 has two back-to-back imaging units 102. The two imaging units 102 are joined together to obtain image data of a spherical image (hereinafter, referred to as a spherical image) in which a circumference of 360 degrees is captured.

An operation unit 103 is provided in the central portion of the housing 101, and the operation unit 103 is a shutter button pressed when a user performs an imaging operation. Further, the upper surface of the image pickup apparatus 100 has a ceiling distance measuring sensor 104 for detecting the distance in the ceiling direction, and the lower surface of the imaging device 100 has a ground direction distance measuring sensor 105 for detecting the distance in the ground direction. There is. Further, a peripheral direction distance measuring sensor 106 for detecting a distance in the peripheral direction is arranged around the housing 101. A plurality of peripheral direction distance measuring sensors 106 are arranged in the periphery, and the larger the number, the higher the resolution of detecting the distance in the peripheral direction.

The distance measurement method may be any method, and examples thereof include a TOF (Time Of Flight) of ultrasonic waves and light, a stereo camera, and a monocular camera capable of detecting a distance. The top direction distance measurement sensor 104, the ground direction distance measurement sensor 105, and the peripheral direction distance measurement sensor 106 may be externally attached.

The top direction distance measurement sensor 104 and the ground direction distance measurement sensor 105 detect the distance between the housing 101 and the ceiling direction and the distance between the housing 101 and the ground direction when the operation unit 103 is pressed, respectively. The peripheral direction distance measuring sensor 106 detects the distance between the housing 101 and the object in the peripheral direction when the operation unit 103 is pressed. The peripheral direction distance measuring sensor 106 may detect the distance to the object in the peripheral direction when the user rotates with the housing 101 while the operation unit 103 is pressed, for example. As a result, the number of peripheral direction distance measuring sensors 106 can be reduced. Further, the top direction distance measurement sensor 104 and the ground direction distance measurement sensor 105 may be used in combination with one sensor.

<Hardware configuration example>
FIG. 2 shows an example of a hardware configuration diagram of the image pickup apparatus 100. The image pickup apparatus 100 includes a CPU 204, a RAM 205, a ROM 206, a secondary storage unit 207 (flash memory), a USB control unit 208, a wireless communication unit 209, an image coding unit 202, a sensor processing unit 203, and the image processing unit 203 connected to the internal bus 210. It has an operation unit 103. Further, an image processing unit 201 is connected to the image coding unit 202, and a first surface imaging unit 102 and a second surface imaging unit 102 are connected to the image processing unit 201 as an imaging unit 102. A top direction distance measurement sensor 104, a ground direction distance measurement sensor 105, a peripheral direction distance measurement sensor 106, and a peripheral direction distance measurement sensor 106 are connected to the sensor processing unit 203.

The image pickup apparatus 100 is controlled by the CPU 204 executing the program stored in the ROM 206. When the user operates the operation unit 103 to perform imaging, the image processing unit 201 performs image processing such as stitching of captured images acquired from the first surface imaging unit 102 and the second surface imaging unit 102 and correction of image quality. ..

The image processing unit 201 outputs an image to the image coding unit 202, and the image coding unit 202 encodes (compresses by a predetermined method). The image coding unit 202 transfers the captured image coded data to the RAM 205 via the internal bus 210.

The sensor processing unit 203 similarly transfers the distance measurement data of the top direction distance measurement sensor 104, the ground direction distance measurement sensor 105, and the peripheral direction distance measurement sensor 106 to the RAM 205 via the internal bus 210.

The CPU 204 that executes the program stores the captured image coding data and the distance measurement data on the RAM 205 in one file, and stores them in the secondary storage unit 207. The stored captured image coded data is transferred to the information processing device (shown in FIG. 3) at the connection destination via the USB control unit 208 and the wireless communication unit 209. The information processing device is, for example, a device capable of executing application software such as a smartphone, a tablet terminal, or a PC (Personal Computer), an electronic device, a computer, or the like.

<About information processing equipment>
FIG. 3 is an example of a schematic perspective view of the information processing apparatus 10. The information processing device 10 in FIG. 3 is assumed to be a PC. The PC has a display 308 as a display device, a keyboard 311 as an input device, and a mouse 312.

The information processing device 10 executes application software. When the information processing device 10 executes this application software, the spherical image and the spherical image are drawn according to the contents programmed by the developer on the 3D computer graphic (3DCG) in which the 3D object and the spherical image are drawn. Objects work together.

The application software may be a Web application. In this case, the information processing device 10 executes the browser software and acquires the Web application described in HTML and JavaScript (registered trademark) from the server.

The application software has a built-in all-sky image three-dimensional drawing library, which will be described later, and the application software calls the functions of the all-sky image three-dimensional drawing library to link the three-dimensional object and the all-sky image. Operate 3D computer graphics.

Since the developer can program using the library, the development process can be reduced.

<< Hardware configuration example of information processing device >>
FIG. 4 shows an example of a hardware configuration diagram of the information processing device 10. The information processing device 10 includes a CPU 301 that controls the operation of the entire device, an IPL (Initial Program Loader), a ROM 302 that stores static data, a RAM 303 that is used as a work area of the CPU 301, a program 304p that is executed by the CPU 301, and various data. Has an HD304 that stores the information. It also has a HDD (Hard Disk Drive) 305 that controls reading or writing of various data to the HD 304 according to the control of the CPU 301, and a media drive 307 that controls reading or writing (storage) of data to a recording medium 306 such as a flash memory. .. In addition, a display 308 that displays various information such as cursors, menus, windows, characters, or images, a GPU (Graphic Processer Unit) 310 that draws on the display, and a network I for data transmission using a communication network. It has a keyboard 311 with a plurality of keys for inputting / F309, characters, numerical values, various instructions, and the like. In addition, an optical storage medium that controls reading or writing of various data to a mouse 312 that selects and executes various instructions, selects a processing target, moves a cursor, and the like, an optical storage medium 313 as an example of a detachable recording medium, and the like. It includes a drive 314 and a bus line 320 such as an address bus or a data bus for electrically connecting each of the above components as shown in FIG.

The program 304p is a file in an installable format or an executable format, and may be recorded and distributed on a computer-readable recording medium such as the recording medium 306 or the optical storage medium. Further, the program 304p may be distributed to the information processing apparatus 10 in a file in a format that can be installed or an executable format from the server for program distribution. Program 304p includes the above application software.

<About functions>
FIG. 5 is an example of a functional block diagram showing the functions of the image pickup apparatus 100 and the information processing apparatus 10 in a block shape. First, the image pickup apparatus 100 includes an image data acquisition unit 11, a distance data acquisition unit 12, an operation reception unit 13, a file creation unit 14, and a file transmission unit 15. Each of these parts included in the image pickup apparatus 100 is a function or means realized by executing a program expanded from ROM 206 to RAM 205 by CPU 204 shown in FIG. 2 and controlling each part shown in FIG.

The image data acquisition unit 11 acquires (images) a spherical image according to the operation of the user. The image data acquisition unit 11 is realized by the CPU 204 executing a program to control the image processing unit 201 and the like.

The distance data acquisition unit 12 acquires distance measurement data in the ceiling direction, the ground direction, and the peripheral direction according to the user's operation. The distance data acquisition unit 12 is realized by the CPU 204 executing a program to control the sensor processing unit 203 and the like.

The operation reception unit 13 receives various user operations on the image pickup apparatus 100. The operation reception unit 13 is realized by the CPU 204 executing a program to control the operation unit 103 and the like.

The file creation unit 14 creates a file in which the spherical image and the distance measurement data are combined into one. The file creation unit 14 is realized by the CPU 204 executing a program or the like. The file is stored in the secondary storage unit 207. Table 1 shows an example of the file format.

The file transmission unit 15 transmits the file stored in the secondary storage unit 207 to the information processing device 10. All the files of the secondary storage unit 207 may be transmitted to the information processing device 10, or only the files selected by the user may be transmitted. The file transmission unit 15 is realized by the CPU 204 executing a program to control the USB control unit 208 or the wireless communication unit 209. In addition to being transmitted, the file may be stored in a storage medium such as a USB memory, and the information processing device 10 may read the file from the storage medium. Further, the file may be transmitted to the server in which the file is stored, and the information processing apparatus 10 may receive (download) the file from the server.

The information processing device 10 includes a file acquisition unit 21, a control unit 22, an operation reception unit 23, a spherical image three-dimensional library 24, and a drawing processing unit 25. Each of these parts included in the information processing apparatus 10 is a function or means realized by the CPU 301 shown in FIG. 4 executing the program 304p developed from the HD 304 to the RAM 303 and controlling each part shown in FIG.

The file acquisition unit 21 acquires the file created by the image pickup apparatus 100. For example, it may be received from the image pickup device 100 wirelessly or by wire, read from a storage medium, or received from a server. The file acquisition unit 21 is realized by the CPU 301 executing the program 304p expanded from the HD 304 to the RAM 303 to control the network I / F 309 and the like.

The control unit 22 controls the entire operation of the information processing device 10. The operation procedure is described in the application software, and the spherical image three-dimensional library 24 is called according to this description. The control unit 22 is realized by the CPU 301 executing the program 304p expanded from the HD 304 to the RAM 303.

The operation reception unit 23 receives various user operations on the information processing device 10. The operation reception unit 23 is realized by the CPU 301 executing the program 304p to control the keyboard 311 and the mouse 312.

The spherical image three-dimensional library 24 is a program having functions for arranging spherical images in the same coordinate system, assigning appropriate attributes, and linking them in order to link them with three-dimensional objects. .. A library is a set of parts in which commonly used functions are described in a program. Libraries have traditionally been used in the development of application software.

In the present embodiment, as an example, it has a drawing object generation unit 26, a camera object generation unit 27, a scene object generation unit 28, and a ball object generation unit 29. The spherical image three-dimensional library 24 has a class of these objects. A class is a blueprint of a program, and an object is an entity in which a specific numerical value is set in the class.

The drawing object performs processing related to drawing the generated object. For example, the presence / absence of drawing is set based on the drawing priority of each object, and the drawing processing unit is requested to acquire the spherical image and paste it on the screen object described later. The camera object generation unit 27 generates an object (camera object) of the image pickup apparatus 100. The camera object is the viewpoint of the spherical image. The scene object generation unit 28 generates a scene object for arranging spherical images and other objects. A scene object can be said to be an object that represents space, and has attributes such as the passage of time and gravity. The ball object generation unit 29 generates the ball object 47 as an example of a three-dimensional object linked with the spherical image. The ball object generation unit 29 can generate an arbitrary three-dimensional object defined (user-defined) by a developer or the like. The ball object 47 is just one example.

The drawing processing unit 25 performs processing related to rendering in which each object is drawn on the display 114, including the spherical image, in response to the drawing request from the drawing object. The drawing processing unit 25 is realized by the CPU 301 executing a program and controlling the GPU 310.

表１は全天球画像のデータと測距データが格納されるファイルのファイルフォーマットを模式的に示す。ファイルは主にファイルヘッダ情報、画像データ部、及び、測距データ部を有する。ファイルヘッダ情報はファイルから測距データをアプリケーションソフトウェアが抽出するためのファイル内の測距データ有無、周囲測距密度、及び、ファイル内の測距データのオフセット位置を示す。

Table 1 schematically shows the file format of the file in which the spherical image data and the distance measurement data are stored. The file mainly has a file header information, an image data unit, and a distance measurement data unit. The file header information indicates the presence / absence of distance measurement data in the file for the application software to extract the distance measurement data from the file, the ambient distance measurement density, and the offset position of the distance measurement data in the file.

Peripheral distance measurement density represents the spatial density of distance measurement data in the peripheral direction. For example, when there are 360 points of distance measurement data for 360 ° around a certain horizontal position, the peripheral distance measurement density is 360 points / 360 ° = 1. In the case of 200 points, 200/360 = 0.56.

The image data unit stores image data (code header and code) of the existing coding method. The distance measurement data unit stores heavenly distance measurement data, ground direction distance measurement data, and peripheral direction distance measurement data.

<Conceptual explanation of objects generated by the spherical image 3D library>
FIG. 6 shows an example of an object generated by the spherical image three-dimensional library 24. Each object is a three-dimensional object arranged in the same coordinate system. The camera object 44 is arranged at a viewpoint position for the information processing apparatus 10 to display a virtual three-dimensional space. The camera object 44 has coordinates, a line-of-sight direction, and a zoom magnification (angle of view) as parameters. The shape of the camera as shown is for illustration purposes only, and the camera object 44 does not have to have a size.

The screen object 43 is an object to which the spherical image is pasted. The screen object 43 is attached to the side of the camera object 44 (inside the screen object 43). The shape of the screen object 43 is a sphere as shown, because the spherical image is a sphere. Other shapes of the screen object 43 will be described later. The screen object 43 is actually constructed by combining triangular or quadrangular polygons.

The coordinates of the screen object 43 are such that the center position coincides with the camera object 44. This is because each object is arranged so that the information processing device 10 can see the spherical image from the position of the image pickup device 100.

When the screen object 43 is a sphere, the radius is often predetermined by the focal length or the like. Since the user can change the angle of view α (zoom magnification), an appropriately determined value (for example, focal length) may be used as the radius.

The ceiling object 41 is an object representing the ceiling as seen from the image pickup apparatus 100, and is generated based on the heavenly distance measurement data included in the file (value obtained by adding the heavenly distance measurement data to the camera object 44). The width is infinite, and the height position is determined based on the celestial distance measurement data with reference to the camera object 44. Since the information processing device 10 does not display the ceiling object 41 but displays the spherical image attached to the screen object 43, the ceiling object 41 itself is colorless and transparent.

Although the ceiling object 41 is outside the screen object 43 in FIG. 6, it may be inside the screen object 43 depending on the height of the ceiling object 41. Even in this case, since the ceiling object 41 is colorless and transparent, the image of the ceiling in the real space attached to the screen object 43 is not obstructed.

The ground object 42 is an object representing the ground as seen from the image pickup apparatus 100, and is generated based on the ground direction distance measurement data included in the file. The width is infinite, and the height position is determined based on the ground-direction distance measurement data with reference to the camera object 44 (value obtained by subtracting the ground-direction distance measurement data from the camera object 44). Further, in the sense of the ground, it may be 0 (reference point of the coordinate system). Since the information processing device 10 does not display the ground object 42 but displays the spherical image pasted on the screen object 43, the ground object 42 itself is colorless and transparent.

Although the ground object 42 is outside the screen object 43 in FIG. 6, it may come inside the screen object 43 depending on the height of the ground object 42. Even in this case, since the ground object 42 is colorless and transparent, it does not block the image of the ground in the real space attached to the screen object 43.

FIG. 7 shows an example of a screen object 43 that is not a sphere. In FIG. 7, the screen object 43 is a plane. In this case, the size of the screen is variable so as to correspond to the size of the angle of view α of the camera object 44. Further, the screen object 43 may be a curved surface having an arbitrary curvature. The distance between the screen object 43 and the camera object 44 when it is not a sphere is a constant adjusted in advance so that the projected spherical image can be seen from the camera object 44 without distortion. For example, the distance is such that the larger the angle of view α, the larger the distance.

FIG. 8 is a diagram schematically showing the side object 46. FIG. 8 is a view of the camera object 44 as viewed from above (ceiling side). The side object 46 is an object representing the side surface (peripheral direction) of the image pickup apparatus 100, and the side object 46 is generated from the peripheral direction distance measurement data of the file. Peripheral direction distance measurement data is a strip-shaped (no thickness) three-dimensional object because it can change depending on the surrounding direction.

Although FIG. 8 shows the undulations of the shape in the peripheral direction, the shape of the imaging environment in the peripheral direction may be a shape having undulations in the height direction as well. Since the information processing device 10 does not display the side object 46 but displays the spherical image attached to the screen object 43, the side object 46 itself is colorless and transparent.

<Physical action>
Each object can operate under the same physical action as in real space, such as collision and fall. Therefore, attributes of physical action (presence / absence of interaction, presence / absence of influence of gravity, etc.) can be set for each object. Each generation unit of the spherical image three-dimensional library 24 sets the presence / absence and size of a physical action for each object according to the description in the application software by the developer.

No physical action is set on the screen object 43. That is, it does not collide with other objects or fall under the influence of gravity. The ceiling object 41, the ground object 42, and the side object 46 are created to interact with other objects. Application software can represent collisions with other objects (eg ball objects). However, the ceiling object 41, the ground object 42, and the side object 46 are set so as not to be affected by gravity. The ball object is set to interact with other objects and be further affected by gravity.

<Drawing priority>
The drawing priority is an attribute for the drawing processing unit 25 to determine which object to display when a plurality of objects overlap from the viewpoint. Drawing objects set and change the drawing priority of each object. This will be described with reference to FIG.

FIG. 9 is an example of a diagram illustrating drawing of a ball based on drawing priority. First, the drawing object sets the drawing priority of the screen object 43 to the lowest. Therefore, the drawing priority of the spherical image pasted on the screen object 43 is treated in the same manner.

The drawing priority of the ball object 47 is higher than that of the screen object 43, the ceiling object 41, the ground object 42, and the side object 46.

In principle, drawing objects display objects that are close to each other and have high drawing priority. When the ball object 47 is between the camera object 44 and the screen object 43, the ball object 47 is closer than the screen object 43 (all-sky image), and the drawing priority of the ball object 47 is higher. The ball object 47 is displayed. When the ball object 47 is between the screen object 43 and the side object 46, the screen object 43 is closer, but the drawing priority of the ball object 47 is higher, so that the ball object 47 is a display candidate. Further, in the ball object 47 and the side object 46, the ball object 47 is displayed because the ball object 47 is closer and has a higher priority. When the ball object 47 is farther than the side object 46, the ball object 47 has a higher priority and the side object 46 is transparent, but the drawing object exceptionally hides (masks) the ball object 47.

By determining the drawing priority in this way, it is possible to control the display and non-display of each object.

<Operation procedure>
The operation procedure of the application software created by the spherical image three-dimensional library 24 will be described with reference to FIGS. 10 and 11. This application software performs the action of throwing a ball from any position.

S1: The control unit 22 calls the drawing object generation unit 26 of the spherical image three-dimensional library 24 to generate a drawing object. Specifically, the drawing object generation unit 26 has a drawing class constructor, and sets initial values to appropriate attributes for the drawing object.

S2: Similarly, the control unit 22 calls the camera object generation unit 27 to generate the camera object 44. Also at this time, the camera class constructor sets the initial values (coordinates, angle of view, line-of-sight direction, etc.) in the attributes.

S3: Similarly, the control unit 22 calls the scene object generation unit 28 to generate a scene object. Also at this time, the scene class constructor sets the initial values (coordinates, time, magnitude of gravity, etc.) in the attributes. In addition, the file name of the spherical image is specified as this initial value.

S4: The scene object generation unit 28 cooperates to generate the screen object 43 when the scene object is generated. Required attribute values (coordinates, size, physical action, drawing priority, etc.) are set.

S5: Further, the scene object generation unit 28 acquires the designated spherical image file from the file acquisition unit 21.

S6: The scene object generation unit 28 generates a ceiling object 41 based on the heavenly distance measurement data stored in the file and adds it to the scene object. In addition, height, size, physical action, transparency, drawing priority, etc. are set as attributes.

S7: Similarly, the scene object generation unit 28 generates the ground object 42 based on the ground direction distance measurement data stored in the file and adds it to the scene object. In addition, height, size, physical action, transparency, drawing priority, etc. are set as attributes.

S8: Similarly, the scene object generation unit 28 generates a side object 46 based on the ambient direction distance measurement data stored in the file and adds it to the scene object. In addition, physical action, transparency, drawing priority, etc. are set as attributes.

S9, S10: Next, the control unit 22 calls the ball object generation unit 29 of the spherical image three-dimensional library 24 as a solid to be expressed in the scene object. The ball object generation unit 29 can generate various solids in addition to the ball. At this time as well, the solid class constructor sets the initial values (coordinates, size, speed, orientation, physical action, drawing priority, etc. are set). The size and position of the ball may be arbitrary.

S11: Next, the control unit 22 adds the generated ball object 47 to the scene object.

S12: Next, the control unit 22 gives a drawing instruction to the drawing object. That is, drawing is started.

S13, S14: The drawing object determines whether or not to draw each object to be drawn based on the definition of drawing priority, and sends the coordinates of each object to the drawing processing unit 25.

S15: Next, the drawing object reads the spherical image file.

S16: The drawing object transmits the image data to the drawing processing unit 25 as texture data for the screen object 43. The spherical image becomes the texture data of the screen object 43.

S17: The drawing processing unit 25 pastes the texture data on the screen object 43, and displays the three-dimensional object on the two-dimensional display based on the coordinates of each object to be drawn sent from the drawing object and the presence or absence of drawing. Perform rendering for. Rendering refers to processing the coordinates, camera position, angle of view, line-of-sight direction, imprint, etc. of each object to obtain a specific set of pixels.

S18: Now, the spherical image and the ball are displayed on the display.

S19: Next, the control unit 22 sets the velocity and the direction with respect to the ball object 47 in order to express the physical movement of the ball object 47.

S20: Next, the control unit 22 instructs the scene object to elapse the time. The passage of time is a set time interval in which the user sees the ball moving like a moving image.

S21: The scene object updates the position of the ball object 47 based on the passage of time, the moving speed, and the orientation. Since the ball object 47 is set to be affected by gravity due to the attribute related to physical action, it falls at the gravitational acceleration. In this embodiment, only the ball moves. In addition, a collision is determined between the ball and each of the other objects, and when a collision occurs, the velocity and direction are updated. In the event of a collision, the speed and direction after the collision are calculated from the angle and speed at which the ball object 47 approaches the side object 46 and the like, the coefficient of restitution between the ball object 47 and the side object 46 and the like.

S22: Next, the control unit 22 gives a drawing instruction to the drawing object again.

S23 to S27: It is not necessary to read the spherical image file, but the other steps are the same as in steps S13 to 18 above.

The application software repeats steps S19 to S27. The speed and orientation of the ball object 47 are repeatedly changed based on the update of the passage of time. As described above, the application software can perform the drawing process in which the spherical image and the three-dimensional object are linked by using the spherical image three-dimensional library 24.

Since the spherical image three-dimensional library 24 generates a screen object, a ceiling object, a ground object, and a side object 46 of the spherical image and interacts with the ball object 47, the three-dimensional object and the spherical image are linked. It will be easier for developers to develop the applied application software.

In this embodiment, a spherical image three-dimensional library 24 capable of generating a light source object will be further described.

The spherical image three-dimensional library 24 of this embodiment estimates the position of the light source in the image when the spherical image is read. Further, it is estimated that the light source is arranged on the ceiling, the side surface, or the ground, the arrangement position of the light source in the space is calculated based on the distance measurement data, and the light source object is automatically generated. This will be described with reference to FIG.

FIG. 12 shows an example of an object generated by the spherical image three-dimensional library 24. In the present embodiment, the components having the same reference numerals in FIG. 6 perform the same functions, and therefore, only the main components of the present embodiment may be mainly described.

FIG. 12 shows the light source object 48. The light source object 48 represents a light source in space. Each of the three-dimensional objects placed in space by the application software is affected by the light source, and the application software represents a shadow on the surface and a shadow 49 of the ball object 47.

The scene object estimates the light source position from the spherical image. For example, the spherical image is binarized with the brightness corresponding to the illumination to remove noise. It is estimated that the light source is located in the pixel range having an area of a certain area or more and similar to the shape of the light source. The shape of the light source may be linear or circular. If the spherical image three-dimensional library 24 cannot estimate the light source, the light source object is placed at the viewpoint position.

The side object 46, the ground object 42, and the ceiling object 41 are colorless and transparent, but the shadow 49 formed by another object (ball) is projected on the surface.

It is assumed that the application software throws the ball in the line-of-sight direction from the vicinity of the camera object 44. The light of the light source object hits a part of the surface of the ball object 47, and a shadow is generated. Further, since the shadow 49 of the ball object 47 is generated on the straight line connecting the light source object and the ball, the shadow 49 is drawn on the ground object 42 and the side object 46.

<Operation procedure>
FIG. 13 is an example of a sequence diagram for explaining the operation procedure of the application software created by the spherical image three-dimensional library 24. In the explanation of FIG. 13, the difference from FIGS. 10 and 11 will be mainly described. The processes in steps S1 to S8 are the same as in FIG.

S8-2: Next, the scene object generation unit 28 estimates the light source position from the spherical image. The position of the light source is specified by latitude and longitude.

S8-3: The scene object determines the coordinates of the light source object based on the light source position and the distance measurement data. Determine where the straight line connecting the camera object 44 and the light source position (latitude / longitude) intersects the ceiling, ground, or side surface, and place the light source object at the intersection of the ceiling, ground, or side surface that is determined to intersect. ..

Subsequent steps S9 to S13 are the same as in FIG.
S14: The drawing object transmits the coordinates of the light source object in addition to the coordinates of each object to be drawn to the drawing processing unit 25. Subsequent processing is the same as in Example 1.

As a result, when the drawing processing unit 25 renders, the shading processing can be performed based on the light source object. Shading is a process of shading based on a light source object, a reflective material of the object, and a normal vector of the object.

As described above, the spherical image three-dimensional library 24 of this embodiment detects the light source from the spherical image and generates the light source object, so that the application software can shade other objects.

In this embodiment, the spherical image three-dimensional library 24 that can create application software that cooperates with a three-dimensional object when the spherical image is captured as a moving image will be described. In the first embodiment, the spherical image was stationary even when the ball object 47 was moved, but in this embodiment, the spherical image of the three-dimensional library 24 is changed as the ball object 47 is moved. Can be made to. It is assumed that the imaging device 100 does not move (fixed) during the imaging of the moving image. Therefore, the viewpoint does not move either.

The distance data acquisition unit 12 of this embodiment continuously detects the distance measurement data during the imaging of the moving image. Therefore, the ranging data has a temporal change in accordance with the imaging of the moving image. Table 2 shows an example of the file format of this embodiment.

表２のファイルフォーマットは、ファイルヘッダ情報、動画データ部、及び測距データ部を有する。動画データ部には動画データと音声データが格納され、測距データ部には、時系列天方向測距データ、時系列地方向測距データ、及び、時系列周囲方向測距データが格納される。

The file format in Table 2 has a file header information, a moving image data section, and a ranging data section. Video data and audio data are stored in the moving image data unit, and time-series heavenly direction distance measurement data, time-series ground direction distance measurement data, and time-series peripheral direction distance measurement data are stored in the distance measurement data unit. ..

<Operation procedure>
The operation of the application software of this embodiment is the same as that of FIG. 13 of the second embodiment, but the processing following FIG. 13 is different in this embodiment.

FIG. 14 is an example of a sequence diagram illustrating an operation procedure of the application software created by the spherical image three-dimensional library 24. In FIG. 14, the spherical image projected by each object and the scene object is updated according to the passage of time. The processing of S19 and S20 is the same as that of FIG.

S21-1: The scene object reads the distance measurement data at the corresponding time from the time-series celestial distance measurement data contained in the spherical moving image data file, and updates the height of the ceiling object 41.

S21-2: The scene object reads the distance measurement data at the corresponding time from the time-series ground direction distance measurement data contained in the spherical moving image data file, and updates the height of the ground object 42.

S21-3: The scene object reads the distance measurement data at the corresponding time from the time-series peripheral direction distance measurement data contained in the spherical moving image data file, and updates the shape of the side object 46.

S21-4: The scene object reads the image data at the corresponding time from the moving image data of the spherical image and updates the light source object. That is, the position of the light source is estimated and the coordinates of the light source are updated.

S21-5: Further, the scene object updates the position of the ball object 47 based on the definitions of velocity, direction, and gravity, and determines the collision between the ball object 47 and each object. In case of a collision, update the speed and orientation.

The processing of steps S22 to 24 is the same as that of the second embodiment.
S24-2: The drawing object updates the moving image data of the whole celestial sphere with the passage of time. That is, the image data corresponding to the specified time is read from the moving image data.

By the above processing, the application software of this embodiment updates the spherical image in accordance with the movement of the three-dimensional object, so that the spherical image of the moving image and the three-dimensional object can be linked.

In this embodiment, the spherical image three-dimensional library 24 capable of creating application software that cooperates with a three-dimensional object even when the spherical image captured by the rotatable imaging device 100 is captured as a moving image will be described. .. The difference from the third embodiment is that the image pickup apparatus 100 rotates. Since the image pickup device 100 detects the rotation angle and the scene object updates the positions of the ceiling object 41, the ground object, the side object 46, the light source object 48, and the ball object 47, camera shake or the like occurs in the image pickup device 100. However, the information processing device 10 can draw an object from a certain viewpoint.

FIG. 15 shows an example of a hardware configuration diagram of the image pickup device 100 of this embodiment, and FIG. 16 is an example of a functional block diagram showing the functions of the image pickup device 100 and the information processing device 10 of this embodiment in a block shape. is there. In the description of FIG. 15, the difference from FIG. 2 will be mainly described. As shown in FIG. 15, the image pickup apparatus 100 has an angular velocity sensor 107 and is connected to the sensor processing unit 203.

Further, in the description of FIG. 16, the difference from FIG. 5 will be mainly described. The image pickup apparatus 100 of FIG. 16 has an angular velocity data acquisition unit 16. The angular velocity data acquisition unit 16 detects the angular velocity generated in the image pickup apparatus 100. The angular velocity data acquisition unit 16 is realized by the CPU 204 executing a program to control the sensor processing unit 203 and the like. The angular velocity data is stored in a file together with the image data as shown in Table 3.

表３は本実施例のファイルフォーマットの一例を示す。表２と比較すると、角速度データ部を有する。角速度データ部には時系列角速度データが格納される。また、ファイルヘッダ情報には、角速度データをアプリケーションソフトウェアが読み出せるように、角速度センサ１０７データ有無及び角速度センサオフセットを有する。

Table 3 shows an example of the file format of this embodiment. Compared with Table 2, it has an angular velocity data unit. Time-series angular velocity data is stored in the angular velocity data section. Further, the file header information includes the presence / absence of the angular velocity sensor 107 data and the angular velocity sensor offset so that the application software can read the angular velocity data.

<Operation procedure>
FIG. 17 is an example of a sequence diagram for explaining the operation procedure of the application software created by the spherical image three-dimensional library 24. In the description of FIG. 17, the difference from FIG. 14 will be mainly described. In FIG. 17, the application software updates each object and the projected spherical image and corrects the coordinates of each object based on the rotation of the image pickup apparatus 100 in accordance with the passage of time. Even if the image pickup device 100 rotates, the front direction of the image pickup device 100 is corrected to be constant.

S21-1: The scene object reads the angular velocity at the corresponding time from the time-series angular velocity data of the moving image data file of the whole sky, calculates the rotation angle, and updates the coordinates of the ceiling object 41. That is, the ceiling object 41 is rotated by the same rotation angle as the image pickup device 100.

S21-2: The scene object reads the angular velocity at the corresponding time from the time-series angular velocity data of the moving image data file of the whole sky, calculates the rotation angle, and updates the coordinates of the ground object 42. That is, the ground object 42 is rotated by the same rotation angle as the image pickup device 100.

S21-3: The scene object reads the angular velocity at the corresponding time from the time-series angular velocity data of the moving image data file of the whole sky, calculates the rotation angle, and updates the coordinates of the side object 46. That is, the side object 46 is rotated by the same rotation angle as the image pickup device 100.

S21-4: The scene object reads the angular velocity at the corresponding time from the time-series angular velocity data of the moving image data file of the whole sky, calculates the rotation angle, and estimates the light source position. Then, the light source object 48 is rotated by the same rotation angle as the image pickup device 100.

S21-5: Further, the scene object updates the position of each object based on the definitions of velocity, orientation and gravity, and rotates the ball object 47 by the same rotation angle as the image pickup device 100. In addition, the collision between the ball and each object is determined. In case of a collision, update the speed and orientation.

The processing of steps S22 to 24 is the same as in FIG.
S24-2: The drawing object updates the moving image data of the whole celestial sphere with the passage of time. Further, the spherical image is rotated by the same rotation angle as that of the image pickup device 100.

By the above processing, the application software of this embodiment rotates each object and the spherical image even if the image pickup device 100 is rotated due to camera shake or the like when capturing a moving image, so that the housing 101 such as camera shake is rotated. It is possible to draw from a certain viewpoint by correcting the viewpoint deviation due to.

In this embodiment, the spherical image three-dimensional library 24 capable of creating application software that cooperates with a three-dimensional object even when the spherical image captured by the image pickup device 100 capable of translating is captured as a moving image will be described. To do. The difference from the fourth embodiment is that the image pickup apparatus 100 moves in parallel. By detecting the acceleration of parallel movement, the application software updates the positions of the ceiling object 41, the ground object 42, the side object 46, the light source object 48, and the ball object 47, so that each object is updated when the imager moves. Can be moved together to draw each object.

FIG. 18 shows an example of a hardware configuration diagram of the image pickup device 100 of this embodiment, and FIG. 19 is an example of a functional block diagram showing the functions of the image pickup device 100 and the information processing device 10 of this embodiment in a block shape. is there. In the description of FIG. 18, the difference from FIG. 15 will be mainly described. As shown in FIG. 18, the image pickup apparatus 100 has an acceleration sensor 108 and is connected to the sensor processing unit 203.

In the description of FIG. 19, the difference from FIG. 16 will be mainly described. The imaging device 100 of FIG. 19 has an acceleration data acquisition unit 17. The acceleration data acquisition unit 17 detects the acceleration generated in the image pickup apparatus 100. The acceleration data acquisition unit 17 is realized by the CPU 204 executing a program to control the sensor processing unit 203 and the like. The acceleration data is stored in a file together with the image data as shown in Table 4.

表４は本実施例のファイルフォーマットの一例を示す。表３と比較すると、加速度データ部を有する。加速度データ部には時系列加速度データが格納される。また、ファイルヘッダ情報には、加速度データをアプリケーションソフトウェアが読み出せるように、加速度センサ１０８データ有無及び加速度センサオフセットを有する。

Table 4 shows an example of the file format of this embodiment. Compared with Table 3, it has an acceleration data unit. Time-series acceleration data is stored in the acceleration data unit. Further, the file header information includes the presence / absence of acceleration sensor 108 data and the acceleration sensor offset so that the application software can read the acceleration data.

<Operation procedure>
FIG. 20 is an example of a sequence diagram for explaining the operation procedure of the application software created by the spherical image three-dimensional library 24. In the description of FIG. 20, the difference from FIG. 17 will be mainly described. In FIG. 20, each object and the projected spherical image are updated according to the passage of time, and the coordinates of the objects are corrected based on the translation of the image pickup apparatus 100.

S21-1: The scene object reads the acceleration at the corresponding time from the time-series acceleration data of the moving image data file of the whole sky, calculates the movement amount, and updates the coordinates of the ceiling object 41. That is, the ceiling object 41 is translated by the amount that the imaging device 100 is translated.

S21-2: The scene object reads the acceleration at the corresponding time from the time-series acceleration data of the moving image data file of the whole sky, calculates the movement amount, and updates the coordinates of the ground object 42. That is, the ground object 42 is translated by the amount that the imaging device 100 is translated.

S21-3: The scene object reads the acceleration at the corresponding time from the time-series acceleration data of the moving image data file of the whole sky, calculates the movement amount, and updates the coordinates of the side object 46. That is, the side object 46 is translated by the amount that the image pickup apparatus 100 is translated.

S21-4: The scene object reads the acceleration at the corresponding time from the time-series acceleration data of the moving image data file of the whole sky, calculates the movement amount, and estimates the light source position. Then, the light source object 48 is translated by the amount that the image pickup apparatus 100 is translated.

S21-5: Further, the scene object updates the position of each object based on the definitions of velocity, direction and gravity, and translates the ball object 47 by the same amount of movement as the imaging device 100. In addition, the collision between the ball and each object is determined. In case of a collision, update the speed and orientation.

The processing of steps S22 to 24 is the same as that of the fourth embodiment.
S24-2: The drawing object updates the moving image data of the whole celestial sphere with the passage of time. In addition, the spherical image is translated by the same amount of movement as that of the image pickup device 100.

By the above processing, the application software of the present embodiment moves each object and the spherical image even if the image pickup device 100 moves at the time of capturing a moving image, so that the viewpoint shift due to the movement of the housing 101 is corrected. You can draw from a certain point of view. When combined with the fourth embodiment, it is possible to correct the deviation of the viewpoint due to the movement and rotation of the viewpoint.

In this embodiment, a spherical image three-dimensional library 24 that can generate a ceiling object 41 and a ground object 42 having a finite size and having an inclination will be described.

FIG. 21 is an example of a diagram illustrating an imaging scene. In Examples 1 to 5, it was assumed that the ceiling and the ground were horizontal. In this embodiment, as shown in FIG. 21, the ceiling 51 in the upward direction of the image pickup apparatus 100 has an inclination and has a finite size. Further, the ground 52 in the downward direction of the image pickup apparatus 100 also has an inclination and has a finite size. In such an imaging scene, it is preferable that the inclination of the ceiling 51 and the ground 52 is reflected in the ceiling object 41 and the ground object 42.

FIG. 22 is an example of an external view of the image pickup device 100 that captures a spherical image. In the description of FIG. 22, the difference from FIG. 1 will be mainly described. In contrast to the image pickup device 100 of FIG. 1, the image pickup device 100 of FIG. 22 has a plurality of top direction distance measurement sensors 104 and a plurality of ground direction distance measurement sensors 105. Therefore, since each can detect the distance at a plurality of measurement points, the inclination of the ceiling or the ground can be detected.

FIG. 23 shows an example of an object generated by the spherical image three-dimensional library 24. The scene object generation unit 28 determines the inclination of the ceiling and the ground based on the distance measurement data, and adds the ceiling object 41 and the ground object 42 having this inclination to the scene object. The size of the ceiling and the ground is determined by the range in which the distance measurement data is detected. As shown in FIG. 23, the ceiling object 41 and the ground object 42 are generated as objects of a finite size having an inclination based on the distance measurement data of a plurality of measurement points.

In this way, even if a ceiling or ground having a finite size and having an inclination or undulation is imaged, it is possible to appropriately generate an object.

<Summary>
As described above, in the spherical image three-dimensional library 24 of the present embodiment, height information from the viewpoint position to the "ceiling" and "ground" and distance information from the viewpoint position to the surrounding direction are converted into spherical images. By utilizing the addition, objects such as ceiling objects, ground objects, and side objects can be arranged in three-dimensional space based on height information and distance information, respectively. In addition, it is possible to calculate a physical action with an arbitrary three-dimensional object. Since object creation and physical actions are supplied by the library, it is easy to develop application software that links 3D objects and spherical images.

<Other application examples>
Although the best mode for carrying out the present invention has been described above with reference to examples, the present invention is not limited to these examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

For example, instead of the information processing device 10 executing the application software, the imaging device 100 may execute the application.

Further, the distance measurement data is not limited to the value captured by the imaging device 100, and may be a value manually set by the user. The user can set when there is no distance measurement data or edit the distance measurement data.

Further, although the indoor embodiment has been mainly described, the image pickup apparatus 100 may image the outdoors and a library suitable for the outdoors may be prepared. In this case, the ceiling object 41 may not be needed.

Further, although the three-dimensional space is arranged on the spherical image in which the surrounding 360 degrees is captured, the image data does not have to be captured in the peripheral 360 degrees. The image data of the present embodiment may be a curved wide-angle image that is easy to see by being arranged in a three-dimensional space. For example, the image may be taken 360 degrees only in the horizontal direction, or the upper half or the lower half of the whole celestial sphere may be imaged. Further, a wide-angle image of about 180 degrees may be used.

The scene object generation unit 28 is an example of an imaging object generation means, the camera object generation unit 27 is an example of a camera object generation means, and the ball object generation unit 29 is an example of a user object generation means, and a drawing object generation unit is generated. Part 26 is an example of drawing means. The ceiling object 41 is an example of the first object, the ground object 42 is an example of the second object, and the side object 46 is an example of the third object. The ball object 47 is an example of a three-dimensional object.

10: Information processing device 24: Spherical image three-dimensional library 41: Ceiling object 42: Ground object 43: Screen object 44: Camera object 46: Side object 47: Ball object 48: Light source object 100: Imaging device

Japanese Unexamined Patent Publication No. 2014-127001

Claims (10)

An information processing device that acquires at least one of wide-angle images and distance measurement data in the ceiling direction, the ground direction, or the surrounding direction.
A screen object to which the wide-angle image is attached is generated, and the first object based on the ceiling-direction ranging data, the second object based on the ground-direction ranging data, or the peripheral-direction ranging data. An imaging object generating means that generates at least one of the third objects based on
A camera object generation means for generating a camera object that serves as a viewpoint of the wide-angle image,
User object generation means to generate user-defined 3D objects,
As a drawing means for sending the screen object, the first object, the second object, the third object, the three-dimensional object, and the wide-angle image to be attached to the screen object to the drawing processing unit. Make it work,
The imaging object generation means is a program characterized in that each object is updated based on the attributes related to the physical action of each object. It has a library including a class of an imaging object generating means, a camera object generating means, and a user object generating means, and by calling the library, the screen object, the first object, the second object, and the like. The program according to claim 1, wherein the third object, the camera object, and the three-dimensional object are generated. The screen object, the first object, the second object, the third object, and the three-dimensional object have the attributes of transparency and drawing priority.
The program according to claim 1 or 2, wherein the drawing means controls the display of each object based on the transparency and the drawing priority. The imaging object generating means detects a light source from the wide-angle image, and represents a light source at a position where the light source is detected among the first object, the second object, or the third object. Create an object and
The program according to any one of claims 1 to 3, wherein the drawing means causes the drawing processing unit to draw a shadow generated by the light source object. The physical action includes settings related to interaction with other objects.
The program according to any one of claims 1 to 4, wherein the imaging object generating means determines the presence or absence of collision of each object by using the setting related to the interaction. The wide-angle image is a moving image, and the distance measurement data is recorded in chronological order together with the moving image.
The imaging object generation means updates the first object, the second object, and the third object with the distance measurement data at a designated time.
The 3D object is updated with the set velocity, orientation and physics.
The program according to any one of claims 1 to 5, wherein the drawing means sends the wide-angle image at the time to the drawing processing unit. The information processing device acquires angular velocity data regarding the rotation angle of the image pickup device that has captured the wide-angle image, and obtains angular velocity data.
The imaging object generation means updates the first object, the second object, the third object, and the three-dimensional object by using the angular velocity data.
The program according to any one of claims 1 to 6, wherein the drawing means rotates the wide-angle image using the angular velocity data and then sends the wide-angle image to the drawing processing unit. The information processing device acquires acceleration data of the image pickup device that has captured the wide-angle image, and obtains acceleration data.
The imaging object generation means updates the first object, the second object, the third object, and the three-dimensional object by using the acceleration data.
The program according to any one of claims 1 to 7, wherein the drawing means moves the wide-angle image using the acceleration data and then sends the wide-angle image to the drawing processing unit. The information processing device acquires distance measurement data in the ceiling direction or distance measurement data in the ground direction of a plurality of measurement points, and obtains the distance measurement data in the ceiling direction.
The imaging object generation means generates the first object having an inclination based on the distance measurement data in the ceiling direction of a plurality of measurement points.
The program according to any one of claims 1 to 8, which generates the second object having an inclination based on the distance measurement data of a plurality of measurement points in the ground direction. An information processing device that acquires at least one of a wide-angle image and distance measurement data in the ceiling direction, the ground direction, or the peripheral direction.
A screen object to which the wide-angle image is attached is generated, and the first object based on the ceiling-direction ranging data, the second object based on the ground-direction ranging data, or the peripheral-direction ranging data. An imaging object generating means that generates at least one of the third objects based on
A camera object generation means for generating a camera object that serves as a viewpoint of the wide-angle image,
User object generation means to generate user-defined 3D objects,
A drawing means for sending the screen object, the first object, the second object, the third object, the three-dimensional object, and the wide-angle image to be attached to the screen object to the drawing processing unit. Have,
The image pickup object generation means is an information processing device characterized in that each object is updated based on an attribute related to a physical action of each object.

Images