JP2020154457A - Image display system, image distribution method, and program - Google Patents

Abstract

To provide an image display system that can display content from an appropriate viewing location.SOLUTION: An image display system 100 includes a terminal device 30 receiving content from an image distribution device 10 that distributes images. The image distribution device comprises a display pattern generation part 16 generating a display pattern that displays one or more pieces of content from different viewing locations, a constraint part 17 that constrains the viewing locations, and a delivery part 12 that delivers the display pattern and the content, or a plurality of display images generated based on the display pattern, to the terminal device.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image display system, an image distribution method, and a program.

A distribution-type display advertisement is known in which an advertiser displays an advertisement in an advertisement space provided on a Web page. Advertisers can display advertisements for products and services in the inventory provided by various websites. Viewers who are interested in the products and services displayed in the inventory click (or tap) the inventory, so advertisers can attract viewers to their website.

Advertisers want to increase the frequency with which viewers click on advertisements, but the number of advertisements posted on one Web page is increasing, and advertisers are devising display methods that attract the attention of viewers. (See, for example, Patent Document 1.). Patent Document 1 discloses a technique for enhancing the effectiveness (click rate) of an advertisement by delivering a video advertisement. Since people tend to be deprived of their eyes by moving objects, it is possible to increase the attention of advertisements and improve the effectiveness of advertisements.

In addition, some image data, such as 3DCG (Three-dimensional computer graphics) images and wide-angle images, have interactive contents whose appearance changes (viewpoint changes) depending on the operation of the viewer. Attempts have also been made to increase the degree of attention and enhance the effectiveness of advertisements by displaying such contents on Web pages.

However, there is a problem that the image is not always displayed from an appropriate viewpoint in the content such as a 3DCG image or a wide-angle image whose viewpoint can be changed. Supplementally, there is a display method in which the terminal device that receives the content displays the content in the advertising space while continuously changing the viewpoint of the content. In this display method, a part of the content is displayed one after another in the ad space, so that the image can be displayed like a moving image even though it is a still image. It is known that the advertising effect is higher than that of a normal still image advertisement because the image of the advertising space is automatically changed.

However, such a content display method may automatically display the content viewed from a viewpoint that the viewer does not specify so much, which may impair the advertising effect. For example, when the viewer looks down at the content from directly above or looks up from directly below, the viewer's sense of top and bottom may be upset.

In view of the above problems, an object of the present invention is to provide an image display system capable of displaying contents from an appropriate viewpoint.

In view of the above problems, the present invention is an image display system in which a terminal device receives content from an image distribution device that distributes images, and the image distribution device displays one or more contents from different viewpoints. A display pattern generation unit that generates the above, a constraint unit that constrains the viewpoint, and a distribution unit that distributes the display pattern and the content, or a plurality of display images generated based on the display pattern to the terminal device. The terminal device has an acquisition unit that acquires the display pattern and the content, or a plurality of display images generated based on the display pattern, and the display of the content acquired by the acquisition unit. It is characterized by having a display unit for displaying the plurality of display images generated based on the display pattern, or displaying the plurality of display images by changing the viewpoint with a pattern.

The present invention can provide an image display system capable of displaying contents from an appropriate viewpoint.

As an example of a wide-angle image, it is an example of a diagram for explaining a spherical image. It is a figure explaining the outline of an image display system. It is a figure explaining the transition prohibition region at the time of animating display of a spherical image or a 3DCG image by a terminal device. It is a block diagram of an example of an image display system. It is a hardware block diagram of an example of a computer system. It is a hardware configuration diagram of an example of a terminal device. It is a use image figure of an example of an omnidirectional camera. It is a figure explaining the outline of the process from the image taken by the omnidirectional camera to the creation of the omnidirectional image. It is a figure explaining the outline of the process from the image taken by the omnidirectional camera to the creation of the omnidirectional image. This is an example of a diagram for explaining the line of sight of the viewer. It is a figure which showed an example of the relationship between the predetermined area information and the image of a predetermined area. It is a figure which shows typically the 3D model and projective transformation. It is a figure which shows an example 3D model which was displayed by changing the viewpoint. This is an example of a diagram schematically explaining information used by a Web page. It is a functional block diagram of an example which shows the function of a terminal device, an advertiser Web server, a partner site Web server, an image distribution device, SSP and DSP in a block form. It is an example function block diagram which shows the function of a content reception part and an image registration terminal in a block shape. It is a figure which shows an example content registration screen displayed on the display of an image registration terminal. This is an example of a flowchart showing a procedure in which the attention point determination unit determines the attention point. It is a figure which shows the partial image of an example of a spherical image. It is a flowchart which shows an example of the whole flow which the display pattern generation part generates a display pattern. It is an example of the figure explaining the parameter of the line-of-sight direction of a spherical image or the position of a virtual camera in a 3DCG image. It is an example of the figure explaining the simplified parameter of the line-of-sight direction of the spherical image or the position of the virtual camera in the 3DCG image. It is a figure which shows an example of the virtual camera in the line-of-sight direction of an omnidirectional image or a 3DCG image when a horizontal orbit is adopted. It is a figure which shows an example of the line-of-sight direction of a spherical image, or the position of a virtual camera in a 3DCG image when the large circular orbit is adopted. It is a figure which shows an example of the position of the virtual camera in the 3DCG image when the spiral trajectory is adopted. It is a flowchart which shows the procedure executed in step S103 of FIG. 20 at the time of generating a display pattern using an element trajectory. It is a figure explaining the cumulative value of the attention degree on the element trajectory. It is a flowchart of an example which shows the procedure which obtains the 2 or more element trajectories. This is an example of a sequence diagram illustrating a procedure for the image display system to deliver a spherical image or a 3DCG image displayed as an advertisement. It is a flowchart of an example explaining the operation of the terminal apparatus which received advertisement data. It is a figure which shows some form in which advertisement data is delivered.

Hereinafter, an image display system and an image distribution method performed by the image display system will be described as an example of a method for carrying out the present invention.

<Outline of the image display system of this embodiment>
FIG. 1 is an example of a diagram for explaining the spherical image 6 as an example of a wide-angle image. FIG. 1 (a) shows an all-sky image 6 represented by a three-dimensional three-dimensional sphere CS, and FIG. 1 (b) is an image represented by equirectangular projection (hereinafter, “equirectangular projection image””. The whole celestial sphere image 6 is shown. The omnidirectional image 6 generated by the omnidirectional camera has a three-dimensional structure in which an equirectangular projection image as shown in FIG. 1B is attached to a three-dimensional sphere CS. The virtual camera IC corresponds to the position of the eyes of the viewer, and in FIG. 1A, the position of the eyes is at the center of the spherical image 6. The viewer rotates the three axes around the X-axis, Y-axis, and Z-axis passing through the virtual camera IC, and sets an arbitrary region of the spherical image 6 as a predetermined region image Q on the display of the terminal device. It can be displayed. It can be displayed like an animation by continuously changing the parameters (A1 to A3 in FIG. 1B) such as the position and size of the predetermined area image.

FIG. 1C is a diagram illustrating a 3DCG image and a display method thereof. A 3DCG (Three-dimensional computer graphics) image is an image obtained (projected) of a 3D model 95 composed of a three-dimensional point cloud, a 2D mesh, a texture, or the like with a virtual camera 330. The terminal device sets the position of the virtual camera 330 and the direction of illumination, and projects the 3D model 95 onto the virtual camera 330 to generate a projected image. It can be displayed like an animation by continuously changing parameters such as the position and angle of view of the virtual camera 330.

FIG. 2 is a diagram illustrating an outline of the image display system 100 of the present embodiment. In the present embodiment, the image display system 100 that uses the spherical image 6 for the advertisement delivered by the third party distribution will be described. Third-party distribution is a mechanism for delivering advertisements to media from a server owned by an advertising agency or the like instead of a server owned by the media company. The third-party distribution server controls the distribution frequency of advertisements and measures the effectiveness of advertisements. In FIG. 2, the image distribution device 10 corresponds to a third-party distribution server.
(S1) When the terminal device 30 opens a Web page having an advertisement frame 7 (display frame for displaying an advertisement) with a browser 8 or the like, a DSP 20 (Demand-Side Platform) that wants to display an advertisement on the terminal device 30 is a terminal. The device 30 is notified. The terminal device 30 requests an advertisement from the DSP 20.
(S2) The DSP 20 requests an advertisement from the image distribution device 10.
(S3) The image distribution device 10 stores the advertisement request, generates access information 2 for the terminal device 30 to access the own device, and transmits the access information 2 to the DSP 20.
(S4) The DSP 20 transmits the received access information 2 to the terminal device 30.
(S5) The terminal device 30 requests advertisement data from the image distribution device 10 based on the access information 2.
(S6) The image distribution device 10 transmits the advertisement data (at least one of the spherical image 6 or the 3DCG image, which is referred to as content) requested in the access information 2 to the terminal device 30.
(S7) The terminal device 30 receives the content, and the browser 8 or the like operating on the terminal device 30 sequentially determines the content (a plurality of attention areas A1 to A3 in the whole celestial sphere image 6 in FIG. It is displayed as a region image Q. For example, the celestial sphere image 6 of FIG. 2 holds the angles of view of the three points of interest A1 to A3 and the points of interest A1 to A3, and the points of interest A1 (angle of view 1) even if the viewer does not operate the image. → Generates a plurality of display images that rotate in the order of attention point A2 (angle of view 2) → attention point A3 (angle of view 3).

FIG. 3 is a diagram illustrating a transition prohibition region 81 when the terminal device 30 animates the spherical image 6 or the 3DCG image. FIG. 3A shows the point of interest P of the 3DCG image, and FIG. 3B shows the point of interest P of the spherical image 6. In FIGS. 3A and 3B, points of interest P having a high degree of attention determined by using techniques such as prominence detection and object detection are shown. The image distribution device 10 creates an attention level map in this way, and generates a display pattern with the trajectory passing through the region of high attention level as the viewpoint.

On the other hand, the display pattern with the orbit passing through the area of high attention as the viewpoint may adopt a viewpoint that the viewer does not specify so much, which may impair the advertising effect. In addition, smoothness may be lost by transitioning from different viewpoints in a complicated trajectory.

Therefore, as shown in FIGS. 3 (c) and 3 (d), the person in charge sets the transition prohibition area 81. FIG. 3C shows a transition prohibited region 81 of the 3DCG image, and FIG. 3D shows a transition prohibited region 81 of the spherical image 6. The transition prohibited area 81 of the 3DCG image means a position that the virtual camera 330 cannot (or is difficult to take) to take, and the transition prohibited area 81 of the spherical image 6 is the whole sky that the virtual camera IC cannot take. It means the coordinates (line-of-sight direction) of the spherical image 6. The transition prohibited area 81 is an area where transition is prohibited. Further, the coordinates (line-of-sight direction) of the spherical image 6 which is difficult to take by the virtual camera IC may be used instead of completely prohibiting it. In this case, the transition prohibited area 81 is an area where transition is prohibited or restricted.

In FIGS. 3 (c) and 3 (d), the shaded portions of the zenith and the nadir are the transition prohibition region 81 of the position of the virtual camera 330 or the coordinates of the spherical image 6. The terminal device 30 of the present embodiment generates a display pattern so as not to enter the transition prohibition area 81.

By restricting the viewpoint in this way, it is possible to prevent the viewpoint from transitioning unnaturally. For example, it is possible to suppress the transition to a viewpoint in which the content is viewed from directly above or looked up from directly below. Further, by restricting the viewpoint after setting the element trajectory described later, it is possible to suppress the transition of different line-of-sight directions in a complicated trajectory. Therefore, the content can be displayed from an appropriate viewpoint, and it becomes easy to enhance the advertising effect.

<Terminology>
The viewpoint is information about from which direction the same content is projected when it is displayed as a flat image.

The plurality of display images generated based on the display pattern are a plurality of images in which slightly different still images are continuous, and may be referred to as, for example, a moving image or an animation.

<System configuration example>
FIG. 4 is an example of a schematic configuration diagram of the image display system 100. The image display system 100 includes a terminal device 30, an image distribution device 10, a DSP 20, an SSP (Supply Side Platform) 50, a partner site Web server 60, and an advertiser Web server 70 that can communicate via the network N. There is.

The network N is constructed by a LAN constructed in a facility or the like where the terminal device 30 is installed, a provider network of a provider that connects the LAN to the Internet, a line provided by a line operator, or the like. When the network N has a plurality of LANs, the network N is called a WAN or the Internet. The network N may be constructed either wired or wireless, and may be a combination of wired and wireless. Further, when the terminal device 30 is directly connected to the public network, it can be connected to the provider network without going through a LAN.

The terminal device 30 is an information processing device that operates as a client terminal in the present embodiment. Browser software or application software having a function equivalent thereto is operating in the terminal device 30, and the Web page requested by the terminal device 30 is received from the partner site Web server 60 and displayed on the display.

The terminal device 30 is, for example, a PC (Personal Computer), a tablet device, a smartphone, a PDA (Personal Digital Assistant), a game machine, a navigation terminal, a wearable PC, or the like, but it is sufficient if a Web page can be displayed. For example, if the printer has a function or a display for displaying a Web page, the printer may be used as the terminal device 30. In addition, digital signage may display a Web page. Digital signage is a system, display device, or information to be displayed that uses an electronic display device such as a display to transmit information in places where people can pass or exist, such as outdoors, storefronts, public spaces, and transportation facilities. Say. In this embodiment, it is assumed that the Web page includes the Web application. A Web application refers to software or a mechanism thereof that operates by coordinating a program in a programming language (for example, JavaScript (registered trademark)) that operates on a browser and a program on the Web server side, and is used on the browser. In the present embodiment, in addition to the Web application, an OS-dependent application called a native application can be operated in the same manner.

The terminal device 30 may be connected to the network N via a wired LAN or wireless LAN access point, or may be connected to the network N by an exchange line type communication such as 3G, 4G or LTE (Long Term Evolution). May be good.

The partner site Web server 60 is a server (general information processing device) that provides information and functions to a client computer (terminal device 30 of the present embodiment) used by a viewer via a network. An advertising space 7 is provided on the Web page provided by the partner site Web server 60. The partner site Web server 60 requests the SSP 50 to display an advertisement in the advertising space 7. As a result, the advertisement tag issued by the SSP 50 can be associated with the advertisement space 7.

The advertiser Web server 70 is also a server (general information processing device) that provides information and functions to the client computer (terminal device 30 of the present embodiment) used by the viewer via a network. The advertiser asks the DSP 20 to purchase the advertising space 7. Further, the image distribution device 10 that holds the advertisement data is registered in the DSP 20. In addition, the advertiser uploads the advertisement data to the image distribution device 10. In the advertisement data, in addition to the all-sky image 6 or the 3DCG image, the URL of the website (landing page) to be browsed by the viewer such as the advertiser Web server 70 is registered (URL is linked).

The advertiser Web server 70, the partner site Web server 60, and the terminal device 30 communicate with each other using, for example, the communication protocol of HTTP or HTTPs. In response to the request from the terminal device 30, the advertiser Web server 70 and the partner site Web server 60 transmit screen information to the terminal device 30. The screen information is a program described in HTML, a script language, and CSS (cascading style sheet). The structure of the Web page is mainly specified by HTML, the operation of the Web page is defined by the script language, and the Web is defined by CSS. The style of the page is specified. In the present embodiment, it is the script language that reflects the operation of the viewer on the Web page in the spherical image 6. As a script language, specifically, a programming language called JavaScript (registered trademark) or ECMAScript is known.

Both the advertiser Web server 70 and the partner site Web server 60 record a cookie in the browser software of the terminal device 30. The advertiser Web server 70 records both its own cookie and the DSP20 cookie, and the partner site Web server 60 records both its own cookie and the SSP50 cookie.

The SSP 50 is a mechanism for the partner site Web server 60 that publishes the advertising space 7 to maximize the profit by selling the advertising space 7. Although it is described as one device in the figure, it is a network of one or more information processing devices. The partner site Web server 60 requests the SSP 50 to sell the advertising space 7. Specifically, the advertisement tag issued by the SSP 50 is described in the advertisement space 7, and when the terminal device 30 displays the Web page of the partner site Web server 60, the advertisement tag requests the SSP 50 to deliver the advertisement (advertisement request). .. The SSP 50 accepts a bid for the advertising space 7 from the DSP 20 and transmits the access information 1 for accessing the successful bid DSP 20 to the terminal device 30.

The DSP 20 is a mechanism for efficiently and effectively delivering advertisements for advertisers, such as purchasing advertising space 7 and delivering advertisements. Although it is described as one device in the figure, it is a network of one or more information processing devices. That is, the DSP 20 acquires a cookie (referred to as an SSP cookie to distinguish it) from the SSP 50, and determines the attributes of the viewer based on the correspondence information between the DSP cookie and the SSP cookie managed by the DSP 20. Then, the bid is placed on the SSP50 at a price determined based on the distribution setting of the advertisement from the advertiser who requests based on the attributes of the viewer, the budget, and the like.

The successful bidder DSP 20 is requested to advertise by the terminal device 30 by the access information 1. The DSP 20 notifies the image distribution device 10 of the advertisement request, and acquires the access information 2 for the terminal device 30 to request the advertisement data from the image distribution device 10 from the image distribution device 10. The advertisement data includes a display program for displaying at least one of the spherical image 6 or the 3DCG image and an operation history program for acquiring the operation history.

The DSP 20 transmits the access information 2 to the terminal device 30. The DSP 20 charges the advertiser according to the contract. Billing is performed according to the contract between the image distribution device 10 and the advertiser. However, whether or not you will be charged just for displaying the advertisement depends on the contract.

The image distribution device 10 is one or more information processing devices that provide advertisement data including the advertisement material (all-sky image 6 or 3DCG image) that is most suitable for the attributes of the viewer for the advertisement space 7 purchased by the DSP 20. is there. The image distribution device 10 holds submitted materials such as banners and texts and advertisement images. Advertising data may be just a banner (an image made up of text, photos or pictures), or it may include a scripting language in addition to the image. In the present embodiment, a display program that rotates the spherical image 6 in the display order and reflects an operation on at least one of the spherical image 6 or the 3DCG image of the viewer on the appearance of the spherical image 6. , An operation history program for recording an operation history for at least one of the spherical image 6 or the 3DCG image is included. Both are written in a scripting language.

When the terminal device 30 requests the advertisement data from the image distribution device 10 based on the access information 2, the image distribution device 10 transmits the advertisement data to the terminal device 30. The advertising data includes the spherical image 6, and the display program automatically follows at least one of the spherical images 6 or 3DCG images of interest. Rotates and also zooms in and out. Further, the operation history program transmits the operation history for the celestial sphere image 6 to the image distribution device 10 together with the cookie of the image distribution device 10 (referred to as an image cookie for distinction) and the image ID. The image cookie is written in the browser software from the image distribution device 10. The operation history is, for example, information as to which viewing angle of view is displayed. Since this operation history is used to determine the point of interest, the operation history may not be transmitted if the operation history is not used to determine the point of interest.

Further, the operation history program transmits to the image distribution device 10 that the click is performed together with the pattern ID of the display pattern described later. The image distribution device 10 defines a plurality of display patterns for one spherical image 6, and by monitoring the click rate of each display pattern, only the display pattern having a high click rate can be distributed.

<Hardware configuration example>
Subsequently, the hardware configurations of the advertiser Web server 70, the partner site Web server 60, the image distribution device 10, the SSP 50, and the DSP 20 in the image display system of the present embodiment will be described.

The advertiser Web server 70, the partner site Web server 60, the image distribution device 10, the SSP 50 and the DSP 20 are realized by, for example, a computer system having a hardware configuration shown in FIG. FIG. 5 is a hardware configuration diagram of an example of the computer system 200 according to the present embodiment.

The computer system 200 shown in FIG. 5 includes an input device 201, a display device 202, an external I / F 203, a RAM (Random Access Memory) 204, a ROM (Read Only Memory) 205, a CPU (Central Processing Unit) 206, and a communication I /. It includes an F207, an HDD (Hard Disk Drive) 208, and the like, and each is connected to each other by a bus B.

The input device 201 includes a keyboard, a mouse, a touch panel, and the like, and is used by the operator to input each operation signal. The display device 202 includes a display and the like, and displays the processing result by the computer system 200.

The communication I / F 207 is an interface for connecting the computer system 200 to the in-house network, the Internet, and the like. As a result, the computer system 200 can perform data communication via the communication I / F 207.

HDD 208 is a non-volatile storage device that stores programs and data. The stored programs and data include, for example, an OS (Operating System), which is basic software that controls the entire computer system 200, and application software that provides various functions on the OS. The HDD 208 manages the stored program 208p and data by a predetermined file system and / or DB (database).

The external I / F 203 is an interface with an external device. The external device includes a recording medium 203a and the like. As a result, the computer system 200 can read and / or write to the recording medium 203a via the external I / F 203. The recording medium 203a includes a flexible disk, a CD (Compact Disk), a DVD (Digital Versatile Disk), an SD memory card (SD Memory card), a USB memory (Universal Serial Bus memory), and the like.

The ROM 205 is a non-volatile semiconductor memory (storage device) capable of holding programs and data even when the power is turned off. The ROM 205 stores programs and data such as a BIOS (Basic Input / Output System), an OS setting, and a network setting that are executed when the computer system 200 is started. The RAM 204 is a volatile semiconductor memory (storage device) that temporarily holds programs and data.

The CPU 206 is an arithmetic unit that realizes control and functions of the entire computer system 200 by reading a program or data from a storage device such as the ROM 205 or the HDD 208 onto the RAM 204 and executing processing.

Although each server may support cloud computing, it may be a so-called single information processing device. Cloud computing refers to a usage pattern in which resources on a network are used without being aware of specific hardware resources.

<< Terminal device >>
FIG. 6 is a hardware configuration diagram of an example of the terminal device 30. The terminal device 30 in FIG. 6 is assumed to be a tablet device, a smartphone, or the like. The terminal device 30 includes a CPU 601, a ROM 602, a RAM 603, an EEPROM 604, a CMOS sensor 605, an acceleration / orientation sensor 606, and a media drive 608.

The CPU 601 controls the overall operation of the terminal device 30. The ROM 602 stores a basic input / output program. The RAM 603 is used as a work area for the CPU 601. The EEPROM 604 reads or writes data under the control of the CPU 601. The CMOS sensor 605 takes an image of the subject under the control of the CPU 601 and obtains image data. The acceleration / azimuth sensor 606 is an electronic magnetic compass, a gyro compass, an acceleration sensor, etc. that detect the geomagnetism.

The media drive 608 controls reading or writing (storage) of data to the media 607 such as a flash memory. The media drive 608 has a structure in which the media 607 from which already recorded data is read out or new data is newly written and stored is detachable.

The program 604p executed by the CPU 601 is stored in the EEPROM 604. The program 604p is application software, an OS, or the like for executing various processes in the embodiment. The program 604p may be distributed in a state of being stored in the media 607 or the like, or may be distributed from a server for program distribution.

Further, the CMOS sensor 605 is a charge coupling element that converts light into electric charges and digitizes an image of a subject. The CMOS sensor 605 may be, for example, a CCD (Charge Coupled Device) sensor as long as it can image a subject. The CMOS sensor 605 can read barcodes and two-dimensional barcodes.

Further, the terminal device 30 includes an RF tag reader / writer 622, an antenna I / F 623, and a vibration actuator 624. The RF tag reader / writer 622 communicates according to a standard such as NFC (Near Field Communication).

The vibration actuator 624 is a motor that vibrates the terminal device 30. For example, when an incoming call or an e-mail is received, it vibrates to notify the viewer to that effect.

Further, the terminal device 30 includes a voice input unit 609, a voice output unit 610, an antenna 611, a communication unit 612, a wireless LAN communication unit 613, a short-range wireless communication antenna 614, a short-range wireless communication unit 615, a display 616, a touch panel 617, and the like. It is equipped with a bus line 619.

The voice input unit 609 converts the voice into a voice signal. The voice output unit 610 converts a voice signal into voice. The communication unit 612 uses the antenna 611 to communicate with the nearest base station device by a wireless communication signal. The wireless LAN communication unit 613 performs wireless LAN communication conforming to the IEEE802.11 standard.

The short-range wireless communication unit 615 is a communication device that uses the short-range wireless communication antenna 614 and conforms to, for example, Bluetooth (registered trademark) or Bluetooth Low Energy (registered trademark) communication standard.

The display 616 is a liquid crystal, an organic EL, or the like that displays an image of a subject, various icons, and the like. The touch panel 617 is mounted on the display 616 and is composed of a pressure-sensitive or electrostatic panel, and detects a touch position on the display 616 by touching with a finger, a touch pen, or the like. The bus line 619 is an address bus, a data bus, or the like for electrically connecting each of the above parts.

In addition, the terminal device 30 includes a dedicated battery 618 and can be driven by either the battery 618 or a commercial power source. The voice input unit 609 includes a microphone for inputting voice. The audio output unit 610 includes a speaker that outputs audio.

<About spherical images>
The omnidirectional image 6 captured by the omnidirectional camera 9 will be described with reference to FIGS. 7 to 11. FIG. 7 is a usage image diagram of the spherical camera 9. As shown in FIG. 7, the omnidirectional camera 9 is an imaging device that the user holds in his / her hand to image a subject around the user. The omnidirectional camera 9 has a structure in which the back surfaces of the two image pickup elements are opposed to each other, and two hemispherical images are obtained by imaging a subject around the user.

Next, the outline of the process from the image captured by the spherical camera 9 to the creation of the spherical image 6 will be described with reference to FIGS. 8 and 9. 8 (a) is a hemispherical image (front side) captured by the spherical camera 9, FIG. 8 (b) is a hemispherical image (rear side) captured by the spherical camera 9, FIG. 8 (c). Is a diagram showing an image represented by the equirectangular projection (rectangular projection image). FIG. 9A is a conceptual diagram showing a state of covering a sphere in an equirectangular cylindrical projection image, and FIG. 9B is a diagram showing a spherical image 6 of the whole celestial sphere.

As shown in FIG. 8A, the image obtained by the spherical camera 9 is a hemispherical image (front side) curved by a fisheye lens. Further, as shown in FIG. 8B, the image obtained by the spherical camera 9 is a hemispherical image (rear side) curved by a fisheye lens. Then, the hemispherical image (front side) and the hemispherical image (rear side) inverted 180 degrees are combined by the spherical camera 9, and as shown in FIG. 8 (c), the equirectangular projection image. Is created. This is a spherical image.

Then, by using OpenGL ES (Open Graphics Library for Embedded Systems), as shown in FIG. 9A, the equirectangular projection image is pasted so as to cover the spherical surface, and FIG. 9 (A) The spherical image 6 as shown in b) is displayed. As described above, the spherical image 6 is represented as an image in which the equirectangular projection image faces the center of the sphere. OpenGL ES is a graphics library used to visualize 2D (2-Dimensions) and 3D (3-Dimensions) data. The spherical image 6 may be a still image or a moving image.

The advertisement data distributed by the image distribution device 10 includes the spherical image 6. Since the spherical image 6 is an image that is pasted and displayed so as to cover the spherical surface, it is curved and gives a sense of discomfort to humans. Therefore, the terminal device 30 displays a part of the predetermined region T of the spherical image 6 as a flat image with little curvature so as not to give a sense of discomfort to humans. The predetermined area T is indicated by the coordinates (X, Y, Z) in the three-dimensional virtual space. On the other hand, since the display 616 is a two-dimensional plane, the terminal device 30 cannot display the predetermined area T as it is. Therefore, the terminal device 30 obtains a predetermined region T by a perspective projection conversion that projects a three-dimensional object onto a two-dimensional plane using a technique of 3D computer graphics.

FIG. 10 is an example of a diagram for explaining the line of sight of the viewer. Since the spherical image 6 has three-dimensional coordinates, the line-of-sight direction is specified by the three-dimensional coordinates or information that specifies the coordinates of the sphere such as latitude and longitude. In the present embodiment, the center point CP of the predetermined region T is the line-of-sight direction. The center point CP of the predetermined region T is specified by (θ, φ) in the spherical coordinate system. θ represents longitude and φ represents latitude.

The viewer can change the line-of-sight direction by operation, but assuming that the virtual camera IC does not translate, the virtual camera IC will be a rigid body with roll (rotation around the Z axis) and yaw (centered on the Y axis). Three rotations are possible: rotation) and pitch (rotation about the X-axis). The line-of-sight direction changes regardless of which of these three rotations occurs. For example, when the viewer rotates the spherical image 6 in the horizontal direction, the yaw angle changes, and when the viewer rotates the spherical image 6 in the vertical direction, the pitch angle changes, and the spherical image 6 is rotated around the center of the display 616. And the roll angle changes. In the present embodiment, the operation of the viewer on the Web page is reflected in the line-of-sight direction (roll angle, yaw angle, pitch angle) and the like. How it is reflected shall be described in advance in the display program. The predetermined area T can be enlarged or reduced.

The viewer can change the line-of-sight direction (rotate the image data) by flicking the ad frame 7 up / down / left / right.

The relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. Note that FIG. 11 is a diagram showing an example of the relationship between the predetermined area information and the image of the predetermined area T. As shown in FIG. 11, rH indicates Horizontal Radian (longitude θ), rV indicates Vertical Radian (latitude φ), and α indicates the angle of view (Angle). That is, the posture of the virtual camera IC is changed so that the gazing point of the virtual camera IC indicated by the imaging direction (θ, φ) becomes the center point CP of the predetermined region T which is the imaging region of the virtual camera IC. Become. The predetermined region image Q is an image of the predetermined region T in the spherical image 6. f is the distance from the virtual camera IC to the center point CP. L is the distance between an arbitrary vertex of the predetermined region T and the center point CP (2L is a diagonal line). Then, in FIG. 11, the trigonometric function generally represented by the following equation (A) holds.

Lf = tan (α / 2) ・ ・ ・ (A)
<Display of projected image using 3D model>
Subsequently, the 3DCG image will be described. Since the appearance of the 3DCG image changes (the viewpoint changes) depending on the operation of the viewer, interactive advertising can be realized.

<About 3DCG>
3DCG is a method of creating an image with a sense of depth (three-dimensional effect) by converting a virtual three-dimensional object in a three-dimensional space into information on a two-dimensional plane by computer calculation. 3DCG is created by processes such as modeling, world transformation, view transformation, projective transformation, and lighting. Modeling and world conversion are performed on dedicated software, and view conversion, projective conversion, and lighting are executed by the display program.

Modeling means that the creator defines the shape of an object in the local coordinate system, for example, creating a shape with polygons. World transformation refers to transforming the coordinates of an object into a world coordinate system that represents the three-dimensional space to be drawn (translation and rotation) (arranging an object in the world coordinate system). View conversion is equivalent to determining the viewpoint of the viewer in the world coordinate system, and is a process of setting the coordinates of the camera, the gazing point of the camera, and the upward direction of the camera, and setting the object to be viewed from a specific direction. .. The viewpoint may be determined by the operation of the viewer, or may be changed automatically. Homography is the projection of an object in three-dimensional space onto a two-dimensional plane. Lighting refers to the process of giving light and darkness according to the position of the light source.

FIG. 12 is a diagram schematically showing a 3D model and a projective transformation. In FIG. 12A, a virtual camera 330 whose optical axis is directed to the 3D model 95 is arranged. The 3D model 95 of FIG. 12A is a car, but it is only an example. The virtual camera 330 can change the angle of the 3D model 95 in the horizontal direction and the vertical direction by 360 degrees. The viewer can browse the 3D model 95 from any direction. Alternatively, the 3D model 95 may be rotated on the turntable 96 while the virtual camera 330 is fixed.

FIG. 12B schematically shows how the 3D model is projected and converted into the virtual camera 330. A 3D model 95 created by polygons and a group of three-dimensional points is projected on a plane. An image projected on a plane is called a projected image. By changing the position of the virtual camera 330 (view conversion), the 3D model 95 can be projected according to the position of the camera. Since the position of the virtual camera 330 can be changed to an arbitrary position of 360 degrees around the 3D model 95, the viewer can view the 3D model 95 from various angles within the limited ad space 7.

FIG. 13 is an example of the 3D model 95 displayed from different viewpoints. Projected images of the 3D model 95 taken from various angles as shown in FIGS. 13 (a) to 13 (c) are shown in FIGS. 13 (d) to 13 (f). Such projected images having different viewpoints are displayed one after another in the advertising space 7 according to the display pattern. In addition, the viewer can rotate the 3D model 95 displayed in the advertising space 7 by his / her own operation, and can browse one 3D model 95 from various angles with one advertising space 7.

<Information used by web pages>
FIG. 14 is an example of a diagram schematically explaining information used by a Web page. FIG. 14A is a Web page provided by the partner site Web server 60. The Web page provided by the partner site Web server 60 has one or more advertising space 7. Further, the browser of the terminal device 30 holds the SSP cookie and the DSP cookie in advance. The URL of the SSP 50, the advertising space ID, and the like are associated with the advertising space 7. The browser of the terminal device 30 (advertising space detection unit 36 described later) transmits the SSP cookie and the request content to the SSP 50. Further, the access information 1 including the URL of the DSP 20 that has made a successful bid for the advertising space 7 and the successful bid ID are transmitted from the SSP 50 to the terminal device 30.

FIG. 14B is a diagram illustrating an operation based on the access information 1. The terminal device 30 transmits the winning bid ID and the DSP cookie to the DSP 20 based on the URL of the DSP 20. The DSP 20 identifies the advertisement request by the winning bid ID, and acquires the access information 2 including the URL of the image distribution device 10 and the advertisement opportunity ID from the image distribution device 10. The terminal device 30 acquires the access information 2 and the advertisement opportunity ID from the DSP 20.

FIG. 14C is a diagram illustrating an operation based on the access information 2. The terminal device 30 transmits the advertisement opportunity ID to the image distribution device 10 based on the URL of the image distribution device 10. The image distribution device 10 generates advertisement data based on an advertisement request or the like specified by an advertisement opportunity ID, and transmits the advertisement data together with an image cookie, an advertisement space ID, and a display pattern to the terminal device 30. In this way, the spherical image 6 is displayed in the advertising space 7 of the Web page.

<About functions>
FIG. 15 is an example of a functional block diagram showing the functions of the terminal device 30, the advertiser Web server 70, the partner site Web server 60, the image distribution device 10, the SSP 50, and the DSP 20 in a block shape.

<< Advertiser Web Server, Partner Site Web Server >>
The advertiser Web server 70 and the partner site Web server 60 have a Web page providing unit 71. Each of these functions possessed by the advertiser Web server 70 and the partner site Web server 60 is a function or means realized by the CPU 206 executing the program 208p expanded from the HDD 208 to the RAM 204.

The Web page providing unit 71 performs, for example, general HTTP communication, constructs a Web page in response to a request for a Web page from the terminal device 30, and transmits the Web page to the terminal device 30. If necessary for processing, the application server is requested to process, and the processing result is arranged on the Web page and transmitted.

The Web page of the advertiser Web server 70 includes an advertiser cookie and a DSP cookie. This is because the advertiser Web server 70 requests the DSP 20 to deliver the advertisement. This enables retargeting advertisements and the like. The Web page of the partner site Web server 60 includes the partner site cookie and the SSP cookie. This is because the partner site Web server 60 requests the SSP 50 to sell the advertising space 7.

<< SSP >>
The SSP 50 has an advertisement request unit 51, an advertisement request acquisition unit 52, an access information transmission unit 53, and a successful bid unit 54. Each of these functions possessed by the SSP 50 is a function or means realized by the CPU 206 executing the program 208p expanded from the HDD 208 to the RAM 204.

The advertisement request acquisition unit 52 acquires the advertisement request requested together with the SSP cookie by executing the advertisement tag associated with the advertisement space 7 by the terminal device 30. The advertisement request acquisition unit 52 sends an advertisement request including an SSP cookie to the advertisement request unit 51.

The advertisement request unit 51 transmits an advertisement request to the DSP 20. The advertisement request includes an SSP cookie, a domain of the partner site Web server 60, an advertisement space ID, an advertisement space 7 size, an advertisement format, a browser type, an OS type, and the like.

The successful bidder 54 accepts bids from the DSP 20 and basically conducts an auction to sell the ad space 7 to the highest bid price DSP 20 (depending on the advertiser's product or service, the bid is not accepted). The successful bidder 54 generates a successful bid ID and notifies the DSP 20.

The access information transmission unit 53 generates access information 1 for requesting an advertisement from the successful bidder DSP 20 and transmits it to the terminal device 30 together with the successful bid ID. The access information 1 includes the URL (IP address) of the DSP 20.

<< DSP >>
The DSP 20 has a request reception unit 21, a bid determination unit 22, a bid unit 23, an advertisement request reception unit 24, and an image request unit 25. Each of these functions possessed by the DSP 20 is a function or means realized by the CPU 206 executing the program 208p expanded from the HDD 208 to the RAM 204.

Further, a cookie information DB 291 and a distribution setting DB 292 are constructed in the storage unit 29. The storage unit 29 is realized by the HDD 208 or the RAM 204 shown in FIG.

Table 1 schematically shows the information stored in the cookie information DB 291. The cookie information DB 291 associates the DSP cookie with the SSP cookie, and the visited domain visited by the viewer is registered. The association between the DSP cookie and the SSP cookie is possible by a technique called cookie sync. In addition, the visit domain is obtained by a visitor visiting a website to which the DSP 20 has a tag (behavior monitoring tag) attached. Therefore, the DSP 20 can identify the DSP cookie from the SSP cookie, and can further determine what kind of website the viewer is interested in.

Table 2 schematically shows the information stored in the distribution setting DB 292. Advertiser ID, advertisement target attribute (favorable target person attribute), and non-advertisement target attribute (undesirable target person attribute) are registered in the distribution setting DB 292. The advertiser ID is identification information for identifying or identifying the advertiser who is the provider of the wide-angle image. Therefore, the DSP 20 or the image distribution device 10 determines whether or not to transmit the wide-angle image by referring to the attributes of the preferred target person or the unfavorable target person set for the provider of the wide-angle image. Can be done.

ID is an abbreviation for Identification and means an identifier or identification information. An ID refers to a name, a code, a character string, a numerical value, or a combination of one or more of these, which is used to uniquely distinguish a specific object from a plurality of objects. The same applies to the following IDs. The advertising target attribute is the attribute of the viewer who the advertiser wants to deliver the advertisement, and the non-advertising target attribute is the attribute of the viewer who the advertiser does not want to deliver the advertisement. The DSP 20 compares the attribute of the viewer determined from the cookie information DB 291 and the information of the distribution setting DB 292, quantifies the attribute of the viewer, and determines the bid amount. In addition, various information such as the time zone, the day of the week, and the area of the viewer may be taken into consideration when determining the bid amount.

(DSP function)
The request reception unit 21 receives an advertisement request from the SSP 50. From the request information, basic information such as the advertising space ID of the terminal device 30 can be obtained.

The bid determination unit 22 determines the bid amount by evaluating the advertisement request received by the request reception unit 21 with reference to the cookie information DB 291 and the distribution setting DB 292 (the bid amount may not be bid). As described above, the DSP cookie is identified from the SSP cookie, and the bid amount is determined based on how much the attribute information stored in association with the DSP cookie matches the advertiser's request stored in the distribution setting DB 292. To do.

The bidding unit 23 bids on the SSP 50 at the determined bid amount. If a successful bid is successful, the successful bid ID is acquired and associated with the advertisement request.

The advertisement request receiving unit 24 acquires an advertisement request (successful bid ID, DSP cookie) from the terminal device 30 based on the access information 1. The advertisement request reception unit 24 identifies the advertisement request by the winning bid ID. The DSP cookie does not have to be present, and is used when the SSP cookie and the DSP cookie are not associated with each other.

The image requesting unit 25 requests the image distribution device 10 for access information 2 together with the DSP cookie, the advertiser ID, and the request contents. The advertiser is identified by the advertiser ID. The ad space 7 is specified by the ad space ID of the request content. Further, the attributes of the viewer may be transmitted to the image distribution device 10. As a result, the image distribution device 10 can distribute the spherical image 6 suitable for the viewer.

<< Terminal device >>
The terminal device 30 has a Web page acquisition unit 31, a Web page analysis unit 32, a Web page display unit 33, an operation reception unit 34, and a script execution unit 35. Each of these functions possessed by the terminal device 30 is a function or means realized by the CPU 601 executing the program 604p developed from the EEPROM 604 to the RAM 603.

The operation reception unit 34 receives various operations on the terminal device 30. Specifically, it accepts operations on the browser executed by the terminal device 30. The operation reception unit 34 receives an operation for a Web page. The image operation receiving unit 43 accepts the operation on the image data based on the script.

The Web page acquisition unit 31 communicates with the advertiser Web server 70 and the partner site Web server 60 by the operation of the viewer or the operation of the script, and acquires the Web page from the advertiser Web server 70 and the partner site Web server 60. ..

The Web page analysis unit 32 reads HTML included in the screen information in order from the beginning and analyzes the structure of sentences and image data included in the HTML. In addition, the association with the HTML text or image data described in CSS is detected, and the style of the text or image data included in the HTML is associated. Further, the Web page analysis unit 32 detects a script tag from HTML and extracts a script written in the script language. The Web page analysis unit 32 sends HTML and CSS to the Web page display unit 33, and sends a script to the script execution unit 35.

The Web page display unit 33 displays the Web pages on the display 616 in the order in which the analysis is completed from the beginning of the HTML. In addition, the Web page is updated according to the operation on the Web page.

The script execution unit 35 executes the script extracted by the Web page analysis unit 32. The specific content of the script varies depending on the Web page. In the present embodiment, as a script, an advertisement tag associated with the advertisement space 7, access information 1 acquired from the SSP 50, access information 2 acquired from the DSP 20, and advertisement data distributed from the image distribution device 10 are detected. The script execution unit 35 is realized by the CPU 601 shown in FIG. 6 executing the program 604p or the like. Further, each function or means possessed by the script execution unit 35 is realized by executing the script by the CPU 601.

The advertisement space detection unit 36 is a function realized by the terminal device 30 executing an advertisement tag associated with the advertisement space 7 included in the screen information acquired by the Web page acquisition unit 31. The ad space detection unit 36 transmits an advertisement request to the SSP 50 together with the SSP cookie based on the URL associated with the ad space 7.

The access information acquisition unit 37 acquires the access information 1 from the SSP 50. Access information 1 is described by a script. The advertisement requesting unit 38 accesses the DSP 20 based on the URL included in the access information 1 and requests an advertisement together with the winning bid ID and the DSP cookie. The advertisement request unit 38 acquires the access information 2 and the advertisement opportunity ID in response to this request. Access information 2 is also described in a script.

The advertisement acquisition unit 39 specifies an advertisement opportunity ID based on the URL of the image distribution device 10 included in the access information 2, and requests the image distribution device 10 for advertisement data. Since the image distribution device 10 generates the advertisement data, the advertisement acquisition unit 39 acquires the advertisement data. The advertising data of the present embodiment includes contents (omnidirectional image 6 and / or 3DCG image), a display pattern, a display program, and an operation history program. The operation history program causes the image distribution device 10 to transmit the operation history. The display program rotates the all-sky image 6 or the 3DCG image with the display pattern, and reflects the operation on the all-sky image 6 or the 3DCG image on the appearance of the all-sky image 6 or the 3DCG image. The operation history program and the display program are also written in the script language and are executed by the script execution unit 35.

The advertisement display unit 40 displays the advertisement data acquired from the image distribution device 10. The advertisement display unit 40 is mainly realized by a display program, displays a predetermined region image generated from the all-sky image 6 or a projected image generated from a 3DCG image on the advertisement frame 7, and further automatically rotates the advertisement display unit 40.

The display image generation unit 44 generates an image (predetermined area image or projected image) to be displayed in the advertising space 7 from the content with the display pattern acquired from the image distribution device 10. That is, a plurality of display images (animations) to be displayed in the advertising space 7 are generated from one content by rotating the spherical image 6 or changing the position of the virtual camera 330 in the 3DCG image one after another. To do. The advertisement display unit 40 displays a plurality of display images generated by the display image generation unit 44.

The image operation reception unit 43 has the coordinates of a pointing device (a finger when the display 616 is a touch panel, a rod-shaped member or a dedicated pen and a touch panel, a mouse pointer when a mouse is used, and a mouse pointer when a trackball is used). When it is included in the ad space 7, the operation on the image data is accepted with priority over the operation on the Web page. As a result, when the viewer flicks or swipes on the ad space 7, the image data can be changed instead of the Web page.

By acquiring the advertisement data from the image distribution device 10, the terminal device 30 can acquire the image cookie of the image distribution device 10 and store it in the storage unit 49. When the advertisement acquisition unit 39 transmits the image cookie to the image distribution device 10, the image distribution device 10 can associate the DSP cookie with the image cookie.

The operation history recording unit 41 is mainly realized by an operation history program, and records operation information for the spherical image 6 or 3DCG as an operation history. Details of the operation history will be described in the image distribution device 10. The operation history transmission unit 42 is realized mainly by the operation history program, and displays the operation history for the spherical image 6 or 3DCG displayed in the advertising space 7 together with the image cookie and the image ID (identifying the spherical image 6). It is transmitted to the image distribution device 10. Further, when the image is clicked, the image distribution device 10 is further transmitted to that effect together with the display pattern.

<< Image distribution device >>
The image distribution device 10 has an image information response unit 11, an advertisement distribution unit 12, a point of interest determination unit 13, an operation history acquisition unit 14, a content reception unit 15, a display pattern generation unit 16, and a constraint unit 17. .. Each of these functions of the image distribution device 10 is a function or means realized by the CPU 206 executing the program 208p expanded from the HDD 208 to the RAM 204.

Further, the image distribution device 10 has a storage unit 19 realized by the HDD 208 or the RAM 204 shown in FIG. A distribution history DB 191, an operation history DB 192, a cookie-compatible DB 193, an advertisement image DB 194, a pattern DB 195, and a display program 196 are constructed in the storage unit 19.

Table 3 schematically shows the information stored in the operation history DB 192. The operation history DB 192 records a history of what kind of operation each viewer has performed on the spherical image 6. Each item of the image ID, the viewing time, the image cookie, and the viewing angle of view 1 to 3 is recorded in the operation history DB 192. The image ID is information for identifying the spherical image 6. The image cookie serves as information for determining the identity of the viewer who operates the terminal device 30 or the terminal device 30 (an example of device identification information). The viewing angles of view 1 to 3 are the angles of view when the viewer views the spherical image 6. The viewing angle of view 1 to 3 stores the viewing time, the presence / absence of enlargement, the angle of view after enlargement when enlarged, and the presence / absence of reduction. In this way, the angle of view considered to be of interest to the viewer is recorded for each image. The viewing angles of view 1 to 3 are angles of view in which the viewer stands still for a predetermined time (for example, 1 second) or more without rotating the spherical image 6. The operation history recording unit 41 of the terminal device 30 records the top three angles of view when it is stationary. It may be only one or four or more. Similarly, the operation history recording unit 41 records whether or not the viewer has expanded or contracted at each viewing angle of view 1 to 3. The image distribution device 10 may set such an operation history as a point of interest P. Further, if the image ID is associated with the image cookie, the retargeting advertisement becomes possible. The angle of view at the time of clicking may be recorded as the operation history.

Table 4 schematically shows the information stored in the advertisement image DB 194. Information about the spherical image 6 or 3DCG to be advertised is registered in the advertisement image DB 194. In the advertisement image DB 194, an advertiser ID, an image ID, a priority, and a target attribute are registered in association with each other. Since the advertiser is determined by the DSP 20, among the images that the advertiser wants to distribute, the image to be distributed is determined based on at least one of the degree of agreement or the priority of the attribute of the viewer and the target attribute in Table 4. At the time of distribution, the initial position of the spherical image 6 is specified (if the display pattern is also distributed, the display pattern has priority). At least one of the spherical image 6 and the 3DCG image is associated with the image ID. In the case of the retargeting advertisement, the image associated with the image cookie is delivered in the operation history DB 192.

Table 5 schematically shows the information stored in the cookie-compatible DB193. A DSP cookie and an image cookie are associated with the cookie-compatible DB 193. The DSP cookie is notified from the DSP 20, and the image cookie is notified from the terminal device 30. By associating the two, the image distribution device 10 can identify the viewer only by the DSP cookie.

Table 6 schematically shows the information registered in the pattern DB 195. In the pattern DB 195, a display pattern for displaying the attention point P of the spherical image 6 or the 3DCG image is registered. A pattern ID is associated with the image ID, and a display pattern is associated with the pattern ID. The display pattern is one in which one or more (preferably a plurality) parameters are set in time series. One parameter means the position of the virtual camera 330 (ie, the viewer's point of view) in the central point CP or 3DCG image of the spherical image 6. Each parameter is determined not to include the transition prohibition region 81 constrained by the constraint unit 17. Details of the parameters will be described later. The number of clicks is the number of clicks by the viewer for each display pattern. That is, the image distribution device 10 displays the same celestial sphere image 6 or 3DCG image on a plurality of terminal devices 30 with the same display pattern, and records that the terminal device 30 clicks. By doing so, it is possible to gradually avoid distribution by a display pattern with a low click rate. It is preferable to record the number of clicks for each viewer's attributes (gender, age, family structure, etc.). As a result, it is possible to display a display pattern in which the number of clicks is large with respect to the attributes of the viewer.

Table 7 schematically shows the information registered in the distribution history DB 191. The image ID and display pattern of the spherical image 6 or 3DCG image distributed in association with the image cookie are registered in the distribution history DB 191. If the image cookie is registered in the terminal device 30 when the image distribution device 10 transmits the advertisement data, the image distribution device 10 can acquire the image cookie from the terminal device 30. The image distribution device 10 can avoid distributing the same spherical image 6 or 3DCG image to the same viewer, or can avoid distributing the same spherical image 6 or 3DCG image in the same display pattern. Conversely, retargeting distribution can also be performed.

(About the function of the image distribution device)
The image information response unit 11 assigns an advertising opportunity ID to the DSP cookie, the advertiser ID, and the request content (mainly the advertising space ID) acquired from the DSP 20, and returns the advertising opportunity ID together with the access information 2 to the DSP 20. The advertisement distribution opportunity can be specified for each advertisement space 7 of the terminal device 30 by the advertisement opportunity ID.

The advertisement distribution unit 12 determines the spherical image 6 and the display pattern to be distributed when the terminal device 30 requests the advertisement data together with the advertisement opportunity ID. First, the spherical image 6 or the 3DCG image associated with the advertiser ID acquired from the DSP 20 is determined from the advertisement image DB 194. Preferably, the attributes of the viewer are considered. The attribute of the viewer may be notified from the DSP 20, or may be stored by the image distribution device 10 in association with the image cookie. Next, the advertisement distribution unit 12 determines the display pattern of the spherical image 6 with reference to the pattern DB 195. For example, the pattern with the largest number of clicks is determined, or the pattern with the number of clicks equal to or greater than the threshold value is arbitrarily determined. It is preferable that the attributes of the viewer are taken into consideration when determining the pattern.

The operation history acquisition unit 14 acquires the operation history together with the image cookie and the image ID from the terminal device 30 and sets it in the operation history DB 192. There are two types of operation history: an operation history for determining a point of interest and an operation history (clicked) for updating the number of clicks.

The image distribution device 10 may have an operation history recording unit 41 of the terminal device 30. Since the operation history recording unit 41 requires time-series operation contents, it may be difficult for the image distribution device 10 to acquire the operation history depending on the communication band or the like. On the other hand, the operation history recording unit 41 of the terminal device 30 only needs to transmit the final operation history to the image distribution device 10, so that the communication load can be reduced.

The attention point determination unit 13 determines the attention point P of the spherical image 6. Various methods for determining the point of interest P, such as estimating the point of interest by image processing, can be considered, but the details will be described later. The attention point P determined by the attention point determination unit 13 is used for the display pattern.

The content receiving unit 15 receives the content for distribution by the image distribution device 10 (that is, accepts the submission of advertisement). Further, the content reception unit 15 accepts the setting of the transition prohibition area 81 by the person in charge. The method of receiving the image by the content receiving unit 15 will be described with reference to FIG.

The display pattern generation unit 16 generates a display pattern that passes through the attention point P determined by the attention point determination unit 13 under the constraint of the restriction unit 17. Details will be described later.

<Reception of content>
FIG. 16 is an example of a functional block diagram showing the functions of the content receiving unit 15 and the image registration terminal 90 in a block shape. The content reception unit 15 has a communication unit 151, a screen information creation unit 152, and an image storage unit 153.

The communication unit 151 communicates with the image registration terminal 90 using, for example, the communication protocol of HTTP or HTTPs. In this embodiment, the screen information of the content registration screen for registering the content in response to the request from the image registration terminal 90 is transmitted to the image registration terminal 90. In addition, the content is received from the image registration terminal 90.

The screen information creation unit 152 creates screen information for the content registration screen shown in FIG. The image storage unit 153 stores the content received by the communication unit 151 in the advertisement image DB 194.

The image registration terminal 90 is, for example, a terminal operated by a person in charge of an advertiser or an agency (hereinafter, simply referred to as a person in charge), and has at least one of the whole celestial sphere image 6 or the 3DCG image shown in FIG. 3 as content. It is used to register with the distribution device 10. The image registration terminal 90 has a communication unit 91, a display control unit 92, and an operation reception unit 93. The hardware configuration of the image registration terminal 90 may be the same as that shown in FIG.

The communication unit 91 receives the screen information of the content registration screen from the content reception unit 15, and transmits the content input by the person in charge to the content reception unit 15.

The display control unit 92 displays the screen information of the content registration screen received by the communication unit 91 on the display of the image registration terminal 90. The operation reception unit 93 accepts operations (omnidirectional images 6, 3DCG, transition prohibited area 81, etc.) of the person in charge on the content registration screen.

FIG. 17 shows an example of the content registration screen 220 displayed on the display of the image registration terminal 90. As shown in FIG. 17A, the content registration screen 220 has a work area 221, a file selection field 222, a numerical setting field 223, a cancel button 224, and a new creation button 225.

The file selection field 222 is a field for the person in charge to select and set a file of the spherical image 6 or the 3DCG image. The spherical image 6 or 3DCG image selected in the file selection field 222 is displayed in the work area 221.

The spherical image 6 displayed in the work area 221 is displayed as, for example, an equirectangular projection image. The person in charge can set the transition prohibition area 81 with a pointing device such as a mouse or a finger. The transition prohibited area 81 of the spherical image 6 is set to a range in which the center point CP in the line-of-sight direction is not or is difficult to be taken. In FIG. 17A, a region where the absolute value of latitude is equal to or higher than a predetermined value is set as a transition prohibition region 81. The person in charge can set any area as the transition prohibition area 81.

FIG. 17B shows an example of the content registration screen 220 in which the 3DCG image is displayed in the work area 221. A 3DCG image is displayed in the work area 221. The person in charge sets the transition prohibition area 81 with a pointing device such as a mouse or a finger. The transition prohibited area 81 of the 3DCG image is set to a range of positions on the sphere that the virtual camera 330 cannot or cannot take. In FIG. 17B, a region where the absolute value of latitude is equal to or higher than a predetermined value is set as a transition prohibition region 81. The person in charge can set any area as the transition prohibition area 81.

In the present embodiment, for convenience, the optical axis of the virtual camera 330 is oriented toward the center of the sphere, but the direction of the optical axis can be set to any direction.

Further, instead of visually setting the transition prohibition area 81 in the setting field 223 numerically, the transition prohibition area 81 can be set numerically. The person in charge specifies the range in the axial direction numerically, for example, "| zt | <a", or sets other parameters such as the speed of parameter change and the angle of view regarding the transition trajectory described later. In addition to the numerical values, the user may select the transition prohibited area in the work area with a mouse, a stylus pen, or the like.

<Determination of points of interest>
Subsequently, a method of determining a point of interest will be described by several methods.
<< How to determine the point of interest from the features >>
FIG. 18 is an example of a flowchart showing a procedure in which the attention point determination unit 13 determines the attention point. In step S11, the point of interest determination unit 13 defines a regular polyhedron having a center common to the unit sphere, and then performs perspective projection conversion with the normal direction of each surface as the line-of-sight direction to obtain a partial image. FIG. 19 (a) shows an example in which a regular octahedron is defined as a projection plane of a spherical image, and FIG. 19 (b) shows an example in which a regular dodecahedron is defined as a projection plane of a spherical image.

In step S12, the point of interest determination unit 13 extracts a predetermined feature amount from each partial image obtained in step S11. Since the input image is divided by the above method and the feature amount is calculated from the partial image with less distortion, it is possible to robustly process a wide-angle image exceeding 180 degrees. As the feature amount, color, edge, prominence, object position / label, and the like can be used.

In the following step S13, the point of interest determination unit 13 calculates the importance for each position (pixel by pixel) of the input image from the feature amount extracted from each partial image using a predetermined regression model. Details are omitted, but the importance and features of each position are expressed by regression as a function. It should be noted that the importance g, which is the output (teacher data), is determined in advance using the training data by an appropriate method, and the relationship between the feature amount and the importance g is obtained.

In the following step S14, the likelihood distribution of the attention points is calculated based on the importance distribution calculated in step S13 under the design concept that the attention point determination unit 13 has the user's attention points in the direction of high importance. calculate. For example, after defining a region R passing through the viewpoint on the unit image plane, an added value obtained by adding the importance of each position in the region R can be calculated as the likelihood of the region of interest of the viewpoint.

In the following step S15, the attention point determination unit 13 calculates the attention point based on the likelihood region likelihood distribution. In the present embodiment, for example, the position corresponding to the imaging direction corresponding to the maximum likelihood value of the likelihood region of interest is calculated as the point of interest.

In this way, some points of interest can be calculated. The attention point determination unit 13 identifies a predetermined number of attention points or attention points equal to or larger than the threshold value among the top N attention points. Then, a point of interest is sent to the display pattern generation unit 16.

<< Determine the person as the point of interest >>
It is also possible to determine the point of interest in consideration of the taste of the viewer. A person can be considered as a subject that a person pays attention to. Face recognition is known as a method for detecting a person. Various machine learning such as deep learning may be adopted for face recognition, or pattern matching may be adopted. The attention point determination unit 13 detects the center of the face as the attention point. When the number of faces is large, the number of faces may be reduced to a fixed number by regarding the faces that are close to each other as one.

<< Determine a smile as a point of interest >>
A person's smile can be considered as a subject that people pay attention to. As a smile detection method, in addition to the above-mentioned face recognition, a known method of pattern matching using a smile pattern held in advance can be used. If you have a large number of faces, you may want to focus only on your smile. In addition to the face, faces such as sadness, anger, and confusion may be determined as points of interest.

<< Determine children as points of interest >>
When a child is present in an adult, one may pay attention to the child. Or, conversely, when an adult is present in a large number of children, a person may pay attention to the adult. The position of the face can be used as a method for detecting a child or an adult. If the position of the face specified by the above face recognition is lower than the position of other faces, it is presumed to be a child, and if the position of the face specified by face recognition is higher than the position of other faces, it is an adult. It is estimated to be.

<< Determine any subject as the point of interest >>
It is considered that the advertiser knows the points of interest that the advertiser wants the viewer to view. Therefore, the attention point determination unit 13 can detect the subject designated by the advertiser and set it as the attention point. For example, advertiser's products and services such as automobiles, furniture, and foods are detected by pattern recognition and set as points of interest.

<< Use operation history >>
When the spherical image or the 3DCG image is clicked, the line-of-sight direction (coordinates) of the clicked spherical image or the position of the virtual camera 330 may be the point of interest.

<Generation of display pattern>
Hereinafter, a method of generating a display pattern in which the transition prohibited area 81 is taken into consideration will be described with reference to FIGS. 20 to 28.

FIG. 20 is a flowchart showing an example of an overall flow in which the display pattern generation unit 16 generates a display pattern.

In step S101, the display pattern generation unit 16 acquires a spherical image or a 3DCG image from the advertisement image DB 194.

In step S102, as described with reference to FIGS. 18 and 19, the attention point determination unit 13 determines the attention point (the area including the attention point and displayed in the advertising space is referred to as the attention area). A method such as Non-Patent Document 1 may be used as a method for determining the point of interest. As a result, the degree of attention for each region (minimum unit is each pixel) of the spherical image or the 3DCG image can be obtained.

In step S103, the constraint unit 17 constrains the parameters of the line-of-sight direction of the spherical image or the position of the virtual camera 330 in the 3DCG image, and then the display pattern generation unit 16 generates the display pattern.

FIG. 21 is an example of a diagram for explaining the parameters of the line-of-sight direction of the spherical image or the position of the virtual camera 330 in the 3DCG image. The parameters of the line-of-sight direction of the all-sky image or the position of the virtual camera 330 in the 3DCG image are the camera parameters arranged in chronological order, and the camera parameters are the virtual all-sky images 6 and 3DCG images. It refers to the parameters (position / orientation, optical axis direction, focal length (angle of view), etc.) of the camera that images when placed in a three-dimensional space.

FIG. 21A shows a situation in which the virtual camera 330 moves on the sphere 340 from the time t = 1 to t = T. The vertical direction of the sphere 340 is the Z direction. In FIG. 21A, it is assumed that the sphere 340 is a spherical image, and the 3D model 95 is arranged at the center. In this case, the terminal device 30 can display the spherical image 6 as the background of the 3D model 95.

21 (b) shows the camera parameters in the situation shown in FIG. 21 (a). Camera parameters are
Camera projection center (x _t , y _t , z _t )
The optical axis direction _{(u t, v t, w} t)
Optical axis rotation θ _t
Horizontal angle of view φ _t
Is.

Since t is taken from t = 1 to t = T, the parameters of the line-of-sight direction of the spherical image or the position of the virtual camera 330 in the 3DCG image are in chronological order as shown in FIG. 21 (c). It will be lined up. One line (camera parameter) in FIG. 21 (c) is one parameter in the pattern DB 195 in Table 6.

Further, as shown in FIG. 22, the number of parameters may be reduced by setting the center of the sphere 340 on which the spherical image 6 is arranged so that the optical axis of the virtual camera 330 passes through. The camera parameters shown in FIGS. 22 (a) and 22 (b) are
Imaging direction (x _t , y _t , z _t )
Camera center displacement _dt
Optical axis rotation θ _t
Horizontal angle of view φt
Is.

Since the optical axis of the virtual camera 330 passes through the center of the sphere 340, the position of the virtual camera 330 can be obtained from the imaging direction and the displacement dt of the camera center. Therefore, the parameters can be reduced because the optical axis is unnecessary.

The constraint unit 17 constrains the camera parameters defined in this way. For example, in the transition prohibited region 81 of FIGS. 17 (a) and 17 (b), for an appropriate threshold value a ∈ [0 1].
| z _t | <a (t = 1,2,…, T)
The constraint can be expressed as follows. That is, the values that can be taken for a predetermined axis are restricted. Since this constraint means that the absolute value of z of the position of the virtual camera 330 in the line-of-sight direction of the spherical image or the 3DCG image is less than a, the transition of FIGS. 17 (a) and 17 (b). It matches the prohibited area 81. The parameter of the position of the virtual camera 330 is limited to the range of the coordinates constrained by the constraint unit, and the points of interest are set in chronological order.

The display pattern generation unit 16 generates a display pattern under such restrictions. For example, there are three possible generation methods: (1) constrained generation, (2) constraint relaxation, and (3) reparameterization. The method (1) means limiting the search range within the constraint condition when searching for the above-mentioned point of interest. In method (2), for example, for the constraint of | z _t | <a (t = 1,2,…, T), the attention of the area outside the constraint is reduced (0 in an extreme example). ), Generate the display pattern as if there were no restrictions. When searching for the above points of interest, the degree of attention of the search range may be multiplied by a value smaller than 1 (for example, 0). A display pattern can be obtained by extracting a predetermined number of points of interest restricted as described above and randomly setting the order. That is, the parameter of the position of the virtual camera 330 is set in chronological order with priority given to the points of interest having a high degree of attention after reducing the degree of attention in the range of coordinates.

The method (3) is a method of re-parameterizing the parameters using the element trajectories, and the details will be described with reference to FIGS. 23 to 28.

Returning to FIG. 20, in step S104, the display parameters generated by the display pattern generation unit 16 are stored in the pattern DB 195. In this embodiment, the display pattern and the content will be delivered to the terminal device 30, but a plurality of display images may be generated from the display pattern and the content and delivered to the terminal device 30. If there is no problem in communication cost and communication band, the processing load of the terminal device 30 can be reduced. Since the plurality of display images are reproduced like a moving image, the plurality of displayed images can be referred to as a moving image or an animation. In this case, the image distribution device 10 has a display image generation unit 44.

<Restriction method using element trajectories >>
Next, (3) restrictions due to reparameterization will be described. In FIGS. 20 to 22, the constraint unit 17 constrains the parameters of the line-of-sight direction of the spherical image or the position of the virtual camera 330 in the 3DCG image, and then generates the display pattern. On the other hand, there is a method of generating a display pattern using the element trajectory.

For example, for convenience of explanation, if the line-of-sight direction of the spherical image or the position of the virtual camera 330 in the 3DCG image is limited to a large circular orbit (the center of the circular orbit coincides with the center of the sphere), an appropriate variable. For w ₁ , w ₂ , w ₃ ∈ R,
w ₁・ x _t + w ₂・ y _t + w ₃・ z _t = 0 and x _t ² + y _t ² + z _t ² − 1 = 0 (t = 1,2,…, T)… (1)
The constraint is set.

The left side means the inner product of (w ₁ , w ₂ , w ₃ ) and (x _t , y _t , z _t ), and zero inner product means orthogonality, so the point on the sphere (x) _t , y _t , z _t ) indicates that they are on the great circle. Various great circles can be expressed by setting the variables w ₁ , w ₂ , and w ₃ . The right side represents a sphere with a diameter of 1.

In general, for the parameter Θ, use N functions g _n : (n = 1,2, ..., N) and M functions h _m : (m = 1,2, ..., M). ,
g ₁ (Θ) <0 ∧… ∧ g _N (Θ) <0 ∧ h ₁ (Θ) ＝ 0 ∧… ∧ h _M (Θ) ＝ 0… (2)
The constraint can be expressed in the form. The parameter space is a real number R.

<< Examples of orbital elements >>
23 and 24 show an example of element trajectories. FIG. 23 (a) shows the horizontal trajectory of the position of the virtual camera 330, and FIG. 23 (b) shows the horizontal trajectory of the spherical image 6 in the line-of-sight direction. The horizontal orbit can be expressed as Eq. (3).
w ₁・ x _t + w ₂・ y _t + w ₃・ z _t − u = 0 and x _t ² + y _t ² + z _t ² − 1 = 0 (t = 1,2,…, T)… ( 3)
In equation (3), when the z-axis is in the vertical direction, if w ₁ = w ₂ = 0 is set, the horizontal orbits shown in FIGS. 23 (a) and 23 (b) due to the relationship of z _t = u / w ₃ . become.

Further, as shown in FIG. 24, an orbit around an arbitrary axis can be set. FIG. 24A shows the large circular orbit 97 of the virtual camera 330, and FIG. 24B shows the large circular orbit 98 in the line-of-sight direction of the spherical image 6. If u = 0 is set in the equation (3), the large circular orbit shown in FIG. 24 is obtained. Therefore, equation (3) is a generalized equation for horizontal orbits and large circular orbits.

Further, as shown in FIG. 25A, the horizontal rotation trajectory can be set for any axis 99. Further, as shown in FIG. 25B, the spiral orbit may be an element orbit. In FIG. 25B, the position of the virtual camera 330 or the line-of-sight direction of the spherical image 6 is spirally moving on the sphere 340. Such a spiral orbit can be expressed by the equation (4).
w ₁・ x _t + w ₂・ y _t + w ₃・ z _t − (u + vt) = 0 and x _t ² + y _t ² + z _t ² − 1 = 0 (t = 1,2,…, T)… (4)
As shown in equation (4), the time component vt is incorporated into the constraint. If v = 0 is set, it becomes the same as equation (3). v is a parameter of the speed of movement in the axial direction.

<< Procedure for generating display patterns using element trajectories >>
With the element trajectory as described above, it is possible to suppress the inconvenience of losing smoothness by transitioning different viewpoints in a complicated trajectory.

When the display pattern is generated by reflecting the constraint on the element trajectory, the processing content of step S103 in FIG. 20 changes. FIG. 26 is a flowchart showing a procedure executed in step S103 of FIG. 20 when generating a display pattern using element trajectories.

In step S201, the display pattern generation unit 16 generates specific values of the element trajectory parameters (w ₁ , w ₂ , w ₃ , u, v). For example, it is randomly generated based on a uniform distribution in a specific region of the parameter of the element trajectory (which is intended to be limited to a certain range and may be determined from the processing power of a computer or the like).

In step S202, the display pattern generation unit 16 calculates the cumulative value of the degree of attention on the element trajectory generated by the parameters randomly generated.

FIG. 27 is a diagram illustrating a cumulative value of attention on the element trajectory. As a method for selecting the element trajectory, a method in which the cumulative value of the degree of attention on the element trajectory takes an extreme value can be selected. If it is a horizontal rotation trajectory, the element trajectory can be expressed with the elevation angle as a parameter. The figure is an example of calculating the cumulative attention level on the element trajectory corresponding to each elevation angle. In this example, there are two extrema, and the element trajectory of the elevation angle that takes the extremum is selected.

Further, as an alternative to the selection method using extrema, the elevation angle may be clustered by regarding the cumulative attention level as the frequency. In the case of a large circular orbit, the same discussion as above can be made with the direction vector of the rotation axis as a parameter.

Returning to FIG. 26, in step S203, the display pattern generation unit 16 determines the adoption of the parameter from the value of the cumulative attention level. If it is larger than the cumulative attention level obtained by searching so far, the parameter is adopted and the cumulative attention level is recorded.

In step S204, if the end condition (for example, repeating a specific number of times) is satisfied, the process ends, otherwise the process returns to step S201. With the above, one element trajectory can be obtained. The display pattern generation unit 16 generates the display pattern having a continuous viewpoint on the element trajectory.

FIG. 26 shows a method of finding the parameters of the element trajectory by a random search, but as another method, there is a method of dividing the parameter space of the element trajectory on the grid and performing a full search for the grid points. In that case, the grid points are sequentially selected in step S201. The end condition is that all grid points have been selected in step S204.

In addition, search methods using metaheuristics such as simulated annealing, genetic algorithms, and tabu search may be used. In this case, in step S201, parameters are generated according to the algorithm of each metaheuristic, and in step S203, the parameters are adopted according to the criteria of each metaheuristic algorithm. When Simulated Annealing is used, parameters are randomly generated near the parameters of the previous iteration in step S201, and adopted in step S203 when the cumulative attention is higher than the parameters of the previous iteration, and is constant even if it is low. Adopted with the probability of.

When FIG. 26 is executed on the element trajectories of the equations (2) to (4), there is a degree of freedom regarding the speed of parameter change on the element trajectories. This is set manually by the person in charge. In addition, the element trajectories of equations (2) to (4) only determine the position of the virtual camera 330 or the line-of-sight direction of the spherical image 6, and other parameters such as the angle of view remain flexible. ing. This is manually set by the person in charge, or generated by the same method as the method of determining the point of interest.

<< How to find multiple element trajectories >>
Since FIG. 26 is an example of selecting the parameter of one element trajectory, when obtaining two or more, the method of executing the above method a plurality of times by changing the seed of the random number and obtaining a plurality of semi-optimal solutions is used.

Another approach is to use a clustering method. The processing flow of the approach using clustering is shown in FIG. 28, and the contents of each step will be described below. FIG. 28 is a flowchart showing a procedure for obtaining two or more element trajectories.

In step S301, the constraint unit 17 appropriately discretizes the parameter space of the element trajectory (limits the value taken by each parameter to a fixed value such as 1, 2, ...), And obtains the cumulative value of the degree of attention for each parameter. ..

In step S302, the cumulative attention level calculated by the display pattern generation unit 16 is regarded as a probability with respect to the parameter of the element trajectory, and the parameter of the element trajectory is divided into a specific number of clusters. For clustering, general clustering methods such as k-means and EM algorithm / variational Bayesian method for mixture distribution can be used. In the above method of determining the attention point, the "attention degree" is regarded as a probability and the "camera parameters" are clustered, but in FIG. 28, the "cumulative attention degree" is regarded as a probability and the "element orbit parameters" are clustered. There is a difference.

In step S303, the display pattern generation unit 16 selects a representative value (average value, median value, mode, etc.) of each cluster. This makes it possible to obtain as many element trajectory parameters as there are clusters.

In step S304, the display pattern generation unit 16 determines the transition order of the obtained plurality of element trajectories. One method is to determine the transition order by putting an appropriate order relationship (for example, the magnitude of the Euclidean distance from the origin) in the parameter space of the element trajectory and sorting. Another method is to recursively select one element trajectory at random and then select the element trajectory closest to the parameter to determine the order.

In step S305, the display pattern generation unit 16 generates a transition orbit between the element orbits having a fixed order. If the element trajectories are simply joined, they are generally discontinuous at the joining point, so the element trajectories are interpolated. This may be linear interpolation or spline interpolation. To show the case of linear interpolation concretely, if the animation parameter at time t corresponding to the i-th element trajectory is written as Θi (t), the animation parameter Θ'that interpolates the i-th and i + 1th element trajectories. i (t) can be generated by the following equation. Note that T is the total number of frames.

If the time lengths of Θi and Θi + 1 are different, copy or resize the elements so that they have the same time length. For example, if Θi (1) ... Θi (10), the time length is 10, and if Θi + 1 (1) ... Θi + 1 (20), the time length is 20, so the time lengths of the two are different. In this case, each element of the former animation parameter is copied to a total length of 20 hours, or every other element of the latter animation is deleted and resized.

Using the display parameters interpolated in this way, an animation that changes continuously by arranging them together with the display parameters before interpolation as (Θ1, Θ'1, Θ2, Θ'2, Θ3, ...) You can get it.

From the above, a display pattern based on a plurality of element trajectories can be generated. It should be noted that the display pattern may be generated by executing the steps after step S304 after obtaining the plurality of element trajectories by the search method of FIG.

<Operation procedure>
FIG. 29 is an example of a sequence diagram illustrating a procedure in which the image display system 100 delivers a spherical image 6 or a 3DCG image displayed as an advertisement.

S1: First, the viewer operates the terminal device 30 so as to connect to the partner site Web server 60. The operation reception unit 34 of the terminal device 30 accepts the operation, and the Web page acquisition unit 31 acquires the Web page.

S2: The Web page analysis unit 32 of the terminal device 30 analyzes the Web page, extracts HTML, CSS, and a script, and the Web page display unit 33 displays the Web page based on the HTML and CSS.

S3: The advertisement space detection unit 36 of the terminal device 30 transmits an advertisement request to the SSP 50 by executing an advertisement tag (script) associated with the advertisement space 7. The advertisement request includes, for example, an SSP cookie, a domain of the partner site Web server 60, an advertisement space ID, an advertisement space 7 size, an advertisement format, a browser type, an OS type, and the like.

S4: The advertisement request acquisition unit 52 of the SSP 50 acquires the advertisement request, and the advertisement request unit 51 accepts the advertisement request. The advertisement request unit 51 detects the SSP cookie, notifies the DSP 20 of the advertisement request, and sells the advertisement space 7.

S5: The request reception unit 21 of the DSP 20 sends an advertisement request to the bid determination unit 22. The bid determination unit 22 of the DSP 20 refers to the cookie information DB 291 and identifies the DSP cookie from the SSP cookie.

S6: The bid determination unit 22 determines the attribute from the DSP cookie. The visiting domain of the cookie information DB 291 may be referred to, or the attribute information of the viewer specified by the DSP cookie may be referred to as appropriate.

S7: The bid determination unit 22 refers to the distribution setting DB 292, and based on the attributes of the preferred target person or the unfavorable target person set for the provider of the wide-angle image or the 3DCG image, the wide-angle image is given to this viewer. Decide whether to send. Here, it is assumed that it is decided to deliver to at least one advertiser.

S8: The bid judgment unit 22 determines the bid amount according to the advertiser's budget, the degree of matching of attributes, and the like.

S9: The bidding section 23 of the DSP 20 bids on the SSP 50. The successful bidder 54 of the SSP50 determines as the winning bidder the DSP20 that has offered the highest bid amount in principle for bids from a plurality of DSP20s.

S10: Here, it is assumed that the illustrated DSP20 has made a successful bid. The SSP 50 notifies the DSP 20 of the winning bid ID.

S11: The access information transmission unit 53 of the SSP 50 transmits the access information 1 to the terminal device 30 together with the successful bid ID.

S12: The advertisement request unit 38 of the terminal device 30 transmits the advertisement request to the DSP 20 together with the winning bid ID by executing the access information 1.

S13: The advertisement request reception unit 24 of the DSP 20 identifies the advertisement request by the winning bid ID. The image requesting unit 25 requests an image from the image distribution device 10 together with the advertiser ID, the DSP cookie, and the advertisement request.

S14: When the image information response unit 11 of the image distribution device 10 receives the image request, it generates an advertisement opportunity ID for specifying the advertisement request.

S15: The image information response unit 11 transmits the access information 2 associated with the advertisement opportunity ID to the DSP 20.

S16: The image requesting unit 25 of the DSP 20 receives the access information 2, and the advertisement request receiving unit 24 transmits it to the terminal device 30.

S17: The advertisement acquisition unit 39 of the terminal device 30 transmits an advertisement request (advertisement opportunity ID) based on the URL of the image distribution device 10 included in the access information 2. When the terminal device 30 holds the image cookie, the image cookie can be transmitted.

S18: The advertisement distribution unit 12 of the image distribution device 10 identifies the image request by the advertisement opportunity ID.

S19: The advertisement distribution unit 12 determines the attribute from the DSP cookie or determines the attribute from the image cookie.

S20: The advertisement distribution unit 12 identifies the spherical image 6 based on the advertiser ID and attributes. That is, the all-sky image 6 or 3DCG image to be transmitted is determined based on the attribute or priority of the distribution target person set for the all-sky image 6 or 3DCG image with reference to the advertisement image DB 194. The display pattern has a click count higher than the threshold value, but is randomly selected while the number of distributions of the spherical image 6 or the 3DCG image is small.

S21: The advertisement distribution unit 12 transmits the advertisement data including the all-sky image 6 or the 3DCG image (which may be both), the initial position, the display pattern, the image ID, the image cookie, and the advertisement space ID to the terminal device 30. To do. The URL of the advertiser Web server 70 is associated with the spherical image 6 or the 3DCG image. Further, when the image distribution device 10 creates a plurality of display images (animations), a plurality of display images are distributed instead of the spherical image 6 or the 3DCG image.

S22: The advertisement acquisition unit 39 of the terminal device 30 acquires the advertisement data, and the advertisement display unit 40 displays the spherical image 6 or the 3DCG image on the advertisement frame 7. When the viewer presses (clicks or taps) the advertisement space 7, the operation reception unit 34 accepts the advertisement frame 7, and the Web page acquisition unit 31 starts communication with the advertiser Web server 70 based on the URL of the advertiser Web server 70.

<Operation of terminal device>
FIG. 30 is an example of a flowchart for explaining the operation of the terminal device 30 that has received the advertisement data. FIG. 30A is an operation procedure when the terminal device 30 creates a plurality of display images, and FIG. 30B is an operation procedure when the image distribution device 10 creates a plurality of display images.

In step S401, the advertisement acquisition unit 39 of the terminal device 30 receives the spherical image or the 3DCG image (which may be both) and the display pattern.

In step S402, the display image generation unit 44 sequentially acquires the time-series parameters included in the display pattern.

In step S403, the display image generation unit 44 generates a predetermined area image or a projected image according to the parameter, and the advertisement display unit 40 displays the predetermined area image or the projected image on the advertisement frame 7.

In step S404, the display image generation unit 44 determines whether or not the reproduction is completed. Judgment as to whether or not playback has ended means determining whether or not the display pattern has been repeated a fixed number of times. The process returns to step S402 until the end of the reproduction, and when the reproduction is completed, the advertisement display unit 40 stops the predetermined area image or the projected image and ends the process.

When the image distribution device 10 creates a plurality of display images, in step S501, the advertisement acquisition unit 39 of the terminal device 30 receives the spherical image or the 3DCG image and the plurality of display images generated from the display pattern.

In step S502, the advertisement display unit 40 reproduces a plurality of display images like a moving image (or animation). Playback ends when it is played to the end.

<Supplementary form of advertisement data distribution>
In the present embodiment, the advertisement is distributed by a mechanism of third-party distribution, but the advertisement may be finally displayed on the Web page displayed by the terminal device 30. FIG. 31 shows some forms in which the advertisement data is delivered.

In FIG. 31A, the Web page is transmitted with the advertisement data included in the Web page distributed by the partner site Web server 60. In this case, the advertiser has previously submitted the image data to the partner site Web server 60.

In FIG. 31B, the advertisement distribution server 80 distributes the advertisement data. The partner site Web server 60 registers the Web page on which the advertisement is to be placed on the advertisement distribution server 80. As a result, the advertisement distribution server 80 outputs the ad tag, so that the partner site Web server 60 attaches the ad tag to the Web page on which the advertisement is to be placed. In addition, the advertiser uploads the advertisement data to the advertisement distribution server 80.

When the viewer accesses the partner site Web server 60, the advertisement distribution server 80 is called by the ad tag. The advertisement distribution server 80 determines which advertisement submitted by each advertiser is to be distributed, and transmits the advertisement data to the terminal device 30.

FIG. 31 (c) is an example of distribution of advertisement data in the embodiment described in this embodiment. As described in this embodiment, the image distribution device 10 may distribute the advertisement data, or the DSP 20 may distribute the advertisement data. The advertiser may upload the advertisement data to the DSP 20. Further, in the configuration of FIG. 31C, the DSP 20 or the image distribution device 10 serves as the advertisement distribution server 80.

Further, although the image data displayed in the advertisement frame 7 has been described as a preferable example of the image data, the image data is not limited to the advertisement as long as the image data is distributed to the terminal device 30 and changes by the operation of the viewer (has interactiveness). The present embodiment can be preferably applied.

<Summary>
As described above, the image display system of the present embodiment can prevent the viewpoint from transitioning unnaturally by restricting the viewpoint in this way. For example, it is possible to suppress the transition to a viewpoint in which the content is viewed from directly above or looked up from directly below. In addition, by constraining the viewpoint after setting the element trajectory, it is possible to suppress the transition of different line-of-sight directions in a complicated trajectory. Therefore, the content can be displayed from an appropriate viewpoint, and it becomes easy to enhance the advertising effect.

<Other application examples>
Although the best mode for carrying out the present invention has been described above with reference to Examples, the present invention is not limited to these Examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

For example, the position of the virtual camera is not limited to a sphere, but may be an ellipse, a rectangular parallelepiped, or the like.

Further, in the present embodiment, the example in which the spherical image 6 is displayed in the advertising space 7 has been described, but the spherical image 6 may be a wide-angle image. That is, the image data may be any image data having an angle of view that does not fit in the advertising space 7 and whose viewing range can be changed by the operation of the viewer. Further, the image displayed on the Web page does not have to be an image for advertisement, and may be image data that can be viewed.

Further, the application of the terminal device 30 for displaying the all-sky image 6 is not limited to the browser software, and the present embodiment can be applied when any application software displays the all-sky image 6 or the 3DCG image on the screen.

In the present embodiment, the terminal device 30 accesses the DSP 20 and then accesses the image distribution device 10 to acquire advertisement data. However, the DSP 20 may acquire the advertisement data directly from the image distribution device 10 and transmit it to the terminal device 30. Since the terminal device 30 can acquire the advertisement data only by accessing the DSP 20, the time until the advertisement is displayed can be shortened. Further, the DSP 20 and the image distribution device 10 are integrally configured, and the DSP 20 may distribute the advertisement data (omnidirectional image) to the terminal device 30.

Further, the advertisement distribution method in which the SSP 50, DSP 20 and the image distribution device 10 linked with each other described in the present embodiment is only an example, and there is no intention of limiting the process until the terminal device 30 displays the advertisement. For example, as a simple process, the partner site Web server 60 may store the advertisement data from the advertiser and distribute it to the terminal device 30. In this case, the SSP 50, the DSP 20, and the image distribution device 10 are not required.

Further, the spherical image 6 is not limited to the case where two images are pasted together, and for example, the spherical image 6 may be created by superimposing overlapping portions of image data captured while moving the imaging direction little by little. Good. That is, the method of creating the spherical image 6 is not limited.

Further, the configuration examples such as FIG. 13 shown in the above examples are divided according to the main functions in order to facilitate understanding of the processing of the image display system 100. However, the present invention is not limited by the method and name of division of each processing unit. The image display system 100 can be divided into more processing units according to the processing content. Further, one processing unit can be divided so as to include more processing.

Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array) and conventional circuit modules.

7 Advertising space 10 Image distribution device 20 DSP
30 Terminal device 40 Advertising display unit 50 SSP
60 Partner site web server 70 Advertiser web server 100 Image display system

Japanese Unexamined Patent Publication No. 2008-192115

C.H. Lee, A. Varshney, D.W. Jacobs, "Mesh saliency," ACM transactions on graphics (TOG), Vol. 24. No. 3. 2005.

Claims (11)

An image display system in which a terminal device receives content from an image distribution device that distributes images.
The image distribution device is
A display pattern generator that generates a display pattern that displays one or more contents from different viewpoints,
The constraint part that constrains the viewpoint and
It has the display pattern and the content, or a distribution unit that distributes a plurality of display images generated based on the display pattern to the terminal device.
The terminal device is
An acquisition unit that acquires the display pattern and the content, or a plurality of display images generated based on the display pattern, and
A display unit that displays the content acquired by the acquisition unit with a different viewpoint in the display pattern, or displays the plurality of display images generated based on the display pattern.
An image display system characterized by having. The image display system according to claim 1, wherein the restriction unit prohibits or restricts the transition of the viewpoint to a specific area at coordinates representing the viewpoint for viewing the content. Claim 1 is characterized in that the constraint unit constrains a value that can be taken with respect to a predetermined axis of coordinates representing the viewpoint among the position of a virtual camera representing the viewpoint or the direction of the line of sight with respect to the content. Or the image display system according to 2. It has a point of interest determination unit that determines the point of interest of the content.
The display pattern generation unit is limited to the range of the coordinates constrained by the constraint unit among the plurality of the attention points determined by the attention point determination unit, and sets the attention points in time series. The image display system according to claim 3, wherein a display pattern is generated. It has a point of interest determination unit that determines the point of interest of the content.
Among the plurality of attention points determined by the attention point determination unit, the display pattern generation unit reduces the degree of attention in the range of coordinates that deviates from the constraint by the constraint unit, and then gives priority to the attention point having a large degree of attention. The image display system according to claim 3, wherein the display pattern is generated by setting the display pattern in time series. The display pattern generation unit is characterized in that it generates the display pattern with the element trajectory as a viewpoint based on the degree of attention of a point of interest through which the element trajectory rotating around the content passes. The image display system according to 2. It has a point of interest determination unit that determines the point of interest of the content.
The display pattern generation unit accumulates the attention points of the points of interest determined by the attention point determination unit through which the element trajectory passes, and determines the element trajectory having the highest accumulated attention.
The image display system according to claim 6, wherein the display pattern with the element trajectory as a viewpoint is generated. The display pattern generation unit discretizes the parameter space of the element trajectory, obtains the cumulative value of the attention level for each parameter, regards the cumulative value of the attention level as a probability, performs clustering, and represents each cluster. The value is determined as the parameter of the element trajectory,
The image display system according to claim 6, wherein the display pattern is generated with the determined element trajectory as a viewpoint. The image display system according to claim 8, wherein the display pattern generation unit generates the display pattern by interpolating the parameters of the element trajectories. This is an image distribution method performed by an image display system in which a terminal device receives content from an image distribution device that distributes images.
The image distribution device is
A step in which the display pattern generator generates a display pattern that displays one or more contents from different viewpoints, and
The step in which the constraint unit constrains the viewpoint,
The distribution unit includes the display pattern and the content, or a step of distributing a plurality of display images generated based on the display pattern to the terminal device.
The terminal device is
A step in which the acquisition unit acquires the display pattern and the content, or a plurality of display images generated based on the display pattern, and
A step in which the display unit displays the content acquired by the acquisition unit from different viewpoints in the display pattern, or displays the plurality of display images generated based on the display pattern.
An image distribution method characterized by having. An image distribution device that distributes images to a terminal device, a display pattern generator that generates a display pattern that displays one or more contents from different viewpoints, and a display pattern generator.
The constraint part that constrains the viewpoint and
A distribution unit that distributes the display pattern and the content, or a plurality of display images generated based on the display pattern to the terminal device.
A program to function as.