JPWO2017056229A1 - Information processing apparatus, information processing method, and program for information processing apparatus - Google Patents

Abstract

  Among the image frames included in the moving image acquired from the moving image storage means (12a) for storing the moving image, at least some of the plurality of image frames are acquired (S11), and each acquired image frame is displayed in the image frame. At least one object is identified (S12), and object information related to the object displayed in each image frame is calculated for each identified object (S13, S20). Based on the additional content display condition for displaying the additional content to be added, an object for displaying the additional content is selected from the identified objects (S21), and an area including at least a part of the selected object According to the object information of the selected object It generates display information for displaying the additional content in the area (S22).

Description

  The present invention relates to a technical field of an information processing apparatus, an information processing method, and a program for an information processing apparatus that display other images in a moving image.

  A service is known in which content provided by a content provider such as a moving image is registered in a content distribution site, and the content is distributed in response to a request from a viewing user in the content distribution site. In the above service, an advertisement related to the distributed content is displayed so as to overlap a part of the content display area, and a reward is obtained in response to a click on the advertisement by a viewing user. As a technique for displaying an image superimposed on a part of a content display area, for example, Patent Document 1 displays either a moving image or a still image in an area where the amount of motion of the image is small, and the other area in the other area. A technique for displaying is disclosed.

JP 2008-042326 A

  However, when the technology of Patent Document 1 is applied to a service for distributing content, an area for displaying a new advertisement in an area that is not originally separated from an adjacent area in a moving image suddenly appears. Therefore, there is a problem in that the viewing user feels uncomfortable, and as a result, the viewing user's willingness to view is hindered.

  The present invention has been made in view of such a problem, and an example of the problem is to provide a service for displaying other images such as advertisements in an area that does not give the viewer a sense of incongruity in a moving image. For the purpose.

  In order to solve the above problem, the invention according to claim 1 is an image frame for acquiring at least some of the image frames included in the moving image acquired from the moving image storage means for storing the moving image. Acquisition means, object specifying means for specifying at least one object displayed in the image frame for each acquired image frame, and the image displayed in each of the image frames for each specified object Object information calculation means for calculating object information related to an object, the calculated object information, and an additional content display condition for displaying additional content to be added to the video, for displaying the additional content Select an object from the identified objects Object selection means, and display information generation means for generating display information for displaying the additional content in an area according to the object information of the selected object, including at least a part of the selected object. It is characterized by having.

  According to the present invention, an object is selected based on the object information of the object displayed in the image frame and the additional content display condition for displaying the additional content to be added to the moving image, and at least one of the selected objects is selected. Display information for displaying additional content in the area according to the object information of the selected object, and adding it to the area according to the object information of the object already displayed in the image frame Since the content is displayed, the additional content can be displayed in an area that does not give the viewer a sense of incongruity in the moving image. Further, the additional content can be displayed flexibly according to the object.

  The invention according to claim 2 is the information processing apparatus according to claim 1, wherein the object information includes an integrated display amount in which the object is displayed by integration of at least one of space and time in the moving image, The object selecting means selects an object for displaying the additional content according to the integrated display amount.

  In this case, an object corresponding to the additional content display condition can be selected as an object corresponding to exposure in at least one of space and time.

  According to a third aspect of the present invention, in the information processing apparatus according to the first or second aspect, the object information includes a continuous display number of the objects that are continuously displayed in the plurality of image frames. The object selection means selects an object for displaying the additional content according to the number of continuous displays.

  In this case, the additional content can be displayed on the moving image without being interrupted.

  According to a fourth aspect of the present invention, in the information processing apparatus according to any one of the first to third aspects, the object information includes a difference data amount of the object between adjacent frames of the image frame. And the object selecting means selects an object having a larger difference data amount than a threshold set by the difference data amount of each object.

  In this case, the amount of data when the moving image is compressed can be reduced by replacing an object having a large amount of difference data with additional content with little change in color or the like for each image frame.

  According to a fifth aspect of the present invention, in the information processing apparatus according to any one of the first to fourth aspects, the object information includes information on the shape of the object, and the additional content display condition is , Including information on the shape of the additional content, and the object selecting means selects an object for displaying the additional content according to the information on the shape of the object so as to match the shape of the additional content. It is characterized by that.

  In this case, since the additional content can be displayed so as to match the shape of the object without changing the shape of the additional content, the process of changing the shape of the additional content can be omitted.

  According to a sixth aspect of the present invention, in the information processing apparatus according to any one of the first to fifth aspects, the display information has a single area according to the object information of the selected object. It includes a moving image masked with the color of

  In this case, since it is masked with a single color, the amount of data when the moving image is compressed can be reduced.

  According to a seventh aspect of the present invention, in the information processing apparatus according to any one of the first to sixth aspects, the color of the pixel in the area according to the object information of the selected object, A color for masking the area is determined for each pixel based on the color of the pixel of the additional content added to the area, and the display information includes a moving image in which the area is masked with the color determined for the pixel. Features.

  In this case, for example, since the area can be masked with an intermediate color for each corresponding pixel, a part of the image frame between the image frame before displaying the additional content and the image frame after displaying the additional content is a pixel. Since each intermediate color is masked, the amount of data when the moving image is compressed can be reduced.

  According to an eighth aspect of the present invention, in the information processing apparatus according to any one of the first to seventh aspects, the object information calculation unit is continuously displayed in a plurality of the image frames. The object information is calculated using an object as one object.

  In this case, the object information is calculated in units that are continuous in the image frame, and the additional content can be displayed at a timing such as when the object is temporarily hidden behind another object, deviated from the shooting angle, or the scene changes. Even when the content is switched, the additional content can be displayed so as not to give the viewer a sense of incongruity.

  The invention according to claim 9 is an image frame acquisition step in which the image frame acquisition means acquires at least some of the image frames included in the moving image acquired from the moving image storage means that stores the moving image. An object specifying means for specifying at least one object displayed in the image frame for each of the acquired image frames; and an object information calculating means for each of the specified objects. An object information calculating step for calculating object information relating to the object displayed in the image frame; and an additional content display for displaying the calculated object information and additional content to be added to the moving image by an object selection means. Based on the conditions, before An object selection step of selecting an object for displaying additional content from the identified object, and a display information generating means includes at least a part of the selected object, and the object information of the selected object And a display information generation step of generating display information for displaying the additional content in the conforming area.

  According to a tenth aspect of the present invention, there is provided the image frame acquiring means for acquiring at least some of the image frames included in the moving image acquired from the moving image storage means for storing the moving image, the acquisition Object specifying means for specifying at least one object displayed in the image frame for each image frame, and calculating object information relating to the object displayed in each image frame for each specified object Object information calculation means, based on the calculated object information and an additional content display condition for displaying additional content to be added to the moving image, an object for displaying the additional content is specified as the specified object Select an object to select from And a display information generating unit that generates at least a part of the selected object and generates display information for displaying the additional content in an area according to the object information of the selected object. It is characterized by.

  ADVANTAGE OF THE INVENTION According to this invention, the service which displays other images, such as an advertisement, in the area | region which does not give a discomfort to a viewing user within the image of a moving image can be provided.

It is a mimetic diagram showing an example of outline composition of an information processing system concerning one embodiment. It is a block diagram which shows an example of schematic structure of the moving image delivery server apparatus which concerns on one Embodiment. It is a figure which shows an example of the content registered into the database constructed | assembled in the memory | storage part of the moving image delivery server apparatus. It is a figure which shows an example of the content registered into the database constructed | assembled in the memory | storage part of the moving image delivery server apparatus. It is a figure which shows an example of the content registered into the database constructed | assembled in the memory | storage part of the moving image delivery server apparatus. It is a figure which shows an example of the content registered into the database constructed | assembled in the memory | storage part of the moving image delivery server apparatus. It is a figure which shows an example of the content registered into the database constructed | assembled in the memory | storage part of the moving image delivery server apparatus. It is a figure which shows an example of the content registered into the database constructed | assembled in the memory | storage part of the moving image delivery server apparatus. It is a figure which shows an example of the functional block of the moving image delivery server apparatus which concerns on one Embodiment. It is a block diagram which shows an example of schematic structure of the content provision apparatus of FIG. It is a block diagram which shows an example of schematic structure of the advertisement provision apparatus of FIG. It is a block diagram which shows an example of schematic structure of the viewing-and-listening user terminal device of FIG. It is a schematic diagram which shows an example of the frame number of the image frame in moving image data. It is a schematic diagram which shows an example of an image frame. It is a schematic diagram which shows an example of an image frame. It is a schematic diagram which shows an example of an image frame. It is a schematic diagram which shows an example of an image frame. It is a schematic diagram which shows an example of an image frame. It is a schematic diagram which shows an example of an image frame. It is a schematic diagram which shows an example of an image frame. It is a schematic diagram which shows an example of an image frame. It is a schematic diagram which shows an example of an image frame. It is a flowchart which shows an example of the moving image registration process of the system control part of the moving image delivery server apparatus which concerns on one Embodiment. It is a flowchart which shows an example of the advertisement registration process of the system control part of the moving image delivery server apparatus which concerns on one Embodiment. It is a flowchart which shows an example of the object specific process of the system control part of the moving image delivery server apparatus which concerns on one Embodiment. It is a schematic diagram which shows an example of the feature point image of an image frame. It is a schematic diagram which shows an example of the feature point image of an image frame. It is a schematic diagram which shows an example of the feature point image of an image frame. It is a schematic diagram which shows an example of the feature point image of an image frame. It is a schematic diagram which shows an example of the object display area in an image frame. It is a schematic diagram which shows an example of the object display area in an image frame. It is a schematic diagram which shows an example of the object display area in an image frame. It is a schematic diagram which shows an example of the object display area in an image frame. It is a schematic diagram which shows an example of an object display area. It is a schematic diagram which shows an example of an object display area. It is a flowchart which shows an example of the control information generation process of the system control part of the moving image delivery server apparatus which concerns on one Embodiment. It is a schematic diagram which shows an example of the image frame to which the advertisement was added. It is a schematic diagram which shows an example of the image frame to which the advertisement was added. It is a schematic diagram which shows an example of the image frame to which the advertisement was added. It is a schematic diagram which shows an example of the image frame to which the advertisement was added. It is a schematic diagram which shows an example of the image frame to which the advertisement was added. It is a schematic diagram which shows an example of the image frame to which the advertisement was added. It is a schematic diagram which shows an example of the image frame in which one part was masked. It is a schematic diagram which shows an example of the image frame in which one part was masked. It is a schematic diagram which shows an example of the image frame in which one part was masked. It is a schematic diagram which shows an example of the image frame in which a part was masked It is a flowchart which shows an example of the moving image delivery process of the system control part of the moving image delivery server apparatus which concerns on one Embodiment. It is a flowchart which shows an example of the control information change process of the system control part of the moving image delivery server apparatus which concerns on one Embodiment. It is a schematic diagram which shows an example of the skip or rewind in moving image data. It is a schematic diagram which shows the modification of the mask which masks a part of image frame. It is a schematic diagram which shows the modification of the mask which masks a part of image frame.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The embodiment described below is an embodiment when the present invention is applied to an information processing system.

[1. Overview of information processing system configuration and functions]
First, the configuration and functional overview of the information processing system 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a schematic diagram illustrating an example of a schematic configuration of an information processing system 1 according to an embodiment.

  As illustrated in FIG. 1, the information processing system 1 includes a moving image distribution server device 10 that distributes moving image content to a viewing user, and a content provider device 20 that provides moving image content to the moving image distribution server device 10 as content. And an advertisement provider device 30 that provides additional content such as an advertisement, and a viewing user terminal device 40 of a viewing user who views a moving image. A moving image is composed of a plurality of image frames.

  The video distribution server device 10, the content providing device 20, the advertisement providing device 30, and the viewing user terminal device 40 are connected by the network 3, and data can be transmitted and received by a communication protocol (for example, TCP / IP). It has become. The network 3 is constructed by, for example, the Internet, a dedicated communication line (for example, a CATV (Community Antenna Television) line), a mobile communication network (including the base station 5 and the like), a gateway, and the like. The moving image distribution server device 10 and the content providing device 20 may be connected by a dedicated line for improving security.

  The moving image distribution server device 10 (an example of an information processing device) registers the moving image provided from the content providing device 20 and distributes the moving image content to the viewing user terminal device 40 in response to a request from the viewing user terminal device 40. In addition, when the moving image distribution server device 10 distributes moving image content to the viewing user terminal device 40 in response to a request from the viewing user terminal device 40, the moving image distribution server device 10 sends additional content such as advertisements regarding products or services to the viewing user terminal device 40. To deliver.

  The content provider uses each content providing device 20 to transmit the moving image to the moving image distribution server device 10 in order to register the moving image provided by the content provider.

  The advertisement provider uses each advertisement providing device 30 to transmit the additional content such as the advertisement to the moving image distribution server device 10 in order to register the additional content such as the advertisement provided by the advertisement provider.

  The types of additional content include advertisement and non-advertisement. For example, as a non-advertisement, in addition to an advertisement for a product or service, a message, a video, or the like that an advertisement provider or the like wants to show for each viewing user as a moving image can be cited. Further, examples of the data format of the additional content include characters and images such as text and design characters, and combinations thereof. In addition, examples of the video include a still image, a moving image, and a combination thereof.

  Further, the advertisement provider transmits information indicating the additional content display condition for displaying the additional content on the moving image to the moving image distribution server device 10 using each advertisement providing device 30.

  Here, as an example of the additional content display condition, there is a shape of the additional content (for example, a rectangle, a circle, an ellipse, a star, etc.). Further, the aspect ratio (the ratio of the vertical length to the horizontal length of the advertisement) is used for a rectangle, and the ratio between the major axis and the minor axis is included for an ellipse. Further, the additional content display condition may be a reference display size (for example, the number of pixels indicating a horizontal or vertical length) of the additional content, or a minimum display size of the additional content.

  Furthermore, as an example of the additional content display condition, there is a display time of the additional content, and there are a plurality of group units, a display time when a plurality of frames that are continuously displayed in time are set as one group unit, There is a group total display time when these group units are combined. Regarding the display time of additional content in these two cases, there are total display time, minimum total display time, maximum total display time, minimum continuous display time, maximum continuous display time, average continuous display time, etc. .

  In addition, as an example of the additional content display condition, the number of times of display of the additional content can be given. There are a plurality of display times when a plurality of frames that are continuously displayed in time are set as one group unit, and a plurality of the group units. There is a group total display count when these group units are combined. With regard to the number of additional content displays in these two cases, there are a total display count, a minimum total display count, a maximum total display count, a minimum continuous display count, a maximum continuous display count, an average display count, and the like.

  In addition, as an example of the additional content display condition, there is a display area of the additional content, and there are a plurality of display areas when a plurality of frames that are continuously displayed in time are set as one group unit, and a plurality of the group units. There is a group total display area when these group units are combined. Regarding the display area of the additional content in these two cases, there are a minimum display area, a maximum display area, an average display area, a predetermined display area such as 1/10 of the image frame, and the like.

  Further, as an example of the additional content display condition, there is an integrated display amount displayed by integrating at least one of space and time in a moving image, and a case where a plurality of frames that are continuously displayed in time are set as one group unit. There are a total display amount and a group total integration display amount when there are a plurality of group units and the group units are added together. Regarding the integrated display amounts in these two cases, there are a minimum integrated display amount, a predetermined integrated display amount such as 5 seconds in an area of 1/10 of the image frame, and the like.

  Further, as an example of the additional content display condition, a display permission position (upper, lower, right, left, lower right, lower left, center, etc.) of an additional content in an image frame constituting a moving image, or a display prohibited position (for example, The position where the advertisement provider does not want to display the additional content in the image frame).

  In addition, as an example of additional content display conditions, conditions such as temporal display positions of moving images in which additional content display is permitted or prohibited (for example, display permission or prohibition in the first half / second half of the moving image, only predetermined chapters are permitted) Or prohibition, only allow / prohibit only the first / last 5 minutes).

  In addition, as an example of additional content display conditions, a video category in which additional content display is permitted or prohibited (such as a video that includes a whitelist, blacklist, or specific keyword), or a video category in which additional content display is permitted or prohibited Etc.

  The viewing user uses the viewing user terminal device 40 to search for a moving image provided in the moving image distribution service by using a search keyword or category selection, and selects a moving image to be viewed. The viewing user terminal device 40 transmits a moving image distribution request according to a user input or selection to the moving image distribution server device 10, receives a moving image corresponding to the distribution request from the moving image distribution server device 10, and receives the received image. Play the recorded video.

[2. Configuration and function of each device]
(2.1 Configuration and Function of the Video Distribution Server Device 10)
Next, the configuration and function of the moving image distribution server device 10 will be described with reference to FIGS.

  FIG. 2 is a block diagram illustrating an example of a schematic configuration of the moving image distribution server device 10 according to an embodiment. 3 to 8 are diagrams illustrating examples of contents registered in a database constructed in the storage unit of the moving image distribution server device 10.

  As shown in FIG. 2, the moving image distribution server device 10 that functions as a computer includes a communication unit 11, a storage unit 12, an input / output interface unit 13, and a system control unit 14. The system control unit 14 and the input / output interface unit 13 are connected via a system bus 15.

  The communication unit 11 is connected to the network 3 and controls communication with the content provider device 20 of the video provider, the advertisement provider device 30 of the advertisement provider, the viewing user terminal device 40, and the like.

  The storage unit 12 includes, for example, a hard disk drive, a silicon disk drive, and the like, and stores various programs such as an operating system and a server program, a moving image provided to a viewing user, and the like.

  Note that the various programs may be acquired from, for example, another server device via the network 3, or may be recorded on a recording medium and read via a drive device (not shown). Also good. Further, the distribution program or the like may be a program product.

  The storage unit 12 also includes a moving image database 12a (hereinafter referred to as “moving image DB 12a”) that stores moving images acquired from the content providing device 20, and an advertisement database 12b that stores advertisements acquired from the content providing device 20 ( Hereinafter referred to as “advertising DB 12b”), and a frame object information database 12c (hereinafter referred to as “frame / database”) that stores information (an example of object information related to the object) related to the object displayed in each image frame constituting the moving image. Object information DB 12c ”), a total object information database 12d (hereinafter referred to as“ total object information DB 12d ”) that stores object information aggregated for each video, and additional content such as advertisements for the video. Control to display Control information database 12e (hereinafter referred to as “control information DB12e”) for storing information (an example of display information), user information database 12f (hereinafter referred to as “user information DB12f”) for storing information relating to viewing users, and the like. Has been built.

  These databases may be configured with the same device or may be configured with different devices.

  Information related to a distributable moving image is registered in the moving image DB 12a (an example of a moving image storage unit). Specifically, as shown in FIG. 3, in the moving image DB 12a, a category ID, a moving image title, a moving image recording time, moving image data, and the like are registered for each moving image in association with the moving image ID. The moving image DB 12a may store the number of image frames constituting the moving image, the additional content display availability flag, the additional content display upper limit number, the additional content display upper limit time, and the like.

  The moving image ID is identification information for identifying a moving image. The moving image ID may be managed in association with the content provider ID for identifying the content provider.

  The category ID is information for identifying the category to which the moving image corresponding to the moving image ID belongs. In the information processing system 1, a plurality of categories are defined, and each moving image belongs to at least one category.

  The moving image data is data for displaying a moving image. The format of the moving image data is, for example, a codec such as MPEG-1, MPEG-2, MPEG-4, MPEG-7, MPEG-21, DivX format, Xvid format, Huffyuv format, AVI format, MOV format, A container format such as ASF format, Ogg format, Matroska format, etc. may be used.

  In the advertisement DB 12b, information related to a distributable advertisement is registered. Specifically, as shown in FIG. 4, advertisement display conditions, advertisement data, provision information URL (Uniform Resource Locator), etc. are registered for each advertisement in the advertisement DB 12b in association with the advertisement ID.

  The advertisement ID is identification information for identifying an advertisement.

  The advertisement display condition is an example of an additional content condition, and is a condition for displaying advertisement information as additional content on a moving image.

  The advertisement data is data that is displayed in association with the distributed video. For example, the advertisement data includes character data, still image data, moving image data, and the like, and may be a combination of at least two types of data among these data. The provided information URL is a URL of provided information associated with the advertisement indicated by the advertisement ID.

  In the frame / object information DB 12c, information related to objects extracted by image processing from image frames constituting a moving image is registered. Specifically, as shown in FIG. 5, in the frame object DB 12c, a moving image ID, a frame ID of an image frame, an object ID, frame object information related to an object existing in the image frame (an example of object information related to an object), etc. Is registered.

  The frame ID of the image frame is identification information for identifying the image frame in the moving image. For example, the frame ID may be a serial number (frame number) sequentially assigned to each image frame.

  The object ID is identification information for identifying the object extracted from the image frame. When it is determined that the objects extracted between different image frames are the same object, the same object ID is assigned.

  Here, as an example of an object, a building, a signboard, a moving object (vehicle, bicycle, person, animal, character, etc.), a tree, clothes worn by a person and a character, a bag displayed as an image frame Etc.

  As an example of the frame object information, the shape of the object extracted in the image frame, the position information where the object in the image frame is displayed (for example, coordinates with the upper left corner of the image frame as the origin, the image frame as a given area Among the divided areas, there are areas where the object is included, etc. Further, there are a center part such as the center of gravity of the object or each end part of the object as a reference part when indicating the position of the object itself. ), Information such as the area of the object display area of the object in the image frame, the coordinates of each pixel included in the object display area, and pixel information such as color.

  In the total object information DB 12d, object information (an example of object information related to objects) obtained by totaling frame object information of each image frame constituting a moving image is registered. Specifically, as illustrated in FIG. 6, the moving object ID, the object ID, the totaled object information, and the like are registered in the total object information DB 12 d.

  As an example of aggregated object information, the number of appearances (display count) of each object in the video, the average number of appearances of multiple objects that satisfy a given condition, the maximum number of appearances of the most frequently occurring object, For example, the minimum number of appearances. Further, instead of the number of appearances, display time (= number of appearances / frame rate) may be used. The frame rate is generally 30 fps (frames per second) to 60 fps, but may be any frame rate such as 10 fps, 24 fps, and 25 fps.

  Further, as an example of the aggregated object information, an integrated display area, an average display area, a maximum display area, a minimum display area, and the like of each object in a moving image can be given. Further, as an example of the aggregated object information, the number of consecutive appearances of objects displayed continuously in an image frame can be cited. Note that the aggregated object information is an example of an integrated display amount in which the object is displayed by integrating at least one of space and time in a moving image.

  In the control information DB 12e, control information for displaying an additional content such as an advertisement in each image frame constituting the moving image is registered. Specifically, as shown in FIG. 7, a moving image ID, control information, and the like are registered in the control information DB 12 e.

  The control information includes information indicating a temporal position at which additional content is displayed in a moving image (for example, a frame ID of an image frame) and information indicating a position within an image frame at which additional content is displayed at the time (for example, coordinate information). ) Etc. Further, the control information may include information for identifying the additional content and data (for example, advertisement data) of the additional content.

  When the additional content is a still image, the control information includes, for each frame ID, position information in an image frame to which the additional content is added, a reduction rate or an enlargement rate of the additional content, and the like. Alternatively, the control information includes, for each frame ID, position information (for example, coordinates with the upper left corner of the image frame as the origin) inside or in the outline of the additional content display area to which the additional content is added.

  When the additional content is a moving image, the control information includes, for each frame ID, image frame data of the additional content, position information in the image frame to which the additional content is added, a reduction rate or an enlargement rate of the additional content, and the like. Alternatively, the control information includes, for each frame ID, the image frame data of the additional content, the position of the pixel in the outline or in the additional content display area to which the additional content is added (an example of an area according to the object information of the selected object) Contains information.

  In the user information DB 12f, user information related to users registered as members in the information processing system 1 is registered for each user in association with the user ID. Specifically, as shown in FIG. 8, in the user information DB 12e, user attributes such as a password, a nickname, and an e-mail address are registered as user information in association with the user ID. The user ID is identification information for identifying the user.

  The storage unit 12 further stores various data such as an HTML document, an XML (Extensible Markup Language) document, image data, text data, and an electronic document for displaying a web page. The storage unit 12 stores various setting values, threshold values, reference values, specified values, constants, tables, and the like.

  The input / output interface unit 13 performs interface processing between the communication unit 11 and the storage unit 12 and the system control unit 14.

  The system control unit 14 includes a CPU 14a, a ROM (Read Only Memory) 14b, a RAM (Random Access Memory) 14c, and the like. The CPU 14a is an example of a processor. The present invention can also be applied to various processors different from the CPU. Each of the storage unit 12, the ROM 14b, and the RAM 14c is an example of a memory. The present invention can also be applied to various memories different from the hard disk, ROM, and RAM.

  The moving image distribution server device 10 may be configured by a plurality of server devices. For example, a server device that registers a moving image of content provided from the content providing device 20, a server device that distributes moving image content, a server device that receives and stores advertisements from the advertisement providing device 30, and a server that manages a database Devices and the like may be connected to each other via a LAN or the like.

  Next, functional blocks of the moving image distribution server device will be described with reference to FIG.

  FIG. 9 is a diagram illustrating an example of functional blocks of the video distribution server device 10 according to an embodiment.

  As shown in FIG. 9, the moving image distribution server device 10 includes an image frame acquisition unit 10a, an object specification unit 10b, an object information calculation unit 10c, an object selection unit 10d, and a display information generation unit 10e.

  The image frame acquisition unit 10a acquires at least a part of image frames among a plurality of image frames constituting the moving image with respect to the moving image acquired from the moving image DB 12a storing the moving image.

  The object specifying unit 10b specifies at least one object displayed in the image frame for each image frame acquired by the image frame acquiring unit 10a.

  The object information calculation unit 10c calculates object information related to the object appearing in a plurality of image frames for each object specified by the object specifying unit 10b.

  The object selection unit 10d includes the object specified by the object specifying unit 10b based on the object information calculated by the object information calculation unit 10c and the additional content display condition for displaying the additional content on the moving image. To select an object to display additional content.

  The display information generation unit 10e generates display information for displaying the additional content in an area including at least a part of the object selected by the object selection unit 10d.

(2.2 Configuration and Function of Content Providing Device 20)
Next, the configuration and function of the content providing apparatus 20 will be described with reference to FIG.

  FIG. 10 is a block diagram illustrating an example of a schematic configuration of the content providing apparatus 20.

  As shown in FIG. 10, the content providing apparatus 20 that functions as a computer is, for example, a personal computer or a server. The content providing apparatus 20 includes a communication unit 21, a storage unit 22, a display unit 23, an operation unit 24, an input / output interface unit 25, and a system control unit 26. The system control unit 26 and the input / output interface unit 25 are connected via a system bus 27. The content providing device 20 may be a computer such as a smartphone or a tablet.

  The communication unit 21 controls communication with the moving image distribution server device 10 and the like through the network 3. The storage unit 22 includes, for example, a hard disk drive and stores programs such as an operating system.

  The display unit 23 (an example of a display screen for each community) is configured by, for example, a liquid crystal display element or an EL (Electro Luminescence) element.

  The operation unit 24 is configured by, for example, a keyboard and a mouse.

  The input / output interface unit 25 is an interface between the communication unit 21 and the storage unit 22 and the system control unit 26.

  The system control unit 26 includes, for example, a CPU 26a, a ROM 26b, and a RAM 26c. In the system control unit 26, the CPU 26 a reads out and executes various programs stored in the ROM 26 b, the RAM 26 c, and the storage unit 22.

(2.3 Configuration and Function of Advertisement Providing Device 30)
Next, the configuration and function of the advertisement providing apparatus 30 will be described with reference to FIG.

  FIG. 11 is a block diagram illustrating an example of a schematic configuration of the advertisement providing apparatus 30.

  As shown in FIG. 11, the advertisement providing apparatus 30 that functions as a computer is, for example, a portable terminal such as a portable wireless telephone or a PDA including a personal computer or a smartphone. The advertisement providing device 30 includes a communication unit 31, a storage unit 32, a display unit 33, an operation unit 34, an input / output interface unit 35, and a system control unit 36. The system control unit 36 and the input / output interface unit 35 are connected via a system bus 47. The configuration and function of the advertisement providing device 30 are substantially the same as the configuration and function of the content providing device 20, but the system control unit 36 displays an advertisement as advertisement information via the communication unit 31. The advertisement display conditions, advertisement data, provided information URL, and the like are transmitted to the moving image distribution server device 10. The display unit 33 displays an advertisement display condition input screen for inputting advertisement display conditions.

(2.4 Configuration and Function of Viewing User Terminal Device 40)
Next, the configuration and functions of the viewing user terminal device 40 will be described with reference to FIG.

  FIG. 12 is a block diagram illustrating an example of a schematic configuration of the viewing user terminal device 40.

  As shown in FIG. 12, the viewing user terminal device 40 that functions as a viewing user's computer is, for example, a portable terminal such as a portable wireless telephone or a PDA including a personal computer or a smartphone. The viewing user terminal device 40 includes a communication unit 41, a storage unit 42, a display unit 43, an operation unit 44, an input / output interface unit 45, and a system control unit 46. The system control unit 46 and the input / output interface unit 45 are connected via a system bus 47. Note that the configuration and function of the viewing user terminal device 40 are substantially the same as the configuration and function of the content providing device 20, but the system control unit 46 transmits the video from the video distribution server device 10 via the communication unit 41. The content and the additional content are received and displayed on the display unit 43.

[3. Operation of information processing system]
Next, the operation of the information processing system 1 according to one embodiment of the present invention will be described with reference to the drawings.

(3.1 Example of video registration processing)
First, an operation example of the moving image registration process in the moving image distribution server device 10 will be described with reference to FIGS.

  FIG. 13 is a schematic diagram illustrating an example of a frame number of an image frame in moving image data. 14A to 14I are schematic diagrams illustrating an example of an image frame. FIG. 15 is a flowchart illustrating an example of a moving image registration process of the system control unit 14 of the moving image distribution server apparatus 10 according to an embodiment.

  First, in order to register content (moving image) to be distributed in the moving image distribution service of the site operated on the moving image distribution server device 10, each content providing device 20 uses the content provider to register for a moving image. The moving image category ID, title, recording time, moving image data, and the like are transmitted to the moving image distribution server device 10 together with the content provider ID for identifying the content provider. In addition, when the information regarding the moving image to be registered is transmitted to the moving image distribution service after the content provider logs in, it is not necessarily transmitted together with the content provider ID.

  For example, in the case of a moving image with a recording time of 10 minutes and an image with a frame rate of 30 fps, the moving image data is composed of 18,000 image frames. As shown in FIG. 13, the image frames in the moving image data are arranged in the order of frame numbers. As shown in FIGS. 14A to 14I, an image frame 50 corresponds to each frame number (frame ID).

  Here, as an example of each image frame, as shown in FIGS. 14A to 14I, frame number “000,001”, frame number “000,015”, frame number “000,060”, frame number “000,120”, frame number “000,135”, Examples include an image frame 50 having frame numbers “000,165”, frame numbers “000,240”, frame numbers “000,330”, and frame numbers “000,360”.

  Next, as shown in FIG. 15, the moving image distribution server device 10 acquires moving image information (step S1). Specifically, the system control unit 14 of the moving image distribution server device 10 receives the moving image information transmitted from each content providing device 20.

  Next, the moving image distribution server device 10 issues a moving image ID to the acquired moving image (step S2). Specifically, when the system control unit 14 of the moving image distribution server device 10 receives moving image information from the content providing device 20, the moving image ID is generated for the moving image. The video ID may be a number indicating the reception order or an ID obtained by combining the content provider ID with the reception time. Note that the timing of issuing the moving image ID is not limited to the time of acquisition, and may be when registering the acquired moving image in the moving image DB 12a in the next step S3.

  Next, the moving image distribution server device 10 stores the moving image information and the moving image ID in association with each other in the moving image DB 12a (step S3). Specifically, the system control unit 14 records the category ID, title, recording time, moving image data, and the like in the moving image DB 12a in association with the moving image ID.

  The moving image distribution server device 10 receives moving image information from each content providing device 20 at any time, and registers the received moving image information together with the moving image ID in the moving image DB 12a. The moving image distribution server device 10 may receive moving image information from the content providing device 20 and temporarily store the moving image information to some extent, and register the moving image information in the moving image DB 12a at a predetermined timing.

(3.2 Operation example of advertisement registration processing)
An operation example of advertisement registration processing in the moving image distribution server device 10 will be described with reference to FIG.

  FIG. 16 is a flowchart illustrating an example of advertisement registration processing of the system control unit 14 of the video distribution server device 10 according to an embodiment.

  First, in order to register an advertisement that an advertisement provider desires to display together with a video on a site operated on the video distribution server device 10, each advertisement providing device 30 displays an advertisement as advertisement information. The advertisement display condition, advertisement data, provision information URL, and the like are transmitted to the moving image distribution server device 10 together with the advertisement provider ID.

  The advertisement display condition is input from the advertisement display condition input screen on the display unit 33 of the advertisement providing device 30, for example.

  Next, as shown in FIG. 16, the moving image distribution server device 10 acquires advertisement information (step S4). Specifically, the system control unit 14 of the video distribution server device 10 receives advertisement display conditions, advertisement data, provision information URL, advertisement provider ID, and the like as advertisement information transmitted from each advertisement provision apparatus 30. .

  Next, the moving image distribution server device 10 issues an advertisement ID to the acquired advertisement (step S5). Specifically, when the system control unit 14 of the moving image distribution server device 10 receives an advertisement from the advertisement providing device 30, it generates an advertisement ID for the advertisement. The advertisement ID may be a number indicating the order of reception or an ID obtained by combining the advertisement provider ID with the reception time.

  Next, the moving image distribution server device 10 associates the advertisement information including at least the advertisement display condition with the advertisement ID and stores them in the moving image DB (step S6). Specifically, the system control unit 14 records advertisement display conditions, advertisement data, provided information URL, and the like in the advertisement DB 12b in association with the advertisement ID.

  Note that the video distribution server device 10 accepts advertisements from the respective advertisement providing devices 30 as needed, and registers the received advertisement information together with the advertisement ID in the advertisement DB 12b.

(3.3 Example of object identification processing)
An operation example of the object specifying process in the moving image distribution server device 10 will be described with reference to FIGS.

  FIG. 17 is a flowchart illustrating an example of the object specifying process of the system control unit of the video distribution server device according to the embodiment. 18A to 18D are schematic diagrams illustrating an example of a feature point image of an image frame. 19A to 19D are schematic diagrams illustrating an example of an object display area in an image frame.

  As illustrated in FIG. 17, the moving image distribution server device 10 acquires one moving image from among a plurality of moving images stored in the moving image DB 12a (step S10). Specifically, the system control unit 14 of the moving image distribution server device 10 refers to the moving image DB 12a by moving image ID of a moving image that has not yet been subjected to object identification processing, and moving image data of one moving image corresponding to the moving image ID. Is read. The system control unit 14 may temporarily store the moving image data read from the moving image DB 12a in the RAM 14c or the storage unit 12 for work.

  Next, the moving image distribution server device 10 acquires at least one image frame among a plurality of image frames included in the acquired moving image (step S11). Specifically, the system control unit 14 extracts an image frame corresponding to the processing target frame ID, for example.

  Next, the moving image distribution server device 10 extracts and specifies an object included in the image frame from the acquired image frame (step S12). Specifically, the system control unit 14 extracts each object in the image frame from, for example, edge extraction by background difference method, differentiation processing, SIFT (Scale-invariant feature transform), or HoG (Histogram of Oriented Gradients). ) Etc. to extract and specify. Note that the object may be tracked by the particle filter method, and the object may be specified by the particle.

  Through the processing in step S12, each object such as a building, a signboard, a moving body (vehicle, bicycle, person, animal, etc.), a tree, clothes, and the like that are reflected in the image frame 50 is specified, and each object has an object ID. Assigned.

  Note that, as shown in any one of FIGS. 18A to 18D, a feature point image 51 such as an edge is obtained and an object is extracted on the image frame 50 by edge detection or the like.

  The system control unit 14 specifies an object based on the feature point image 51. For example, based on the edge groups ed1, ed2, ed3, ed4, ed5, and ed6 of the feature point image 51 of the frame number “000,001” in FIG. 18A, the system control unit 14 sets the frame number “000,001” as shown in FIG. 14A. Are extracted from the image frame 50 of the building, the object ob1 of the building, the object ob2, the object ob3 of the building, the bird object ob4, the tree object ob5, and the vehicle object ob6, and an object ID is assigned to each object. Is identified.

  In addition, when it is determined that the object is the same as the object included in the other image frame based on the comparison with the object included in the other image frame whose frame number is within the predetermined range, the other image frame May be assigned the same object ID as the object ID assigned to the object included in.

  For example, the same object ID is assigned to each of the objects ob1, ob2, ob3, ob4, ob5, and ob6 in FIG. 14A and the objects ob1, ob2, ob3, ob4, ob5, and ob6 in FIG. 14B.

  As shown in FIG. 14E, when the object ob3 and the object ob13 are determined to be the same object by object tracking, object shape similarity, or the like, as shown in FIG. 14F, the same object ID is assigned. It may be assigned. As shown in FIG. 14E, when the object ob6 and the object ob16 are determined to be the same object by object tracking, object shape similarity, or the like, as shown in FIG. 14F, the same object ID is assigned. It may be assigned.

  As described above, the moving image distribution server device 10 functions as an example of an object specifying unit that specifies an object displayed in the image frame for each acquired image frame. Note that the moving image distribution server device 10 may specify a plurality of objects displayed in the image frame for each acquired image frame. In addition, when the object cannot be specified, such as the object is too small, unclear, or unclear, the moving image distribution server device 10 may not specify the object. Moreover, the moving image distribution server apparatus 10 may specify an object with a given upper limit number as a limit. For example, the moving image distribution server device 10 may preferentially specify an object having a large display area in the image frame.

  Next, the moving image distribution server device 10 specifies an object display area including the specified object (step S13). Specifically, the system control unit 14 sets an object display area corresponding to the area in which the object is displayed according to the extracted feature points such as edges.

  For example, as shown in FIGS. 18A and 19A, the system control unit 14 specifies the outline of the object ob1, that is, the object display area ar1 having the edge group ed1 of the object ob1 as the outer edge, and the outline of the object ob2, that is, An object display area ar2 whose outer edge is the edge group ed2 of the object ob2 is specified.

  Further, as shown in FIGS. 18A and 19A, the system control unit 14 includes a range including at least a part of the edge group ed3 of the object ob3 corresponding to a predetermined shape (for example, a rectangle). The object display area ar3 may be set in For example, the system control unit 14 searches for a place where a rectangular frame matches the edge image in FIG. 18A or the like, and sets an object display area.

  Further, as shown in FIGS. 18A and 19A, the object display area may be an object display area ar4 including all the objects ob4. The predetermined shape may be a circle, an ellipse, or a polygon such as a rectangle. The object display area ar4 may be determined so as to be inscribed or circumscribed with the object ob4.

  As shown in FIGS. 18A and 19A, a predetermined shape inscribed in at least a part of the edge group ed5 of the object ob5 may be used as the object display area ar5.

  Further, as shown in FIGS. 18A and 19A, a predetermined shape including at least part of the edge group ed6 of the object ob6 may be used as the object display area ar6.

  In addition, the size of the object in the moving image changes depending on the perspective with respect to the viewpoint of the camera by the zoom function of the camera, or, as shown in FIG. 19B or 19C, the shape of the object changes, When the shape of the object itself changes, the system control unit 14 may set the object display area so that the shape of the object display area ar6 changes in accordance with the change of the object.

  Specifically, the system control unit 14 determines that, for example, the object parameters such as the position, color, shape, and size of the object are within a predetermined range between the objects in the image frames close in time. Are determined to be the same object. Objects that fall within a predetermined range with respect to the size of the object and are changing are determined to be the same object. Objects that fall within a predetermined range with respect to the shape of the object are determined to be the same object (for example, the coordinates of the contour of one object and the contours of other objects with respect to the center of the object). When the minimum coordinate and distance are within the specified range). Note that the system control unit 14 may weight each parameter. The system control unit 14 may change the predetermined range depending on the time difference between frames.

  As shown in FIGS. 19A to 19D, when the object display area is rectangular, the aspect ratio changes as in the object display area ar6 and the object display area ar16.

  Also, as shown in FIGS. 14E to 14F, when the size of the object changes due to the change in the zoom (magnification) of the camera capturing the image, as shown in FIGS. 19B to 19C, the system control unit 14 may change the size of the object display area in accordance with the change of the object. The system control unit 14 may determine that the object display area ar3 and the object display area ar13 are the same object display area, or may be determined as separate object display areas.

  Further, the object display areas may overlap each other. For example, as shown in FIG. 19C, the object display area ar12 and the object display area ar16 may overlap.

  Further, the object display areas may not overlap each other. For example, as illustrated in FIG. 19D, the system control unit 14 sets each object display area so that the object display area ar12 and the object display area ar16 do not overlap.

  As shown in FIG. 19E, the range in which the object display area is set includes, for example, a protruding portion of the edge group ed3 from the edge group ed3, such as the object display area ar23, instead of the object display area ar3. The edge group ed3 may be completely included as in the object display area ar33.

  The system control unit 14 may specify a rectangular portion (object display area ar3) surrounded by the edge group ed3 as a further different object. Further, the system control unit 14 may specify the object display area ar23 and the object display area ar33 as further different objects.

  As illustrated in FIG. 19F, the system control unit 14 may set the object display area ar06 in a portion where an advertisement in a part of the object ob6 is easily inserted. For example, the system control unit 14 searches the inside of the object ob6 based on the feature point image of the object ob6, and sets the object display area ar06 in a portion where the rectangle is most suitable. Specifically, the system control unit 14 scans a rectangular frame having a predetermined size in the vicinity of the object ob6 to obtain the maximum degree of overlap with the edge group, and further determines the size of the rectangular frame. Increase the scan a little and find the maximum degree of overlap with the edge group. The system control unit 14 stops the rectangular frame scanning when the maximum degree of overlap starts to decrease to some extent. The system control unit 14 sets the size of the rectangular frame having the maximum value among the maximum overlapping degrees and the position of the rectangular frame having the maximum value as the object display area.

  As illustrated in FIG. 19F, the system control unit 14 may set the object display area ar26 inside a contour having the outer edge of the object ob6 as a contour. In this case, the object itself becomes an object display area.

  Next, the moving image distribution server device 10 calculates information related to the object display area (step S14). That is, after the object display area is set, the moving image distribution server device 10 calculates information on the object display area such as the coordinates and area of the position included in the object display area, thereby obtaining object information for each image frame 50. The frame object information is calculated.

  Here, as an example of the frame object information, the coordinates (xy coordinates) that serve as the reference of the object display area, for example, when the object display area is a rectangle, the coordinates of any of the four corners of the rectangle or the coordinates of the center of the rectangle If the object display area is a circle such as an ellipse, for example, there are coordinates of the center of the circle, coordinates of the center of gravity, etc. The coordinates of the pixel near the center of gravity, the coordinates of the pixel at the center, and the like can be given.

  Also, as an example of the frame / object information, the area and shape of the object display area, the coordinates of the pixels of the outline of the object display area, the coordinates of all the pixels included in the object display area, or the coordinates of a plurality of pixels, etc. It is done.

  The frame object information includes the aspect ratio, the vertical length and the horizontal length when the object display area is rectangular, and the ratio between the long axis and the short axis, the length of the long axis, and the short when the shape is an ellipse. It may be the length of the shaft.

  The frame object information may include the color of the pixel in the object display area.

  Further, in the frame object information, when the outer edge of the object display area is the outline of the object, that is, the object display area may be the area of the object itself.

  In addition, you may perform the process of step S12 and step S13 together. That is, the system control unit 14 may set the object itself extracted in step S12 as an object display area.

  Note that the moving image distribution server device 10 may perform the processing from step S12 to step S14 at a time. For example, the position information of the object when the system control unit 14 extracts and specifies the object in step S12 may be object display area information, that is, object information.

  As described above, the moving image distribution server device 10 functions as an example of an object information calculation unit that calculates object information related to the object displayed in each image frame for each identified object.

  Next, the moving image distribution server device 10 stores the moving image ID, the frame ID for identifying the image frame, the information on the object display area, and the object ID for identifying the object in association with each other (step S15). Specifically, the system control unit 14 associates the moving image ID, the frame ID, information on each object display area (frame / object information), and the object ID, and stores them in the frame / object information DB 12c. .

  The system control unit 14 associates the moving image ID, the frame ID, information on each object display area (frame / object information), and the object ID, and temporarily stores them in a cache memory such as the RAM 14c. May be.

  Next, the moving image distribution server device 10 determines whether or not processing has been performed on all image frames (step S16). For example, when processing a moving image in order from the first image frame, the system control unit 14 determines whether the frame number that is the frame ID is the last frame number of the moving image. Note that the moving image distribution server device 10 may determine whether or not processing has been performed on a part (for example, half) of image frames.

  When the processing is not performed on all the image frames (step S16; NO), the moving image distribution server device 10 returns to the processing of step S11 and acquires the next image frame 50.

  In this way, the moving image distribution server device 10 functions as an example of an image frame acquisition unit that acquires at least some of the image frames included in the moving image acquired from the moving image storage unit that stores the moving image. To do.

  When the process is performed on all the image frames (step S16; YES), the moving image distribution server device 10 ends the object specifying process.

  Next, the moving image distribution server device 10 performs the processing from step S10 to step S16 based on the moving image ID of another moving image, and constructs the frame / object information DB 12c.

(3.4 Example of control information generation processing)
Next, an operation example of control information generation processing in the moving image distribution server device 10 will be described with reference to FIGS. 20 to 22.

  FIG. 20 is a flowchart illustrating an example of control information generation processing of the system control unit of the video distribution server device according to the embodiment. 21A to 21F are schematic diagrams illustrating an example of an image frame to which an advertisement is added. 22A to 22D are schematic diagrams illustrating an example of an image frame partially masked.

  First, the moving image distribution server device 10 extracts a moving image ID of a moving image that has not been subjected to the control information generation process.

  As illustrated in FIG. 20, the moving image distribution server device 10 refers to the frame / object information DB 12c, and information on the object display area of an image frame having a matching moving image ID and a different frame ID, and an object for identifying the object Object information for each object is calculated based on the ID (step S20).

  Specifically, the system control unit 14 searches the frame / object information DB 12c for the same moving image ID in the order of frame number (frame ID), and totals the frame / object information for each object ID. The calculated object information is calculated.

  Here, as a counting method, for example, the system control unit 14 adds the number of appearances (display count: display frame count) as an example of object information for each object ID in the order of frame numbers. For example, as illustrated in FIG. 14A, the system control unit 14 counts the number of appearances for each of the object ob1, the object ob2, the object ob3, the object ob4, the object ob5, and the object ob6. Further, as shown in FIG. 14C, since the bird object ob4 does not exist in the frame number “000,060”, the system control unit 14 assigns the object ob1, the object ob2, the object ob3, the object ob5, and the object ob6 of the signboard. In contrast, each counts one.

  As shown in FIG. 14F, the system control unit 14 adds the number of appearances to the object ob12, the object ob13, and the object ob16, but determines that the object ob13 and the object ob3 are the same object. (For example, when the same object ID is set), the system control unit 14 adds the number of appearances as the same object ob3. Note that the method for determining the same object is that the system control unit 14 determines that each parameter in the object parameters such as the position, color, shape, size, etc. The objects within a predetermined range are determined to be the same object. Then, the system control unit 14 adds the number of appearances between the same objects.

  The system control unit 14 performs processing on all or some of the image frames, and calculates the total number of appearances (an example of the integrated display amount included in the object information) for each object. As an example of the integrated display amount, the total display time (= total number of appearances / frame rate may be calculated.

  Further, the system control unit 14 may calculate the continuous appearance number (an example of the integrated display amount) of the appearing object in the continuous image frames. For example, the system control unit 14 counts the number of appearances in the order of frame numbers, and when a certain object does not appear in a certain image frame, records the added count value and resets the counter. When the object appears again, it starts again from zero.

  Further, the system control unit 14 calculates the maximum and minimum number of continuous appearances (number of continuous appearances / frame rate = continuous display time) in the recorded count value. Further, the continuous appearance number may be an average of the recorded count values.

  When the system control unit 14 sums up all or some of the frame numbers, it divides by the number of frames to calculate the appearance ratio. For example, the appearance ratio is 8000/18000 for the object ob1, 4000/18000 for the object ob2, 10000/18000 for the object ob3, 35/18000 for the object ob4, 100/18000 for the object ob5, 9000/18000 for the object ob6, etc. .

  As a counting method, the system control unit 14 may add the size (area) of each object for each object ID in the order of frame numbers. As shown in FIG. 19A, when the system control unit 14 has a frame number “000,000”, the area of the object display area ar1 of the object ob1, the area of the object display area ar2 of the object ob2, the area of the object display area ar3 of the object ob3, The area of the object display area ar4 of the object ob4, the area of the object display area ar5 of the object ob5, and the area of the object display area ar6 of the object ob6 are added for each object ID. As shown in FIG. 19B, when the frame number is “000,135”, the system control unit 14 determines the area of the object display area ar1 of the object ob1, the area of the object display area ar2 of the object ob2, the area of the object display area ar3 of the object ob3, The area of the object display area ar5 of the object ob5 and the area of the object display area ar6 of the object ob6 are added for each object ID.

  The system control unit 14 adds up the area of the object display area for each object with respect to all or some of the frame numbers, and calculates an integrated display area (an example of an integrated display amount). Note that the system control unit 14 may calculate an average display area (an example of an integrated display amount) obtained by dividing the integrated display area by all or part of the number of frames. Further, the system control unit 14 may calculate the maximum and minimum display areas for each object.

  Further, the system control unit 14 may calculate the continuous display area (an example of the integrated display amount) of the objects that appear in the continuous image frames as if the number of continuous appearances was calculated. For example, when the system control unit 14 accumulates the display areas in the order of frame numbers and no object appears in a certain image frame, the accumulated value of the accumulated display areas is recorded, and the accumulated value When the object appears again after resetting, the integration is restarted from zero.

  The system control unit 14 may total the frame object information for each predetermined period, not the entire moving image. For example, the system control unit 14 calculates the total object information by dividing the frame object information at the time of chapter information or when the image is switched.

  Further, the system control unit 14 may calculate object information for each group by grouping a plurality of objects as an example of the integrated display amount. For example, the system control unit 14 determines whether the shape or the like of each object is the same or similar, and assigns the same group ID to objects having the same or similar shape. Then, the system control unit 14 counts the number of appearances in group units (counts when at least one object belonging to the group appears, counts when at least a predetermined number of objects appear), display area in group units (objects belonging to the group) The total display amount of the group unit such as the area of the display area is added) is calculated.

  It should be noted that the method for determining whether or not each object is the same or similar is that the system control unit 14 determines that the object such as the position, color, shape, size, etc. Among the parameters, each parameter determines that objects within a predetermined range are the same or similar objects. Then, the system control unit 14 groups the same or similar objects.

  Note that the system control unit 14 may calculate an integrated display amount in units of groups for each group of similar objects having the same or similar colors. In addition, the system control unit 14 performs grouping for each category to which an object such as a stationary object such as a building, a signboard, or a tree, a moving object such as a vehicle, a person, or an animal belongs, and calculates an integrated display amount for each group. Also good.

  As described above, the moving image distribution server device 10 functions as an example of an object information calculation unit that calculates object information related to the object displayed in each image frame for each identified object.

  Next, the system control unit 14 stores the aggregated object information in the aggregate object information DB 12d in association with the moving image ID and the object ID. Further, the frame number or frame ID such as the frame number of the aggregated period, the frame number indicating the maximum display area of the object, the frame number indicating the minimum display area of the object, etc. At the same time, it may be stored in the total object information DB 12d. The system control unit 14 may also store the object shape, the aspect ratio in the case of a rectangle, and the ratio of the major axis and the minor axis in the case of an ellipse in the aggregate object information DB 12d together with the object information. Further, the system control unit 14 may store the coordinates of the most frequently appearing position of the object in the moving image and the coordinate information of the object of the average appearing position in the total object information DB 12d together with the object information.

  Next, the moving image distribution server device 10 selects an object for displaying an advertisement based on the object information for each object and the advertisement display condition set by the advertiser of the advertisement to be displayed (step S21). Specifically, the system control unit 14 refers to at least one of the frame / object information DB 12c and the aggregate object information DB 12d, and compares the advertisement display condition with the object information to extract the object.

  For example, when the advertisement display condition of the advertisement “AD-a1” is that the advertisement has a rectangular shape, a display time of 3 seconds or more, a display area of a predetermined value or more, and a display on a moving object on the screen, FIG. To the object ob6 as shown in FIG. 21F. The system control unit 14 selects the object ob6 with reference to at least one of the frame object information DB 12c and the aggregate object information DB 12d.

  For example, when the advertisement display condition of the advertisement “AD-b1” is within 4 seconds of the first half of the moving image, the shape of the advertisement is rectangular, and the upper part of the image frame, as shown in FIGS. 21A to 21F, the object ob1 Is applicable.

  For example, when the advertisement display condition of the advertisement “AD-d1” is that the aspect ratio of the advertisement shape is long and narrow (for example, a bar shape with an aspect ratio of 5: 1) and the lower side of the screen is 10 seconds or longer. 21A to 21F, in the objects ob3 and ob13 (the object ob3 in FIGS. 14A to 14E and the object ob13 in FIGS. 14F to 14I), the object display areas ar3 and ar13 (the objects in FIGS. 19A and 19B) This corresponds to the lower side of the display area ar3 and the object display area ar13) of FIGS. 19C and 19D.

  When an object for displaying an advertisement is selected, the system control unit 14 determines, in each image frame for displaying the advertisement, a ratio between the object display area of the object to which the advertisement is added and the size of the reference advertisement (enlargement rate or (Reduction rate) may be calculated and stored in the frame object information DB 12c, the aggregate object information DB 12d, or the like.

  Further, the system control unit 14 refers to at least one of the frame object information DB 12c and the total object information DB 12d, and has a relatively large number of appearances among the appearance numbers of each object (the order of the appearance numbers is predetermined from the top). Several objects may be selected. In this case, the system control unit 14 may select an object based on the advertisement display condition, or may select an object based only on the object information without being based on the advertisement display condition. When the system control unit 14 is also based on the advertisement display condition, the object that satisfies the advertisement display condition is selected from the objects having a relatively large number of appearances.

  Further, the system control unit 14 refers to at least one of the frame object information DB 12c and the total object information DB 12d, and the average display area is relatively large among the average display areas of the respective objects (the rank of the average display area is A predetermined number of objects from the top may be selected. In this case, the system control unit 14 may select an object based on the advertisement display condition, or may select an object based only on the object information without being based on the advertisement display condition. When the system control unit 14 is also based on the advertisement display condition, the object that satisfies the advertisement display condition is selected from objects having a relatively large average display area.

  In addition, the system control unit 14 refers to at least one of the frame object information DB 12c and the total object information DB 12d, and among the integrated display numbers of each object, the integrated display number is relatively large (the order of the integrated display number is A predetermined number of objects from the top may be selected. In this case, the system control unit 14 may select an object based on the advertisement display condition, or may select an object based only on the object information without being based on the advertisement display condition. When the system control unit 14 is also based on the advertisement display condition, the object that satisfies the advertisement display condition is selected from the objects having a relatively large cumulative display number.

  Further, the system control unit 14 refers to at least one of the frame object information DB 12c and the aggregate object information DB 12d, and has a relatively large number of continuous appearances among the continuous appearance numbers of each object (the rank of the continuous appearance number is A predetermined number of objects from the top may be selected. In this case, the system control unit 14 may select an object based on the advertisement display condition, or may select an object based only on the object information without being based on the advertisement display condition. When the system control unit 14 is also based on the advertisement display condition, the object that satisfies the advertisement display condition is selected from objects having a relatively large number of consecutive appearances.

  Further, the system control unit 14 may select an object when the difference data amount of the object (or object display area) between the image frames is larger than a predetermined threshold. The system control unit 14 sets a predetermined threshold according to the difference data amount of each object. For example, in step S20, the system control unit 14 calculates, as object information for each object, the compression rate of the moving image (or the amount of data reduced when the moving image is encoded) when each object is separately painted with a single color. To do. Then, in step S21, the system control unit 14 compares the compression ratios for each object, and selects an object having a relatively high compression ratio (the object having a predetermined compression order from the object having the highest compression ratio). Good.

  In this case, the system control unit 14 may select an object based on the advertisement display condition, or may select an object based only on the object information without being based on the advertisement display condition. When the system control unit 14 is also based on the advertisement display condition, the object that satisfies the advertisement display condition is selected from the objects that are larger than the predetermined threshold.

  Here, for distribution, the system control unit 14 converts the moving image content into distribution moving image data and compresses it using a predetermined encoding method. As a moving image coding method, there is a coding method using a difference between frames, and inter-frame prediction is used in order to increase a compression rate. The system control unit 14 encodes a predetermined image frame using a difference between a previous image frame and a predicted image frame from at least one image frame of the subsequent image frame. Furthermore, in order to perform motion compensation that compensates for motion of an object or the like between image frames, the system control unit 14 also obtains a motion vector and encodes the motion vector. Difference information between one image frame and a predicted image frame, information on a motion vector, and the like are examples of difference data.

  As described above, when an object whose difference data amount is larger than the predetermined threshold is selected, and the selected object is filled with a single color, the object with a large amount of information is erased, and the code of the video is deleted. The compression rate by the conversion method increases, and the data volume decreases.

  The system control unit 14 may select an object by combining at least two of the number of appearances, the average display area, the integrated display amount, the number of continuous appearances, and the difference data amount.

  Further, the system control unit 14 may select an object by combining at least two of advertisement display conditions, such as an advertisement display time, an advertisement display area, and an advertisement shape. Further, the system control unit 14 combines at least two of the number of appearances, the average display area, the cumulative display amount, the continuous appearance number, the difference data amount, the advertisement display time, the advertisement display area, the advertisement shape, etc. May be selected.

  By combining these object information and advertisement display conditions, an object that meets various needs of the advertisement provider can be selected accurately.

  As described above, the object for the moving image distribution server device 10 to display the additional content based on the calculated object information and the additional content display condition for displaying the additional content to be added to the moving image. As an example of an object selection means for selecting from the identified objects. In addition, the moving image distribution server device 10 functions as an example of an object selection unit that selects an object for displaying the additional content according to the integrated display amount. Further, the moving image distribution server device 10 functions as an example of an object selection unit that selects an object for displaying the additional content according to the number of continuous displays.

  Next, the moving image distribution server device 10 generates control information for displaying an advertisement so as to correspond to the selected object (step S22). Specifically, the system control unit 14 refers to at least one of the frame object information DB 12c and the aggregate object information DB 12d, and displays each frame ID for displaying an advertisement based on the object ID of the selected object, and these images. The coordinate information of the object in the frame is acquired, and control information including each frame ID and coordinate information corresponding to these frame IDs is generated. For example, as illustrated in FIGS. 21A to 21F, the system control unit 14 generates control information so that a moving image including the image frames 52 is reproduced.

  This control information is information for displaying an advertisement superimposed on an object appearing in the moving image when the moving image is reproduced. The area in which the advertisement is superimposed and displayed is an advertisement display area (an additional content display area when the additional content is an advertisement), which is an example of an area according to the object information of the selected object.

  The advertisement display area includes at least a part of the selected object or the object display area. For example, the system control unit 14 may set the advertisement display area ad-ar1 to coincide with the object display area (ar1 in FIGS. 19A and 19B) as illustrated in FIGS. 21A to 21C. Further, the system control unit 14 adapts the advertisement display area ad-ar2 to a part of an edge group (ed2 in FIGS. 18A and 18B) as an example of object information, as shown in FIGS. 21A to 21C. Alternatively, it may be set inside the object display area (ar2). Further, the system control unit 14 adapts the advertisement display area ad-ar3 to a part of the edge group (ed3) which is an example of the object information as shown in FIG. 21A to FIG. 21C and satisfies the advertisement condition. Thus, it may be set inside the object display area (ar3).

  Further, the system control unit 14 sets the advertisement display area ad-ar5 inside the object display area (ar5) so as to be inscribed in the edge group (ed5) which is an example of the object information as illustrated in FIG. 21A. May be. Further, the system control unit 14 may set the advertisement display area ad-ar5 so as to partially protrude from the object display area (ar5) as shown in FIGS. 21B and 21C.

  Further, as shown in FIGS. 21A and 21B, the system control unit 14 adjusts the object display area ad-ar6 so that it matches a part of the edge group (ed6) which is an example of the object information. ar6) may be set.

  In addition, as shown in FIG. 21C, when a part of the object (ob6 in FIG. 14E) is hidden, the system control unit 14 sets the advertisement display area ad-ar6 only in the part where the object (ob6) is reflected. May be.

  Further, as shown in FIGS. 21C to 21D, when the scene is changed, the system control unit 14 changes the advertisement display area ad-ar2 to the advertisement display area ad-ar12, the advertisement according to the change in the size of the object. The size of the advertisement display area may be changed from the display area ad-ar3 to the advertisement display area ad-ar13 and from the advertisement display area ad-ar6 to the advertisement display area ad-ar16.

  Moreover, regarding the display of the advertisement in the advertisement display area, as shown in FIGS. 21A and 21B, the system control unit 14 controls to change the advertisement displayed in the advertisement display area ad-ar5 according to the advertisement display condition. Information may be generated.

  Further, regarding the display of the advertisement in the advertisement display area, as shown in FIGS. 21A to 21C, when the area of the advertisement display area ad-ar6 does not consider the advertisement display condition, as shown in FIG. 21C, the system control unit 14 may generate the control information so as to delete the advertisement “AD-a1”. Further, as shown in FIGS. 21E and 21F, when the area of the advertisement display area ad-ar12 stops considering the advertisement display condition, the system control unit 14 controls to delete the advertisement “AD-e2”. Information may be generated.

  Further, regarding the display of the advertisement in the advertisement display area, as shown in FIGS. 21E to 21F, when the area of the advertisement display area ad-ar16 satisfies the advertisement display condition, as shown in FIG. 21F, the system The control information may be generated so that the control unit 14 displays the advertisement “AD-a2”.

  As an example of display information for displaying the additional content in an area in accordance with the object information of the selected object, including at least a part of the selected object, the control information includes an outline of the object display area. Instead of the coordinate information such as the above, coordinate information of one point on the outer edge or inside of the object and information on the size of the object display area (or advertisement display area) may be included. In this case, the system control unit 14 changes the size of the advertisement according to the size of the object display area (or advertisement display area), and displays the control information so that the advertisement is displayed at a position including one point of the object. It may be generated.

  The control data may or may not include advertisement data of advertisements that meet the advertisement display conditions. When advertisement data is included in the control information, the advertisement set in the control information can be flexibly displayed at the position of the selected object at a timing according to the situation. When this control information does not include advertisement data, various advertisements corresponding to the situation of user information such as operation information of the viewing user and purchase history when the moving image is played on the viewing user terminal device 40 are displayed on the selected object. The position can be displayed flexibly.

  As described above, the moving image distribution server device 10 generates display information for displaying the additional content in an area according to the object information of the selected object, including at least a part of the selected object. It functions as an example of display information generating means.

  Next, the moving image distribution server device 10 stores the generated control information in association with the moving image ID (step S23). Specifically, the system control unit 14 displays control information including an advertisement ID to be displayed, each frame number of an image frame for displaying the advertisement, and coordinate information of an object for displaying the advertisement corresponding to the frame number. In association with the video ID, it is registered in the control information DB 12e.

  As shown in FIGS. 22A to 22D, the system control unit 14 is composed of an image frame 53 in which the whole or part of the object display area (mask area ms1, ms2) of the selected object is filled with a single color. May be generated as masked display information. In step S22, the system control unit 14 sets the mask area at a position where an advertisement is displayed so as to correspond to the selected object. The system control unit 14 generates display information composed of image frames in which the mask area is filled with the same color. In step S22, the system control unit 14 registers the masked display information in the moving image DB 12a or the like in association with the moving image ID.

  Further, as shown in FIGS. 22A to 22D, the system control unit 14 may change the size and shape of the mask region ms1 and the mask region ms3 in accordance with the change in the size and shape of the object.

  In the masked display information, the amount of moving image data is reduced by the amount masked. In particular, the compression rate when a moving image is compressed by a moving image encoding method during encoding is increased.

  When using the masked display information, the masked display information is transmitted to the viewing user terminal device 40 together with the control information generated in step S22. When the viewing user terminal device 40 reproduces the masked display information, the viewing user terminal device 40 superimposes and displays an advertisement on the mask area based on the control information.

  When advertisement data is included in this control information, the advertisement set in the control information can be flexibly displayed in the mask area at a timing according to the situation. When this control information does not include advertisement data, various advertisements according to the situation of user information such as operation information of the viewing user and purchase history can be flexibly applied to the mask area when the moving image is played on the viewing user terminal device 40. Can be displayed.

  The system control unit 14 determines the color to be set in the mask area based on the color of the object or object display area, the background color around the object or object display area, and the color of the advertisement to be added.

  Specifically, the system control unit 14 determines the pixel color information of the object or the object display area, the pixel color information of a predetermined range outside the boundary of the object or the object display area, and the advertisement color to be added. At least one piece of information is acquired. As a color to be set in the mask area, for example, the system control unit 14 selects a color similar to the color of the pixel in the object or the object display area (for example, a difference between RGB values is within a predetermined range), the boundary of the object or the object display area A color similar to the color of the pixels in a predetermined range outside or a color similar to the color of the advertisement to be added is set. Alternatively, as the color to be set in the mask area, for example, the system control unit 14 may select a color intermediate between the color of a pixel such as an object and the color of a pixel in a predetermined range outside the boundary of the object (for example, RGB of the object). RGB value between the RGB value of the background color and the RGB value of the background color), the intermediate color between the color of the pixel of the object etc. and the color of the advertising pixel, the color of the pixel in a predetermined range outside the boundary of the object etc. And an intermediate color between the pixel color of the advertisement and the like.

  When the colors set in the mask area are similar or intermediate colors, when the moving image is compressed by the moving image encoding method, the color change between the image frames is reduced, and the moving image compression rate is increased. .

  21A to 21D, the system control unit 14 may generate moving image data in which advertisements are embedded in advance as display information.

  For example, the system control unit 14 may delete an area in which an advertisement is embedded (for example, an object display area) and insert advertisement data for each image frame instead. In this case, control information is unnecessary, and moving picture information is transmitted to the viewing user terminal device 40.

  Further, when distributing the moving image to the viewing user terminal device 40, the system control unit 14 reads out the masked display information from the moving image DB 12a and the like, selects an advertisement according to the information such as the user information, and the mask. An advertisement may be embedded in the mask area of the displayed display information to generate display information that is new moving image information. In this case, control information is unnecessary, and moving picture information is transmitted to the viewing user terminal device 40.

  Note that when selecting an object or object display area to be masked, the system control unit 14 causes the object or object when the difference data amount of the object or object display area between image frames is larger than a predetermined threshold in step S21. A display area may be selected. The system control unit 14 sets a predetermined threshold according to the difference data amount of the object. For example, when each object is replaced with a masked image frame, the system control unit 14 compares the compression rate of the moving image according to the moving image encoding method, and the object having the predetermined order from the object compressed with the highest compression rate. Are selected as objects to be masked.

(3.5 Example of video distribution processing)
First, an operation example of the moving image distribution process in the moving image distribution server device 10 will be described with reference to FIG.

  FIG. 23 is a flowchart illustrating an example of a moving image distribution process of the system control unit of the moving image distribution server device according to the embodiment.

  As shown in FIG. 23, the moving image distribution server device 10 acquires the content condition of the moving image desired to be viewed from the viewing user terminal device 40 that desires to view the moving image (step S30). Specifically, the system control unit 14 acquires information on moving image content conditions such as a search keyword (movie title, actor name, director name, etc.) and a moving image category from the viewing user terminal device 40.

  Next, the moving image distribution server device 10 extracts a moving image that satisfies the acquired content condition from the moving image DB 12a (step S31). Specifically, the system control unit 14 refers to the moving image DB 12a, performs a search based on the content condition, extracts moving images satisfying the content condition, and generates display information of the search results listed.

  Next, the moving image distribution server device 10 presents the extracted moving image to the viewing user terminal device 40 (step S32). Specifically, the system control unit 14 transmits display information of the generated search result to the viewing user terminal device 40.

  Next, the moving image distribution server device 10 acquires the selection of the moving image from the viewing user terminal device 40 (step S33). The viewing user terminal device 40 transmits the moving image ID of the moving image selected by the viewing user from the search result list to the moving image distribution server device 10 together with the user ID and the moving image display request. The system control unit 14 receives the moving image ID of the selected moving image from the viewing user terminal device 40.

  Next, the moving image distribution server device 10 provides the viewing user terminal device 40 with the moving image and control information corresponding to the moving image (step S34). Specifically, the system control unit 14 searches the moving image DB 12a based on the received moving image ID, and acquires moving image data corresponding to the received moving image ID. Further, the system control unit 14 searches the control information DB 12e based on the received moving image ID, and acquires control information corresponding to the received moving image ID.

  The system control unit 14 transmits the moving image information and control information of the moving image ID to the viewing user terminal device 40.

  Note that the moving image distribution server device 10 may transmit the entire data of the moving image with the moving image ID and control information corresponding to the moving image data to the viewing user terminal device 40 or for each chapter of the moving image. Alternatively, part of the moving image data of the moving image ID and control information corresponding to the moving image data may be transmitted to the viewing user terminal device 40 in a streaming manner.

(3.6 Example of control information change processing)
Next, an operation example of control information change processing in the moving image distribution server device 10 will be described with reference to FIGS. 24 and 25.

  FIG. 24 is a flowchart illustrating an example of the control information change process of the system control unit of the video distribution server device according to the embodiment. FIG. 25 is a schematic diagram illustrating an example of skipping or rewinding in moving image data.

  As shown in FIG. 24, the moving image distribution server device 10 acquires operation information from the viewing user terminal device 40 (step S40). Specifically, the operation information for the moving image being reproduced on the display unit 43 of the viewing user terminal device 40 (for example, the frame ID of the image frame that has started skipping and the frame ID of the image frame that has been skipped, or rewinding) The system control unit 14 acquires information on the frame ID of the image frame that has started and the frame ID of the image frame that has been rewound from the viewing user terminal device 40.

  As shown in FIG. 25, when the skip is started from the frame number “000,015” and the skip is ended at the frame number “000,135”, the viewing user terminal device 40 determines the skip start frame number “000,015” and the skip end. Information of the frame number “000,135” is transmitted to the moving image distribution server device 10 as operation information together with the user ID.

  As shown in FIG. 25, when the rewinding is started from the frame number “000,360” and the rewinding is finished at the frame number “000,165”, the viewing user terminal device 40, the rewinding start frame number “000,360” and The information of the frame number “000,165” at the end of rewinding is transmitted as operation information to the moving image distribution server device 10 together with the user ID.

  Next, the moving image distribution server device 10 calculates object information based on the operation information, the information on the object display area in the image frame corresponding to the operation information, and the object ID (step S41). When skipped, the system control unit 14 determines, for each object, based on each frame / object information and the object ID in the image frame excluding the image frame of the skipped frame number as in step S20. Re-aggregate object information.

  In the case of rewinding, the system control unit 14 in the image frame that overlaps the image frame of the frame number of the rewinded portion as in step S20, based on each frame object information and the object ID, Re-aggregate object information for each object. That is, the system control unit 14 adds up the frame object information of the object existing in the image frame of the frame number at the rewinded portion.

  Thus, the newly aggregated object information may be recorded in the aggregate object information DB 12d.

  Next, the moving image distribution server device 10 selects an object for displaying an advertisement based on the newly calculated object information and the advertisement display condition set by the advertiser of the advertisement to be displayed (step S42). Specifically, as in step S21, the system control unit 14 refers to at least one of the frame object information DB 12c and the aggregate object information DB 12d, searches for object information that meets the advertisement display conditions, and displays an advertisement. Select an object.

  Next, the moving image distribution server device 10 generates control information for displaying an advertisement so as to correspond to the selected object (step S43). Specifically, the system control unit 14 refers to at least one of the frame object information DB 12c and the aggregate object information DB 12d as in step S22, and matches the advertisement display condition based on the object ID of the selected object. Each frame ID for displaying the advertisement and the coordinate information of the object in these image frames are acquired, and control information including each frame ID and coordinate information corresponding to these frame IDs is generated. Note that this control information may be modified examples of various control information including display information described above.

  Next, the moving image distribution server device 10 provides the control information to the viewing user terminal device 40 (step S44). Specifically, the system control unit 14 transmits the newly generated control information to the viewing user terminal device 40 as in step S34. In addition, in the case of newly generated control information compared to the previous control information, when the number of appearances of advertisements decreases and the advertisement display conditions are not satisfied, the system control unit 14 newly The generated control information may not be transmitted to the viewing user terminal device 40.

  As illustrated in FIG. 21A, when the delete button bt of the advertisement “AD-e1” is clicked, the viewing user terminal device 40 uses the advertisement ID and the delete request of the advertisement “AD-e1” as operation information. You may transmit to the moving image delivery server apparatus 10 with user ID. When an advertisement such as an advertisement “AD-e1” is clicked, the viewing user terminal device 40 uses the advertisement ID of the advertisement “AD-e1” and information indicating that the advertisement has been selected as operation information together with the user ID. Alternatively, it may be transmitted to the moving image distribution server device 10. Then, as in step S <b> 40, the system control unit 14 acquires operation information from the viewing user terminal device 40.

  When the system control unit 14 receives the operation information for deleting the advertisement, the system control unit 14 generates display information including the control information in which the advertisement corresponding to the advertisement ID is deleted, or displays the display frequency and the display area of the advertisement corresponding to the advertisement ID. Display information including the reduced control information is generated, and the display information is updated.

  When the system control unit 14 receives operation information indicating that an advertisement has been selected, it generates control information for opening a linked web page corresponding to the advertisement ID in another window, or deletes the advertisement corresponding to the advertisement ID. The display information including the control information generated is generated, or the display information including the control information with the display count and display area of the advertisement corresponding to the advertisement ID reduced is generated to update the display information.

  Note that the operation information from the viewing user terminal device 40 may be information on an operation for changing the display size of a moving image being reproduced on the display unit 43 of the viewing user terminal device 40. And the moving image delivery server apparatus 10 may update an advertisement according to the change operation of the display size of this moving image.

  When the operation for changing the display size of the moving image is an operation for reducing the display window, the display area of each object becomes smaller as the display size of the moving image is reduced. If the display area of each object is reduced, the advertisement display condition may not be satisfied, so the moving image distribution server device 10 performs a process of replacing with a new advertisement or a process of deleting the advertisement.

  On the other hand, when the operation for changing the display size of the moving image is an operation for enlarging the display window, the display area of each object increases as the display size of the moving image increases. If the display area of each object is increased, there is a possibility that there is an advertisement that satisfies the advertisement display condition higher than the currently displayed advertisement. Therefore, the video distribution server device 10 sets the advertisement display condition more than the current advertisement. If there is an advertisement with a high degree of satisfaction, processing is performed to replace it with that advertisement.

  In step S <b> 40, the moving image distribution server device 10 acquires operation information for reducing or expanding the display size of the moving image and the frame ID of the operated image frame from the viewing user terminal device 40.

  In step S41, the moving image distribution server device 10 calculates object information after the operated image frame. This object information includes information on the area of the object display area as the display size is reduced or enlarged.

  In step S42, the video distribution server device 10 selects an object for displaying an advertisement based on the newly calculated object information and the advertisement display condition set by the advertiser of the advertisement to be displayed. If there is no advertisement that satisfies the advertisement display condition, the advertisement may be deleted from the object. Further, with the change of the display size of the moving image, the currently displayed advertisement may be changed so as to be displayed on another object that satisfies the advertisement display condition.

  If there is another advertisement satisfying the advertisement display condition in step S43, the video distribution server device 10 generates control information for displaying this advertisement.

  When there is another advertisement satisfying the advertisement display condition in step S44, the video distribution server device 10 provides the viewing user terminal device 40 with control information for displaying this other advertisement.

  As described above, according to the present embodiment, the object is selected based on the object information of the object displayed in the image frame 50 and the advertisement display condition for displaying the advertisement to be added to the video, and the selected object The control information for displaying the advertisement in the area according to the object information of the selected object including at least a part is generated, and the advertisement is displayed in the area for the object information of the object already displayed in the image frame 50. Therefore, the advertisement can be displayed in an area in the moving image that does not give the viewer a sense of incongruity. Further, according to the position, shape, size, etc. of the object, the advertisement can be displayed flexibly so as to meet the advertisement condition.

  Further, the object information includes an integrated display amount (appearance number, display time, continuous appearance number, integrated display area, etc.) in which the object is displayed by integration of at least one of space and time in the moving image, depending on the integrated display amount When an object for displaying an advertisement is selected, an object corresponding to exposure in at least one of space and time is selected as an object more suitable for the advertisement display condition.

  Further, when the object information includes the continuous display number of objects displayed continuously in the plurality of image frames 50, and the object for displaying the advertisement is selected according to the continuous display number, the display of the advertisement is interrupted. Without being displayed on the video.

  In addition, when the object information includes an object difference data amount between adjacent frames of the image frame 50 and selects an object having a difference data amount larger than a threshold set by the difference data amount of each object, the difference data amount By replacing an object with a large advertisement with an advertisement with little change in color or the like for each image frame, it is possible to reduce the amount of data when the moving image is compressed by the moving image encoding method.

  In addition, the object information includes information on the shape of the object, and the advertisement display condition includes information on the shape of the advertisement. When the object is selected, since the advertisement can be displayed so as to match the shape of the object without changing the shape of the advertisement, the process of changing the shape of the advertisement can be omitted.

  In addition, if the display information includes a movie in which the area according to the object information of the selected object is masked with a single color, it is masked with a single color, so the amount of data when the movie is compressed is reduced. it can.

  In addition, when calculating object information with an object displayed continuously in a plurality of image frames 50 as one object, the object information is calculated in units that are continuous in the image frame, and the object is temporarily hidden by other objects. Since the advertisement can be displayed at a timing such as when the scene is switched from the shooting angle or when the scene is switched, the advertisement can be displayed so as not to give the viewer a sense of incongruity even when the advertisement is switched.

  Next, a modification of the image frame partially masked will be described with reference to FIGS.

  FIG. 26 and FIG. 27 are schematic diagrams showing a modification of the mask for masking a part of the image frame.

  As shown in FIG. 26, the system control unit 14 may generate a mask image ms composed of intermediate colors for each pixel in the area from the object obj and the advertisement ad. As shown in FIG. 17, for example, the system control unit 14 creates an image frame in which a mask image ms is embedded in a partial area of the object ob6 at a predetermined frame number for the object ob6. By displaying the image frame in which the mask image ms is embedded before the image frame in which the advertisement ad is embedded in a partial area of the object ob6, the image information changes with respect to the partial area of the object ob6. Reduce.

  Based on the color of the pixel in the area according to the object information of the selected object obj and the color of the pixel of the advertisement ad added to the area, the color for masking the area is determined for each pixel, and the display information is If the area contains a video that has been masked with the color determined for each pixel, for example, the area can be masked with an intermediate color for each corresponding pixel, so before the image frame where the advertisement is displayed and after the advertisement is displayed Since a part of the image frame between these images is masked with an intermediate color for each pixel, the amount of data when the moving image is compressed can be reduced.

  In addition, regarding the object appearing in the moving image provided by the content provider, the system control unit 14 may select the object in step S21 based on information on permission or prohibition of display of the additional content on the object. In this case, display information can be generated so as to satisfy the display conditions of the content provider.

  Furthermore, the present invention is not limited to the above embodiments. Each of the embodiments described above is an exemplification, and any configuration that has substantially the same configuration as the technical idea described in the claims of the present invention and has the same operational effects can be used. It is included in the technical scope of the present invention.

1: Information processing system 10: Moving image distribution server device 12: Storage unit 12a: Moving image DB (moving image storage means)
12b: Advertising DB
12c: Frame object information DB
12d: Total object information DB
12e: Control information DB
14: System control unit 20: Content providing device 30: Advertisement providing device 40: Viewing user terminal device 50: Image frame

In order to solve the above problem, the invention according to claim 1 is an image frame for acquiring at least some of the image frames included in the moving image acquired from the moving image storage means for storing the moving image. Acquisition means, object specifying means for specifying at least one object displayed in the image frame for each acquired image frame, and the image displayed in each of the image frames for each specified object Object information calculation means for calculating object information related to an object, the calculated object information, and an additional content display condition for displaying additional content to be added to the video, for displaying the additional content Select an object from the identified objects And object selection means, comprise at least a portion of the selected object, and a display information generating means for generating display information for displaying the additional content in the area in accordance with the object information of the selected object, wherein display information, and transmitting means for transmitting to the terminal device of the user to view the video, and a receiving means for receiving operation information operation on the video from the terminal device, said object information calculating means The object information is calculated based on the operation information, and the object selecting means selects an object for displaying the additional content based on the object information calculated based on the operation information, and generates the display information. Means displays display information according to the object information of the selected object. And degrees generated, the transmission means, the display information the is again generated, and transmits to the terminal device.

The invention according to claim 2, Oite the information processing apparatus according to claim 1, wherein the object information includes a cumulative display amount the object is to be displayed on at least one accumulation of the space and time in the video The object selection means selects an object for displaying the additional content according to the integrated display amount.

The invention according to claim 3, Oite the information processing apparatus according to claim 1 or claim 2, wherein the object information, the continuous display speed of the objects displayed in succession to the plurality of image frames And the object selecting means selects an object for displaying the additional content in accordance with the number of continuous displays.

Invention according to claim 4, Oite the information processing apparatus according to any one of claims 1 to 3, wherein the object information, the differential data of the object between adjacent frames of said image frame The object selecting means selects an object having a larger amount of difference data than a threshold set by the difference data amount of each object.

Invention of claim 5, Oite the information processing apparatus according to any one of claims 1 to 4, wherein the object information includes information of the shape of the object, the additional content display conditions Includes information on the shape of the additional content, and the object selection unit selects an object for displaying the additional content according to the information on the shape of the object so as to conform to the shape of the additional content. It is characterized by doing.

The invention according to claim 6, Oite the information processing apparatus according to any one of claims 1 to 5, wherein the display information is a single area in accordance with the object information of the selected object It includes a moving image masked with one color.

The invention according to claim 7, the color of the pixel of claim Oite from 1 to the information processing apparatus according to any one of claims 6, in the area in accordance with the object information of the selected object, The color for masking the area is determined for each pixel based on the color of the pixel of the additional content added to the area, and the display information includes a moving image in which the area is masked with the color determined for the pixel. It is characterized by.

The invention according to claim 8, Oite the information processing apparatus according to any one of claims 1 to 7, wherein the object information calculation means is continuously displayed in a plurality of said image frame The object information is calculated using the object as one object.

The invention according to claim 9 is an image frame acquisition step in which the image frame acquisition means acquires at least some of the image frames included in the moving image acquired from the moving image storage means that stores the moving image. An object specifying means for specifying at least one object displayed in the image frame for each of the acquired image frames; and an object information calculating means for each of the specified objects. An object information calculating step for calculating object information relating to the object displayed in the image frame; and an additional content display for displaying the calculated object information and additional content to be added to the moving image by an object selection means. Based on the conditions, before An object selection step of selecting an object for displaying additional content from the identified object, and a display information generating means includes at least a part of the selected object, and the object information of the selected object A display information generating step for generating display information for displaying the additional content in the conforming area; a transmitting step for transmitting the display information to a terminal device of a user who views the moving image; and a receiving unit. but seen including a receiving step, the receiving operation information is operated with respect to the video from the terminal device, in the object information calculating step to calculate the object information based on the operation information, the object selection step In this case, the operator is calculated based on the operation information. Based on the object information, an object for displaying the additional content is selected, and in the display information generation step, display information according to the object information of the selected object is generated again. In the transmission step, the object is displayed again. The generated display information is transmitted to the terminal device .

According to a tenth aspect of the present invention, there is provided the image frame acquiring means for acquiring at least some of the image frames included in the moving image acquired from the moving image storage means for storing the moving image, the acquisition Object specifying means for specifying at least one object displayed in the image frame for each image frame, and calculating object information relating to the object displayed in each image frame for each specified object Object information calculation means, based on the calculated object information and an additional content display condition for displaying additional content to be added to the moving image, an object for displaying the additional content is specified as the specified object Select an object to select from Means, before Symbol comprise at least a portion of the selected object, the display information generating means for generating display information for displaying the additional content in the area in accordance with the object information of the selected object, the display information , Functioning as transmission means for transmitting to the terminal device of a user who views the video, and reception means for receiving operation information operated on the video from the terminal device , wherein the object information calculation means The object information is calculated based on information, the object selecting means selects an object for displaying the additional content based on the object information calculated based on the operation information, and the display information generating means is Regenerate display information according to the object information of the selected object The transmission means, the display information the is again generated, and transmits to the terminal device.

Claims (10)

Image frame acquisition means for acquiring at least some of the plurality of image frames among the image frames included in the video acquired from the video storage means for storing the video;
Object specifying means for specifying at least one object displayed in the image frame for each acquired image frame;
Object information calculating means for calculating object information relating to the object displayed in each of the image frames for each of the identified objects;
Object selection for selecting an object for displaying the additional content from the specified object based on the calculated object information and an additional content display condition for displaying the additional content to be added to the video Means,
Display information generating means for generating display information for displaying the additional content in an area according to the object information of the selected object, including at least a part of the selected object;
An information processing apparatus comprising: The information processing apparatus according to claim 1,
The object information includes an accumulated display amount in which the object is displayed as an accumulation of at least one of space and time in the moving image,
The information processing apparatus, wherein the object selection unit selects an object for displaying the additional content according to the integrated display amount. In the information processing apparatus according to claim 1 or 2,
The object information includes a continuous display number of the object displayed continuously in a plurality of the image frames,
The information processing apparatus, wherein the object selecting means selects an object for displaying the additional content according to the number of continuous displays. In the information processing apparatus according to any one of claims 1 to 3,
The object information includes a difference data amount of the object between adjacent frames of the image frame;
The information processing apparatus, wherein the object selection unit selects an object having a larger difference data amount than a threshold set by the difference data amount of each object. In the information processing apparatus according to any one of claims 1 to 4,
The object information includes information on the shape of the object,
The additional content display condition includes information on the shape of the additional content,
The information processing apparatus, wherein the object selection unit selects an object for displaying the additional content according to information on the shape of the object so as to match the shape of the additional content. In the information processing apparatus according to any one of claims 1 to 5,
The information processing apparatus, wherein the display information includes a moving image in which a region according to the object information of the selected object is masked with a single color. In the information processing apparatus according to any one of claims 1 to 6,
Based on the color of the pixel in the area according to the object information of the selected object and the color of the pixel of the additional content added to the area, a color for masking the area is determined for each pixel,
The information processing apparatus, wherein the display information includes a moving image in which the area is masked with a color determined for each pixel. In the information processing apparatus according to any one of claims 1 to 7,
The information processing apparatus, wherein the object information calculation unit calculates the object information with the object displayed continuously in the plurality of image frames as one object. An image frame acquisition unit that acquires at least some of the plurality of image frames among the image frames included in the moving image acquired from the moving image storage unit that stores the moving image;
An object specifying unit for specifying at least one object displayed in the image frame for each of the acquired image frames;
An object information calculating unit that calculates object information about the object displayed in each of the image frames for each identified object;
Based on the calculated object information and an additional content display condition for displaying the additional content to be added to the moving image, an object selection unit selects an object for displaying the additional content as the specified object. An object selection step to select from,
A display information generating step for generating display information for displaying the additional content in an area according to the object information of the selected object, the display information generating means including at least a part of the selected object;
An information processing method comprising: Computer
Image frame acquisition means for acquiring at least some of the image frames included in the moving image acquired from the moving image storage means for storing the moving image;
Object specifying means for specifying at least one object displayed in the image frame for each acquired image frame;
Object information calculating means for calculating object information related to the object displayed in each of the image frames for each of the specified objects;
Object selection for selecting an object for displaying the additional content from the specified object based on the calculated object information and an additional content display condition for displaying the additional content to be added to the video Means and
And a display information generation unit configured to generate display information for displaying the additional content in an area according to the object information of the selected object, including at least a part of the selected object. A program for an information processing apparatus.

Images