JP2017182706A - Server device, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a server device that can generate and transmit a representative image adequately representing the contents of a transmitted moving image without requiring additional data such as meta-data.SOLUTION: A distribution server 1 for transmitting content made up of a plurality of frames to a plurality of clients 2 obtains camera work data indicating a display range at each client 2 from each client 2, and calculates the number of display times at the client 2 by the predetermined region in an image frame for each image frame. The distribution server extracts a region of the image frame where the calculated number of display times meets a predetermined reference, and generates a digest image representing contents of the content on the basis of the region.SELECTED DRAWING: Figure 1

Description

  The present invention belongs to the technical field of server devices, information processing methods, and programs. More specifically, the present invention belongs to the technical field of a server device and an information processing method for transmitting a moving image to a terminal device, and a program for the server device.

  In recent years, research / development on a so-called virtual client system in which the function of each client terminal connected to a server device via a network such as the Internet is provided from the server device has been performed. As a prior art document regarding such a virtual client system, for example, there is Patent Document 1 below. In the technique disclosed in Patent Literature 1, the content of the moving image is represented by cutting out a part of the moving image transmitted to each client terminal according to the preference of the user who uses the client terminal. A so-called digest image is generated.

Japanese Patent Laying-Open No. 2004-126811 (FIG. 23, etc.)

  However, in the virtual client system described in Patent Document 1, it is necessary to create metadata corresponding to the content of the original moving image in order to generate a digest image. Here, as metadata in this case, metadata including the appearance time information of the object in the original video and its attribute information is necessary, and it takes a lot of labor to create this. was there. In addition, the creation of such metadata requires an object identification process for detecting the position of the object in the original moving image, and the processing load for the object identification process increases. there were.

  Therefore, the present invention has been made in view of the above problems, and it is possible to generate and transmit a representative image that accurately represents the content of a moving image to be transmitted without using additional data such as metadata. A server device, an information processing method, and a program are provided.

  In order to solve the above problem, the invention according to claim 1 is a server device that transmits the moving image including a plurality of frames to each of a plurality of terminal devices that display a moving image. In the terminal device for each preset area in the frame, based on the range data acquired by the acquisition unit, and the range data acquired by the acquisition unit, the range data indicating the display range of the moving image Based on the calculation means for calculating the display count for each frame and the display count calculated by the calculation means, the region of the frame that satisfies a preset criterion for the display count is extracted from the moving image. Extraction means; and generation means for generating a representative image representative of the content of the moving image based on the region extracted by the extraction means; Characterized in that it comprises.

  According to a second aspect of the present invention, in the server device according to the first aspect, the acquisition unit belongs to a terminal device group obtained by classifying the plurality of terminal devices by a preset classification method. The range data is obtained from a device.

  According to a third aspect of the present invention, in the server device according to the first or second aspect, the calculation unit calculates the display count for each pixel as the region, and the extraction unit calculates the display count. The pixel that satisfies the criterion is extracted from the moving image, and the generation unit generates the representative image based on the pixel extracted by the extraction unit.

  According to a fourth aspect of the present invention, in the server device according to any one of the first to third aspects, the extraction unit is configured such that the display count calculated by the calculation unit is a playback time axis of the moving image. The region reproduced at a preset reproduction time before and after the timing including the peak timing is extracted from the moving image.

  According to a fifth aspect of the present invention, in the server device according to any one of the first to fourth aspects, the extraction unit is configured such that the display count calculated by the calculation unit is equal to or greater than a preset threshold value. The region is extracted from the moving image.

  According to a sixth aspect of the present invention, in the server device according to any one of the first to fifth aspects, the extraction means is centered on a center of the region where the display count is maximum in the frame. An image for a minimum resolution set in advance in the transmission of the moving image is extracted from the moving image.

  The invention according to claim 7 is the server device according to any one of claims 1 to 5, wherein the extraction unit includes the region where the number of display times is maximum in the frame, and An image having an aspect ratio set in advance in the transmission of the moving image is extracted from the moving image.

  According to an eighth aspect of the present invention, in the server device according to any one of the first to fifth aspects, in the transmission of the moving image, the size of the area in which the display count is greater than a preset threshold value. When the resolution is equal to or lower than a preset minimum resolution, the extraction unit extracts an image corresponding to the minimum resolution from the moving image centering on a center of the area where the display count is greater than the threshold, and the display count When the size of the area larger than the threshold is larger than the minimum resolution, the extraction means includes the area where the display count is larger than the threshold, and the aspect set in advance in the transmission of the moving image A ratio image is extracted from the moving image.

  The invention according to claim 9 is an information processing method executed in a server device that transmits the moving image including a plurality of frames to each of a plurality of terminal devices that display a moving image. An acquisition step of acquiring range data indicating a display range from each of the terminal devices, and a display count in the terminal device for each preset area in the frame based on the range data acquired in the acquisition step Calculating step for each frame, and extracting step for extracting the region of the frame that satisfies the preset reference number from the moving image based on the display number calculated in the calculation step And representing the content of the video based on the region extracted in the extraction step. Characterized in that it comprises a generation step of generating a representative image.

  According to the tenth aspect of the present invention, a display range of the moving image in each terminal device is set in a computer included in a server device that transmits the moving image including a plurality of frames to each of a plurality of terminal devices that display the moving image. The acquisition step of acquiring the range data shown from each of the terminal devices, and based on the range data acquired in the acquisition step, the number of display times in the terminal device for each preset region in the frame, A calculation step of calculating for each frame; an extraction step of extracting the region of the frame satisfying a predetermined criterion from the moving image based on the display count calculated in the calculation step; Based on the region extracted in the extraction step, a representative image representing the content of the moving image is obtained. A generation step of forming, characterized in that for the execution.

  According to the invention described in any one of claims 1, 9, and 10, a representative image that accurately represents the content of a moving image can be generated without additional data such as metadata. .

  According to the second aspect of the present invention, it is possible to generate a representative image optimized for a user of a terminal device belonging to the terminal device group.

  According to the third aspect of the present invention, it is possible to generate a representative image that more accurately represents the content of the moving image.

  According to the fourth aspect of the present invention, even if the number of times of display is small, it is possible to generate a representative image including a region where the number of times of display reaches the peak before and after that.

  According to the fifth aspect of the present invention, it is possible to generate a representative image including a region with a larger number of display times and accurately represent the content of the moving image.

  According to the sixth aspect of the present invention, it is possible to generate a representative image consisting of an image with the lowest resolution.

  According to the seventh aspect of the present invention, it is possible to generate a representative image that accurately represents the content of a moving image while maintaining a predetermined aspect ratio.

  According to the eighth aspect of the present invention, it is possible to generate a representative image that accurately represents the content of the moving image in accordance with the relationship between the size of the region where the number of display times is greater than the threshold and the predetermined minimum resolution.

It is a figure which shows the example of a schematic structure of the communication system of this embodiment. (A) thru | or (c) is a figure which illustrates calculation of the maximum display frequency of this embodiment, respectively. (A) is a figure which illustrates the peak value of the maximum display frequency of this embodiment. (B) is a figure which illustrates extraction of the maximum display frequency area of this embodiment. (C) is a figure which illustrates extraction of the maximum display frequency area | region of this embodiment. (D) is a figure which illustrates extraction of the maximum display frequency area of this embodiment. (E) is a figure which illustrates extraction of the maximum display frequency area of this embodiment. (A) is a flowchart which shows the user clustering process of this embodiment. (B) is a flowchart showing an image frame extraction process of the present embodiment. (C) is a flowchart showing digest image generation processing of the present embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(1) Configuration and Operation Overview of Communication System FIG. 1 is a diagram illustrating a schematic configuration example of a communication system according to the present embodiment. As shown in FIG. 1, the communication system S includes a distribution server 1 and a plurality of clients 2. The distribution server 1 of this embodiment is an example of a server device of the present invention. The client 2 is an example of a terminal device of the present invention. The distribution server 1 and the client 2 can communicate with each other via the network NW. The network NW includes, for example, the Internet, a mobile communication network, a gateway, and the like.

  For example, the distribution server 1 transmits content including moving image data to the client 2 in response to a request from the client 2. Movie data is data representing a movie composed of a plurality of image frames. The content may include audio data. The content is transmitted using, for example, HTTP (Hyper Text Transfer Protocol) live streaming via the network NW. For example, the distribution server 1 may execute live distribution in which content transmitted from the video camera is distributed in real time during shooting by a predetermined video camera. Alternatively, the distribution server 1 may store the content in advance and distribute the stored content on demand. Taking a moving image shot by a predetermined video camera as an example, a moving image in the entire shooting range of the video camera is called an entire moving image. A moving image in a part of the shooting range of the video camera is referred to as a partial moving image. The distribution server 1 includes content including moving image data representing a moving image (which may be an entire moving image or a partial moving image) in an area within a display range designated based on an operation described later by a user of the client 2. To the client 2.

  The client 2 receives content distributed from the distribution server 1. The client 2 reproduces the received content, thereby displaying the moving image represented by the moving image data included in the content on the screen of the display unit 24a described later. During the reproduction of the content, the client 2 accepts a pseudo camera work operation by the user using the operation unit 25a described later. The camera work operation refers to an operation of determining the position of the camera with respect to the subject, the angle (orientation) of the camera with respect to the subject, and the size of the subject by moving the camera. In the present embodiment, camera work (moving a virtual camera) for a display target in a plurality of image frames constituting moving image data representing an entire moving image is operated by operating the operation unit 25a so that the user moves an actual camera. Simulate. Such an operation is referred to as a “display range changing operation”. By this display range changing operation, the user can designate a desired display range in the entire moving image, and can change the display range by the designation. The display range is a range displayed on the screen of the display unit 24a in the entire image frame constituting the entire moving image. The user can specify the position coordinates of the display range with respect to the entire moving image so as to be different for each image frame by changing the display range. Examples of such change operations include pan and tilt operations. The pan and tilt operations are operations for changing the direction of the viewpoint with respect to the image frame with reference to the virtual camera. Also, the user can enlarge or reduce the size of the display range for the entire moving image by changing the display range. That is, the user can enlarge or reduce the size of the display range without changing the content of the displayed image. Thereby, the image within the display range can be enlarged or reduced. An example of such a change operation is a zoom operation. The zoom operation is an operation for changing the angle of view from the viewpoint with respect to the image frame. The zoom operation includes a zoom-in operation and a zoom-out operation. Zooming in means an operation for narrowing the angle of view. Zoom out refers to the operation of widening the angle of view. The client 2 receives content including moving image data representing a moving image in an area within the display range specified based on a user operation from the distribution server 1. Based on the received moving image data, the client 2 causes the display unit 24a to display a moving image in an area within the display range of the entire moving image. The client 2 may be, for example, a personal computer, a smartphone, a mobile phone, a television, a video game machine, or the like.

  On the other hand, the distribution server 1 accepts upload of camera work data from the client 2 that reproduces the content. The camera work data is information regarding pseudo camera work indicating the display range of the moving image on the client 2 as the content playback time elapses. The camera work data includes a set of the position coordinates and size of the display range with respect to the image frame and the reproduction position of the image frame at least for each reproduction position where the display range change operation has been performed. Note that the position coordinates and size of the display range may be represented by pan, tilt, and zoom. Panning refers to a left / right swing of a video camera (virtual camera). Tilt refers to swinging the video camera (virtual camera) up and down. Zoom refers to display magnification. The reproduction position refers to a temporal position from the start of reproduction of moving image data. The playback position is also referred to as a playback time in that it is the time elapsed since the start of playback of the video data. Camera work data is an example of range data of the present invention.

  Here, the communication system S of the present embodiment operates as a virtual client system. That is, the communication system S transmits, from the distribution server 1 to the client 2, content including moving image data representing a moving image in an area within the display range of the client 2 based on the camera work data transmitted from each client 2. At this time, the distribution server 1 reads content including moving image data to be transmitted to each client 2 from the storage unit 12 and transmits the content to each client 2. Accordingly, each client 2 can receive content including moving image data representing a moving image in a display range corresponding to the user's arbitrary camera work operation and display the content on the display unit 24a.

  In addition, the distribution server 1 generates a digest image of the present embodiment. The digest image of this embodiment is a digest image that represents the content to be transmitted to each client 2. The digest image of the present embodiment is composed of a part of moving image data included in the content or one or a plurality of still image data constituting the moving image data. The still image data making up the digest image is, for example, a so-called thumbnail image that represents the content. The moving image data or still image data as a digest image is hereinafter simply referred to as “moving image data as a digest image”. The moving image data or the like as a digest image is distributed to the client 2 in response to a request from the client 2, for example. The user of the client 2 that has received the moving image data or the like as the digest image can recognize the content or the outline of the content represented by the digest image by reproducing the moving image data or the like. The digest image of this embodiment is generated based on camera work data transmitted from each client 2. The generated moving image data or the like as a digest image is stored in, for example, the distribution server 1 and transmitted in response to a request from the client 2. For example, a method similar to the method for transmitting content represented by the digest image to the client 2 can be used as a method for transmitting the generated moving image data as the digest image to the client 2. The generation of the digest image of this embodiment will be described in detail later.

(2) Configuration of Each Device Next, the configuration of each device included in the communication system S of the present embodiment will be described with reference to FIG. As shown in FIG. 1, the distribution server 1 includes a control unit 11, a storage unit 12, an interface unit 13, and the like. These components are connected to the bus 14. The interface unit 13 is connected to the network NW. The interface unit 13 is an example of an acquisition unit of the present invention. The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like as a computer. Each of the control units 11 is an example of a calculation unit, an example of an extraction unit, and an example of a generation unit of the present invention. The storage unit 12 is configured by, for example, a hard disk drive. The storage unit 12 stores an OS (Operating System), a server program, and the like. The server program is a program that causes the CPU to execute a content or digest image transmission process or the like. The storage unit 12 stores moving image data and audio data constituting the content. The moving image data and audio data are stored in the storage unit 12 in association with the content ID and the reproduction position or reproduction time in the content of the moving image data or audio data. The content ID is identification information for identifying the content.

  Next, as shown in FIG. 1, the client 2 includes a control unit 21, a storage unit 22, a video RAM 23, a video control unit 24, an operation processing unit 25, an audio control unit 26, an interface unit 27, and the like. These components are connected to the bus 28. A display unit 24 a including a display is connected to the video control unit 24. The control unit 21 includes a CPU, ROM, RAM, and the like as a computer. An operation unit 25 a is connected to the operation processing unit 25. The operation unit 25a includes, for example, a mouse, a keyboard, a remote controller, and the like. A touch panel serving both as the display unit 24a and the operation unit 25a may be applied. The control unit 21 receives an operation instruction from the operation unit 25 a by the user via the operation processing unit 25. A speaker 26 a is connected to the audio control unit 26. The interface unit 27 is connected to the network NW. The storage unit 22 is configured by, for example, a hard disk drive or a flash memory. The storage unit 22 stores an OS, player software, and the like. The player software is a program that causes the CPU to execute processing for receiving and reproducing content or a digest image.

  The storage unit 12 of the distribution server 1 also stores a server program that causes the CPU of the control unit 11 to execute the camera work data reception process and the like. Furthermore, the camera unit data received from any one of the clients 2 is stored in the storage unit 12. Each camera work data is stored in the storage unit 12 in association with the content ID and the user ID. The content ID indicates content that has been played back by the client 2 within the display range indicated by the target camera work data. The user ID is identification information for identifying a user who has performed a pseudo camera work operation indicated by target camera work data. The user ID is also information for identifying the client 2 used by the user. Note that the IP address of the client 2 or the like may be used as the terminal device identification information.

(3) Generation and Transmission of Digest Image According to this Embodiment Next , generation of a digest image and transmission to the client 2 according to this embodiment will be described with reference to FIGS. FIG. 2 is a diagram illustrating calculation of the maximum number of display times according to the present embodiment. FIG. 3 is a diagram illustrating the peak value of the maximum number of display times of the present embodiment and the extraction of the maximum number of display times region of the present embodiment. In the communication system S of the present embodiment, a digest image of the present embodiment that represents the content distributed to the client 2 is generated in the distribution server 1 and transmitted to the client 2 in response to a request from the client 2. . The distribution server 1 generates a digest image for the content based on the camera work data transmitted from each client 2 for the distributed content. Specifically, the distribution server 1 receives camera work data indicating a display range designated by each client 2. Then, the distribution server 1 uses an image frame including an image region in which the maximum display count described later in each client 2 becomes a peak value or a maximum value on the reproduction time axis of the moving image data included in the distributed content. Then, a digest image for the content is generated. The image area in this case is an image area composed of one or a plurality of pixels in one image frame. In other words, it is assumed that the image frame constituting the moving image data included in the content distributed to one client 2 is the image frame G illustrated in FIG. Then, it is assumed that the display range designated by the client 2 in the image frame G is the display range CW1 corresponding to FIG. The display range CW1 is an area in the image frame G, and is composed of one or a plurality of pixels. In this case, camera work data indicating the display range CW1 is transmitted from the client 2. In the case of FIG. 2A, the number of times that the image in the area inside the display range CW1 is displayed on the client 2 is one. This is because the image of the area inside the display range CW1 is displayed on the one client 2 that has transmitted the camera work data indicating the display range CW1. On the other hand, the image of the area outside the display range CW1 in the image frame G is not displayed by any client 2. Therefore, the number of times the client 2 displays the image in the region outside the display range CW1 in the image frame G is zero. In FIGS. 2A to 2C, the number of display times in the client 2 for each region of the image frame G (that is, a region composed of one or a plurality of pixels) is shown in parentheses. In other words, the display count is, in other words, the number of clients 2 on which an image in the area of the image frame G is displayed. Next, assuming that content including moving image data composed of image frames G is transmitted to, for example, three clients 2, each client 2 displays, for example, display range CW1 to display range illustrated in FIG. Camera work data indicating each CW3 is transmitted. Here, it is assumed that the position in the image frame G of the display range CW1 to the display range CW3 designated in each client 2 is the position illustrated in FIG. In this case, in the display range CW1 to the display range CW3, an overlapping range illustrated in FIG. 2B, the number of times the client 2 displays the image for each region of the image frame G is the number of times indicated by parentheses in FIG. 2B. That is, for example, as indicated by hatching of 45 degrees in FIG. 2B, the number of times the client 2 displays the image in the area inside the display range CW1 and outside the display range CW2 and the display range CW2 is one. On the other hand, the number of times the client 2 displays the images inside the display range CW1 and the display range CW2 and outside the display range CW3 is two. Further, as shown by cross-hatching in FIG. 2B, the number of times the client 2 displays the images in the inner regions of the display range CW1 to the display range CW3 is three. Then, there is no region in the image frame G illustrated in FIG. 2B in which the number of times the image is displayed on the client 2 is greater than the inner region of all the display ranges CW1 to CW3. Therefore, the maximum value of the number of display times in the client 2 of the image in the region in the image frame G illustrated in FIG. 2B is “3”. In the following description, the maximum value of the number of display times in the client 2 in the region in the image frame G is simply referred to as “maximum display number”. That is, the maximum display count corresponding to the image frame G illustrated in FIG. 2B is “3”. Next, assuming that content including moving image data composed of image frames G is transmitted to, for example, seven clients 2, each client 2 displays, for example, display range CW1 to display range illustrated in FIG. Camera work data indicating each CW7 is transmitted. Here, it is assumed that the position in the image frame G of the display range CW1 to the display range CW7 designated in each client 2 is the position illustrated in FIG. In this case, in the display range CW1 to the display range CW7, an overlapping range illustrated in FIG. The number of display times of the image for each region of the image frame G in the case illustrated in FIG. 2C is the number of display times indicated by parentheses in FIG. That is, for example, as shown by hatching of 45 degrees in FIG. 2C, the display in the client 2 of the image in the area inside the display range CW4 and outside the display range CW1 to the display range CW3 and the display range CW5 to the display range CW7. The number of times is one. On the other hand, for example, as indicated by hatching of 135 degrees in FIG. 2C, the number of times the client 2 displays the image in the area inside the display range CW4 and the display range CW2 and outside the display range CW5 is two. . Further, as shown by cross-hatching in FIG. 2C, the number of times of display in the client 2 of the images in the inner areas of all the display ranges CW1 to CW5 is five. In addition, there is no region in the image frame G in which the number of times of display in the client 2 is greater than the inner region of all the display ranges CW1 to CW5. Therefore, the maximum number of display times corresponding to the image frame G illustrated in FIG. 2C is “5”. The distribution server 1 of the present embodiment calculates the maximum number of display times for each image frame G as illustrated in FIG. 2 based on the camera work data from each client 2 belonging to the communication system S. The distribution server 1 temporarily stores the calculated maximum number of display times for each image frame G, for example, in the storage unit 12. Then, the distribution server 1 detects, for each image frame G, the maximum number of display times corresponding to the reproduction time of all or part of the content represented by the digest image. Here, the image frame G is displayed on the client 2 at a reproduction timing according to the reproduction of the content. Therefore, the image frame G is displayed on the client 2 at the playback time of the content to which it corresponds. That is, the maximum number of times of display corresponding to the playback time of the content changes as exemplified in FIG. 3A, for example, where the playback time is on the horizontal axis and the maximum number of display times is on the vertical axis. Note that the horizontal axis in FIG. 3A may be the number of image frames from a preset reproduction timing instead of the reproduction time. In the case illustrated in FIG. 3A, the timing at which the maximum number of display times takes a peak value with respect to the preceding and following playback times is 40 seconds, 190 seconds, and 280 seconds in playback time. It is assumed that the peaks become peaks P1 to P3, respectively. The fact that the maximum number of display times is peak P1 to peak P3 means that the image frames corresponding to the peaks P1 to P3 are displayed on more clients 2 than the image frames before and after that. In this case, it can be said that the content of the image frame corresponding to each of the peak P1 to the peak P3 represents the content of the content whose change in the maximum display number is shown in FIG. Therefore, in the case illustrated in FIG. 3A, the distribution server 1 displays the contents of the content whose change in the maximum number of display is shown in FIG. 3A based on the image frames corresponding to the peaks P1 to P3. A representative digest image is generated. In the case of content in which the maximum number of times of display is simply increased or decreased along the playback time, the content is not based on the peak P1 or the like as described above but based on the image frame corresponding to the maximum value of the maximum number of times of display. A digest image representing the contents of the above may be generated. In this case, the image frame corresponding to the maximum value of the maximum display count corresponds to the image frame corresponding to the peak P1 and the like. Further, the distribution server 1 may generate a digest image based on a region where the number of times of display in the client 2 of the region in the image frame is equal to or greater than a preset threshold value, instead of the maximum number of times of display for each image frame. Thereafter, the distribution server 1 transmits moving image data or the like as the generated digest image to the client 2 in response to a request from the client 2.

  Next, a method of generating a digest image that represents the content content whose change in the maximum number of display times is shown in FIG. 3A will be specifically described with reference to FIGS. 3B and 3E. In the following description, generation of a digest image based on an image frame corresponding to the peak P2 illustrated in FIG. When generating the digest image of the present embodiment based on the image frame for the peak P2, the distribution server 1 first extracts the image frame used for generating the digest image from the moving image data representing the entire moving image based on the image frame for the peak P2. To do. Here, as the image frame extraction method by the distribution server 1, two methods are conceivable. As illustrated in FIG. 3B, the first extraction method is a method for extracting an image frame reproduced at a preset reproduction time DA1 before and after the timing including the timing of the peak P2 in the reproduction time. is there. That is, as a first method of extracting image frames, the distribution server 1 is played back at preset playback times before and after the timing at which the maximum display count reaches the peak P2 with respect to the playback time axis of the video data. Extracted image frames from the video data. In this case, the distribution server 1 extracts image frames for a total of 10 seconds that have been reproduced, for example, for 5 seconds before and after the timing of the peak P2 in the reproduction time. In the second extraction method, as illustrated in FIG. 3C, the maximum display number equal to or greater than the maximum display number obtained by multiplying the maximum display number as the peak P2 in the reproduction time by a coefficient less than 1 set in advance. This is a method of extracting an image frame reproduced at a reproduction time DA2 corresponding to the number of times. That is, as a second method of extracting image frames, the distribution server 1 extracts image frames whose maximum display count is equal to or greater than a preset value (ratio) from the moving image data. In this case, the distribution server 1 reproduces the image reproduced at the reproduction time DA2 corresponding to the maximum number of times of display equal to or greater than the maximum number of times of display L obtained by multiplying the maximum number of times of display at the peak P2 by, for example, a coefficient 0.9. Extract the frame.

  Next, the distribution server 1 cuts out a digest image generation area in the image frame extracted from the image frame extracted by using the method illustrated in FIG. 3B or 3C. For example, the distribution server 1 cuts out a digest image generation area from the image frame based on the minimum resolution and aspect ratio of the display in each client 2 set in advance in the distribution of content by the communication system S, for example. The area of the image frame for the digest image is an area composed of one or a plurality of pixels. Here, there are two possible methods for extracting a region of an image frame for generating a digest image by the distribution server 1. In the first cutting method, as illustrated in FIG. 3D, in the image frame G used for generating the digest image, the size of the area M of the maximum display count in the image frame G is equal to or less than the above minimum resolution. This is a cutting method. In this case, the distribution server 1 cuts out the area AR having a size corresponding to the minimum resolution centered on the center of the area M with the maximum number of display times illustrated in FIG. 3D from the image frame G for generating a digest image. . In this case, the area M of the maximum number of display times illustrated in FIG. 3D is smaller than the area AR having a size corresponding to the minimum resolution. As illustrated in FIG. 3E, the second clipping method is a case where, in the image frame G used for generating the digest image, the size of the region M of the maximum display count in the image frame G is larger than the minimum resolution. This is a cutting method. In this case, the distribution server 1 cuts out the area AR including the area M having the maximum number of display times illustrated in FIG. 3E and having the aspect ratio from the image frame G for generating a digest image.

  The distribution server 1 is exemplified in either FIG. 3 (d) or FIG. 3 (e) from the image frame G extracted by the extraction method exemplified in either FIG. 3 (b) or FIG. 3 (c). 3A is used to generate a digest image that represents the contents of the content whose change in the maximum number of times of display is shown in FIG. The distribution server 1 repeats the generation of digest images for the peaks P1 to P3 illustrated in FIG. The distribution server 1 temporarily stores the generated digest image in the storage unit 12 and transmits it to the client 2 in response to a request from the client 2.

  The distribution server 1 classifies the plurality of clients 2 constituting the communication system S in advance based on, for example, preset user attributes, and associates the classified clients with each of the classified client groups. An image may be generated. In this case, the classification of the client 2 into a plurality of client groups is, in other words, the classification of the user of the client 2. Such classification is hereinafter referred to as “clustering”. Each classified client group is hereinafter referred to as a “cluster”. In this case, the distribution server 1 stores information indicating each classified cluster in the storage unit 12 together with a user ID for identifying the client 2 belonging thereto. The distribution server 1 receives the camera work data from the clients 2 belonging to each cluster for each cluster, calculates the maximum number of display times for each cluster, and determines the maximum for the reproduction time illustrated in FIG. 3A for each cluster. Detect changes in the number of impressions. The distribution server 1 generates a digest image of the present embodiment for each cluster using an image frame corresponding to the peak based on the detected change in the maximum number of display times. The distribution server 1 transmits the generated digest image for each cluster to which the client 2 that transmitted the camera work data used for the generation belongs.

  Further, when there are a plurality of maximum display count areas in one image frame corresponding to each of the peaks P1 to P3, a digest image may be generated using the plurality of areas. Furthermore, as a method of using the generated digest image, for example, it may be used on a list screen that displays a list of distributable content, an explanation screen for each content, etc. It may be used for scene selection or the like. Moreover, the digest image of this embodiment can be used not only for the on-demand delivery but also for live delivery. In the case of use in live distribution, a digest image generation process may be performed in real time, and displayed, for example, during a break time in a live performance. Furthermore, for example, it may be used to facilitate so-called catch-up to a live by presenting a digest image to a live participant (a viewer from the middle).

(4) Operation of Communication System S Next, the operation of the communication system S will be described with reference to FIG. FIG. 4A is a flowchart showing user clustering processing of the present embodiment. FIG. 4B is a flowchart showing image frame extraction processing of the present embodiment. FIG. 4C is a flowchart showing digest image generation processing of the present embodiment.

(I) User Clustering Process in Distribution Server 1 The user clustering process of this embodiment will be described with reference to FIG. For example, the control unit 11 of the distribution server 1 in the communication system S of the present embodiment starts the user clustering process shown in FIG. 4A every preset period. First, in step S <b> 1, the control unit 11 calculates a user feature amount indicating a user feature of each client 2. Here, for example, when a certain user requests participation in the communication system S in order to receive distribution of content by the communication system S, the user information set in advance indicating the user identifies the user. The information is stored in the storage unit 12 in association with the ID. The user information in this case includes, for example, age information indicating the user's age, gender information indicating the user's gender, or hobby information indicating the user's hobbies and preferences. Further, the control unit 11 stores the viewing history information of past contents in each client 2 included in the communication system S in the storage unit 12 in association with the user ID and the content ID. In step S1, the control unit 11 calculates a user feature amount indicating the feature of each user for each user by a preset method based on the user information and the viewing history information stored for each user. Then, the control unit 11 clusters the users of the clients 2 included in the communication system S at that time using a conventional clustering method such as a hierarchical clustering method based on the user feature amount calculated in step S1. (Step S2). By this step S2, a user cluster corresponding to the user feature amount is generated. And the control part 11 memorize | stores the information which shows the cluster produced | generated by step S2 in the memory | storage part 12 with the user ID which identifies the client 2 which belongs to it (step S3). Thereafter, the control unit 11 ends the user clustering process.

(II) Image Frame Extraction Processing in Distribution Server 1 The image frame extraction processing of this embodiment will be described with reference to FIG. The image frame extraction process described below is an image frame extraction process when the digest image of the present embodiment is generated as a moving image including a plurality of image frames. The image frame extraction process of the present embodiment is a process of extracting a content scene for generating a digest image for generating a digest image of the actual form. When the control unit 11 of the distribution server 1 generates, as a moving image, the digest image of the present embodiment for a certain content that has already been distributed based on an instruction from the administrator, for example, the present embodiment shown in FIG. Image frame extraction processing is performed. First, in step S10, the control unit 11 extracts, from the storage unit 12, the camera work data transmitted from the client 2 belonging to the cluster generated by the user clustering process shown in FIG. (Refer to FIG. 2B or FIG. 2C). That is, the control unit 11 acquires camera work data indicating the display range of the content in each client 2 for each cluster from each client 2 and stores it in the storage unit 12, and extracts and aggregates it. Next, the control unit 11 calculates the maximum number of display times in each image frame by the method described with reference to FIG. 2 or 3 based on the camera work data tabulated in step S10 (step S11). That is, the control unit 11 calculates the maximum number of display times for each predetermined area in the image frame for each image frame based on the acquired camera work data. Next, the control unit 11 performs a so-called smoothing process using, for example, a preset moving average process for the maximum number of display times calculated for each image frame (step S12). Next, the control unit 11 calculates the peak value of the maximum number of display times in the content using the result of step S12 (step S13). The peak value calculated in step S13 is, for example, the maximum number of display times for each of the peaks P1 to P3 illustrated in FIG. Next, the control unit 11 rearranges the peak values calculated in step S13 in descending order of the peak values from the order of the reproduction times (see FIG. 3A). Further, the control unit 11 extracts the peak values by the number of image frames used for generating the digest image in order from the largest peak value among the rearranged peak values, and corresponds to the extracted peak values. A plurality of image frames to be extracted are extracted from the storage unit 12 (step S14). That is, the control unit 11 extracts a plurality of image frames corresponding to the maximum display count based on the calculated maximum display count. More specifically, as a first example of step S14, the control unit 11 determines the timing in the reproduction time of each peak having an extracted peak value (an example is the peak P2 illustrated in FIG. 3B). A plurality of image frames reproduced in a preset reproduction time before and after the timing (see the code “DA1” in FIG. 3B) are extracted. The first example of step S14 corresponds to the first extraction method described with reference to FIG. Alternatively, as a second example of step S14, the control unit 11 sets 1 in advance for each peak having an extracted peak value (an example is the peak P2 illustrated in FIG. 3C). A plurality of images reproduced at a reproduction time (see symbol “DA2” in FIG. 3C) corresponding to a peak value equal to or higher than a peak value (see symbol “L” in FIG. 3C) obtained by multiplying a coefficient less than Extract the frame. The second example of step S14 corresponds to the second extraction method described with reference to FIG. Thereafter, the control unit 11 ends the image frame extraction process of the present embodiment.

(III) Digest Image Generation Processing in Distribution Server 1 The digest image generation processing of this embodiment will be described with reference to FIG. The digest image generation process described below is a digest image generation process when generating the digest image of the present embodiment as a moving image including a plurality of image frames. For example, the control unit 11 of the distribution server 1 starts the digest image generation process illustrated in FIG. 4C from the timing when the image frame extraction process illustrated in FIG. The digest image generation process shown in FIG. 4C is executed for each peak value extracted in step S14 of FIG. 4B, for example. First, in step S20, the control unit 11 sets the digest image generation process of this embodiment as a target of the image frame extracted from each image frame corresponding to each peak by the image frame extraction process shown in FIG. 4B. Select an image frame. Next, in step S21, the control unit 11 acquires information indicating the size of the region of the maximum number of times of display (see FIG. 3D or FIG. 3E symbol M) in the image frame selected in step S20. Next, the control part 11 determines whether the magnitude | size of the area | region acquired by step S21 is larger than the said minimum resolution preset in the communication system S (step S22). In the determination in step S22, when the size of the maximum display count area in the image frame is equal to or less than the minimum resolution (step S22: NO), the control unit 11 is centered on the center of the maximum display count area M. A region having the size corresponding to the minimum resolution and having the above aspect ratio is cut out from the image frame selected in step S20 for generating a digest image (see step S24, FIG. 3D). That is, the control unit 11 extracts a region of the image frame that becomes the maximum number of display times based on the calculated maximum number of display times. On the other hand, if it is determined in step S22 that the area of the maximum display count in the image frame is larger than the minimum resolution (step S22: YES), the control unit 11 includes the area of the maximum display count and includes the aspect ratio. Is cut out from the image frame selected in step S20 for digest image generation (see step S23, FIG. 3E). Thereafter, the control unit 11 generates a digest image representing the content of the content that has been distributed using the region cut out from the image frame selected in step S20 in step S23 or step S24, and stores the digest image in the storage unit 12. Store (step S25). That is, the control unit 11 generates a digest image that represents the content based on the clipped region. More specifically, the control unit 11 generates a digest image representing the contents as the image frame using the region cut out in step S23 or step S24 from the image frame selected in step S20. Thereafter, the control unit 11 determines whether or not the digest image generation processing of the present embodiment has been completed for all the image frames extracted for each peak by the image frame extraction processing shown in FIG. 4B (step S26). . If it is determined in step S26 that the digest image generation process has not been completed for all the image frames extracted for each peak value by the image frame extraction process shown in FIG. 4B (step S26: NO), the control unit 11 Returns to step S20 and selects the next image frame to be subjected to digest image generation processing. On the other hand, if it is determined in step S26 that the digest image generation process has been completed for all the image frames extracted for each peak value by the image frame extraction process shown in FIG. 4B (step S26: YES), the control unit 11 ends the digest image generation process as it is. Note that the moving image data or the like as the digest image generated by the digest image generation process of the present embodiment shown in FIG. 4C is obtained from, for example, the client 2 belonging to the cluster for each cluster in step S10 of FIG. Is transmitted to the client 2 in response to the request.

  As described above, according to the present embodiment, the display count in the client 2 for each region in the image frame is calculated for each image frame based on the camera work data acquired from each client 2. Then, an area of the image frame that satisfies a preset reference (for example, the maximum display count or a display count equal to or greater than a predetermined threshold) is extracted, and a digest image is generated based on the extracted area. . Accordingly, it is possible to generate a digest image that accurately represents the contents without adding additional data such as metadata.

  In addition, when generating a digest image as a still image, the control unit 11 performs only each image frame having each peak value extracted in step S14 in FIG. 4B (that is, each maximum display count is a peak value). Extract image frames only). And the control part 11 produces | generates the digest image representing the content of the content containing the moving image data comprised by each image frame using the area | region cut out by the method shown in FIG.4 (c) from each extracted image frame. . Further, the control unit 11 generates a digest image using, for example, image frames that are reproduced at predetermined reproduction times before and after each image frame having each peak value extracted in step S14 of FIG. 4B. May be.

DESCRIPTION OF SYMBOLS 1 Distribution server 2 Client 11, 21 Control part 12, 22 Storage part 13, 27 Interface part 24a Display part 25a Operation part S Communication system NW network G Image frame P1, P2, P3 Peak DA1, DA2 Playback time M, AR area

Claims (10)

In each of a plurality of terminal devices that display a moving image, a server device that transmits the moving image including a plurality of frames,
Acquisition means for acquiring range data indicating the display range of the moving image in each terminal device from each terminal device;
Based on the range data acquired by the acquisition means, a calculation means for calculating the number of display times in the terminal device for each preset area in the frame, for each frame;
Extraction means for extracting from the video the region of the frame that satisfies the preset reference number based on the display count calculated by the calculation means;
Generating means for generating a representative image representative of the content of the moving image based on the region extracted by the extracting means;
A server device comprising: The server device according to claim 1,
The server device, wherein the acquisition unit acquires the range data from the terminal devices belonging to a terminal device group obtained by classifying a plurality of the terminal devices by a preset classification method. In the server apparatus according to claim 1 or 2,
The calculation means calculates the display count for each pixel as the region,
The extraction unit extracts the pixels satisfying the criterion for the number of display times from the moving image,
The server device, wherein the generation unit generates the representative image based on the pixels extracted by the extraction unit. In the server apparatus as described in any one of Claims 1-3,
The extraction means includes the area reproduced at a preset reproduction time before and after the timing including a timing at which the display count calculated by the calculation means reaches a peak with respect to the reproduction time axis of the moving image. Is extracted from the moving image. In the server device according to any one of claims 1 to 4,
The server device, wherein the extraction unit extracts the region in which the number of display times calculated by the calculation unit is greater than or equal to a preset threshold value from the moving image. In the server apparatus according to any one of claims 1 to 5,
The extraction means extracts, from the moving image, an image corresponding to a minimum resolution set in advance in the transmission of the moving image, centering on the center of the region where the number of display times is maximum in the frame. apparatus. In the server apparatus according to any one of claims 1 to 5,
The extraction unit extracts from the moving image an image that includes the region where the number of times of display is maximized in the frame and has a preset aspect ratio in the transmission of the moving image. . In the server apparatus according to any one of claims 1 to 5,
When the size of the area in which the number of times of display is greater than a preset threshold is less than or equal to a preset minimum resolution in the transmission of the moving image, the extraction unit is configured to extract the area in which the number of times of display is greater than the threshold. Centering on the center of the image, the image for the minimum resolution is extracted from the video,
When the size of the area where the number of times of display is greater than the threshold is greater than the minimum resolution, the extraction means includes the area where the number of times of display is greater than the threshold and is preset in transmission of the moving image. A server device that extracts an image having a certain aspect ratio from the moving image. In the information processing method executed in the server device that transmits the moving image including a plurality of frames to each of the plurality of terminal devices that display the moving image,
An acquisition step of acquiring range data indicating a display range of the moving image in each terminal device from each terminal device;
Based on the range data acquired in the acquisition step, a calculation step of calculating, for each frame, the number of display times in the terminal device for each preset region in the frame;
Based on the display count calculated in the calculation step, an extraction step for extracting the region of the frame satisfying a preset criterion for the display count from the moving image;
A generating step for generating a representative image representative of the content of the moving image based on the region extracted in the extracting step;
An information processing method comprising: To each of a plurality of terminal devices that display a moving image, a computer included in a server device that transmits the moving image including a plurality of frames,
An acquisition step of acquiring range data indicating a display range of the moving image in each terminal device from each terminal device;
Based on the range data acquired in the acquisition step, a calculation step of calculating, for each frame, the number of display times in the terminal device for each preset region in the frame;
Based on the display count calculated in the calculation step, an extraction step for extracting the region of the frame satisfying a preset criterion for the display count from the moving image;
A generating step for generating a representative image representative of the content of the moving image based on the region extracted in the extracting step;
A program characterized by having executed.