WO2014132988A1

WO2014132988A1 - Information processing device and information processing method

Info

Publication number: WO2014132988A1
Application number: PCT/JP2014/054651
Authority: WO
Inventors: 翼梅津; 建太郎牛山
Original assignee: ブラザー工業株式会社
Priority date: 2013-02-27
Filing date: 2014-02-26
Publication date: 2014-09-04
Also published as: JP2014164685A; JP5870944B2

Abstract

Provided are an information processing device and information processing method by which it is possible to efficiently retrieve the camera work desired by a user. When a request for simulated camera work data is received from a terminal device, the information processing device determines, on the basis of scene information which has been associated with sub-regions in the image frames which compose a moving image and stored in advance, the sub-region associated with the scene information corresponding to the retrieval key included in the request, and determines simulated camera work data indicating the display region included in the determined sub-region as the simulated camera work data to be provided to the user of the terminal device.

Description

Information processing apparatus and information processing method

This disclosure relates to the technical field of terminal devices that display moving images.

2. Description of the Related Art Conventionally, there is a known video posting site for streaming video data uploaded to a server via the Internet in response to a distribution request from a terminal device. The moving image data stream-distributed from the moving image posting site is displayed on the window screen of the browser of the terminal device, for example. At this time, a representative still image in the moving image data related to the streaming-distributed moving image data may be displayed on the window screen. With this still image, it is possible to recommend moving image data related to the moving image data distributed in a streaming manner to the user of the terminal device.

On the other hand, Patent Document 1 discloses a system in which a user distributes editing data indicating a reproduction position of moving image data to other users by mail in order to recommend a specific scene of moving image data to other users. Yes. Thereby, the user who created edit data can quickly show a specific scene in the moving image data to other users.

JP 2009-124516 A

By the way, for example, a pseudo camera work may be performed by a user on a moving image such as a panoramic video. In the pseudo camera work, for example, the display range in a plurality of image frames constituting the moving image is designated by instructing the direction of the virtual camera, the field of view, and the like from the operation unit. In recent years, there is a need to upload camera work data indicating pseudo camera work to a server and provide it to other users. In this case, for example, it is assumed that pseudo camera work performed by another user on a moving image being displayed on the terminal device is recommended to the user of the terminal device. However, the pseudo camera work is characterized by the movement of the time-series display range in a plurality of image frames constituting a moving image. Therefore, it is difficult to convey the characteristics of the pseudo camera work to the user in an easy-to-understand manner using a still image as in the past. Moreover, even if the technique disclosed in Patent Document 1 is applied, it is difficult to convey the characteristics of the pseudo camera work to the user in an easy-to-understand manner. Further, for example, it is assumed that camera work data indicating pseudo camera work performed by each of a plurality of users is uploaded to the server. In this case, a method for efficiently searching for a camera work desired by the user from pseudo camera work corresponding to each camera work data has not been known.

The present disclosure has been made in view of the above points, and provides an information processing apparatus and an information processing method capable of efficiently searching for camera work desired by a user.

In order to solve the above-mentioned problem, according to a first aspect of the present invention, in the image frame constituting the moving image, the display range displayed by the display unit can change the display range of the moving image according to the moving image playback position. Receiving means for receiving from the terminal device a request for display range information shown for each, the request including first search information used for searching for an area composed of one or more pixels in the image frame of the moving image Storage means for storing second search information associated with each of the plurality of areas, the second search information indicating the characteristics of the areas, and the second search information stored in the storage means Based on search information, first determination means for searching and determining the area associated with the second search information corresponding to the first search information received by the receiving means from the storage means , The display range information indicating the display range including the region determined by the first determining means, characterized in that it comprises a second determining means for determining a display range information to be provided to the terminal device.

According to a second aspect of the present invention, in the information processing apparatus according to the first aspect, the second search information associated with the area indicates a subject appearing in a display range of the moving image including the area, and the receiving unit The request including the first search information indicating a subject is received, and the first determination unit receives the request received by the reception unit based on the second search information associated with the region. The area including the display range in which the subject indicated by the first search information appears is determined, and the second determining means determines the display range information indicating the display range including the area determined by the first determining means. The display range information to be provided to the terminal device is determined.

3. The information processing apparatus according to claim 1, wherein the receiving unit is configured to reproduce the moving image when the first search information is input while the moving image is displayed on the terminal device. The request including the reproduction position information indicating the first search information is received, and the first determination unit is configured to determine the region in the image frame at the reproduction position indicated by the reproduction position information received by the reception unit. The region corresponding to the second search information corresponding to the first search information received by the receiving unit is determined, and the second determination unit is determined by the first determination unit. The display range information indicating the display range including the region thus determined is determined as display range information to be provided to the terminal device.

A fourth aspect of the present invention is an information processing method performed by a computer, wherein the display range displayed by the display means in the image frame constituting the moving image is a reproduction position of the moving image. A display range information request indicating, for each playback position, the display range of a moving image that can be changed according to a first position, which is used to search for an area composed of one or more pixels in the image frame of the moving image. A receiving step for receiving the request including search information from the terminal device, and a second search information associated with each of the plurality of areas, and storing the second search information indicating the characteristics of the areas Based on the second search information stored in the means, the second search information corresponding to the first search information received by the receiving step is associated. A first determination step of searching and determining the determined region, and display range information indicating the display range including the region determined by the first determination step is determined as display range information to be provided to the terminal device And a second determination step.

According to

Claims

1 and 4, it is possible to efficiently search for display range information indicating a display range corresponding to the camera work desired by the user.

According to claim 2, it is possible to efficiently search for display range information indicating a display range including a scene in which a subject such as a person desired by the user appears.

According to claim 3, it is possible to efficiently search the display range information indicating the display range at the timing desired by the user in the entire playback time of the moving image.

It is a figure which shows the example of a schematic structure of the communication system S of this embodiment. It is a conceptual diagram which shows the example which divided | segmented three-dimensional moving image space into several moving image block. It is a figure which shows the example of a virtual screen, and the example of the display range with respect to a virtual screen. It is a figure which shows the example of a virtual screen, and the example of the display range with respect to a virtual screen. It is a figure which shows the example of a virtual screen, and the example of the display range with respect to a virtual screen. It is an example of the camera work data produced | generated by the control part. It is an example of the camera work data produced | generated by the control part. It is an example of the camera work data produced | generated by the control part. 7 is a flowchart showing main processing in the control unit 21 of the client 2. It is a flowchart which shows an example of the input process in step S4 shown to FIG. 5A. It is a figure which shows the example of a screen for inputting scene information. It is a figure which shows the example of a screen for inputting scene information. It is a flowchart which shows the process in the control part 11 of the delivery server 1. FIG. 4 is a flowchart showing processing in a control unit 21 of the client 2. It is a figure which shows the example of a screen for inputting a search key. It is a figure which shows the example of a screen for inputting a search key. It is a figure which shows an example of the main screen MV and the sub screens SV1 to SV5. It is a flowchart which shows the process in the control part 11 of the delivery server 1. FIG. It is a figure which shows the display range when the partial area | region in 1 image frame which comprises a moving image is made into the pixel. It is a figure which shows the display range when the partial area | region in 1 image frame which comprises a moving image is made into a moving image block. It is a conceptual diagram which shows the example complemented so that between the display ranges R31-R33 determined every 5 second interval may change continuously. It is a flowchart which shows the automatic generation process of the camera work data in the control part 11 of the delivery server 1. It is a flowchart which shows the other example of the automatic production | generation process of camera work data. It is a conceptual diagram which shows the other example of the automatic production | generation process of camera work data. It is a conceptual diagram which shows the other example of the automatic production | generation process of camera work data. It is a conceptual diagram which shows the other example of the automatic production | generation process of camera work data. It is a conceptual diagram which shows the other example of the automatic production | generation process of camera work data. It is a conceptual diagram which shows the other example of the automatic production | generation process of camera work data.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[1. Configuration and operation overview of communication system S]
First, with reference to FIG. 1 etc., the structure and operation | movement outline | summary of the communication system of this embodiment are demonstrated. FIG. 1 is a diagram illustrating a schematic configuration example of a communication system S of the present embodiment. As shown in FIG. 1, the communication system S includes a distribution server 1 and a plurality of clients 2. The distribution server 1 is an example of an information processing apparatus according to the present disclosure. The client 2 is an example of a terminal device according to the present disclosure. Further, the distribution server 1 and the client 2 can communicate with each other via the network NW. The network NW is configured by, for example, the Internet.

The distribution server 1 accepts upload of content from the client 2 or camera work data of the content, for example. For example, the distribution server 1 transmits content to the client 2 in response to a content request from the client 2. The content includes moving image data. The moving image data is data representing a moving image in which the display range displayed by the display means can be changed in accordance with the reproduction position of the moving image within the image frame constituting the moving image. An example of such a moving image is a panoramic moving image. A panoramic video is a video in which a subject is shot by a camera equipped with a lens capable of shooting a wide range, for example, with a high-resolution camera. Examples of lenses capable of photographing a wide range include a wide lens, a fisheye lens, and a 360 lens. The playback position is an elapsed time from the start of playback of moving image data. Note that the content may include audio data. The content is transmitted by streaming distribution via the network NW, for example. The client 2 receives the content streamed from the distribution server 1.

Also, the distribution server 1 transmits the camera work data of the content to the client 2 in response to a request for camera work data from the client 2, for example. The camera work data is an example of display range information indicating, for each reproduction position, a display range displayed by the display unit in an image frame constituting a moving image. This display range corresponds to a drawing area drawn on the screen of the display means in one image frame. In other words, the display range is a range cut out from the shooting range defined by the image frame. Such a display range is designated by, for example, pseudo camera work (hereinafter referred to as “pseudo camera work”). For example, assume that image frames constituting a moving image are projected onto a virtual screen arranged in a three-dimensional virtual space, and that there is a person who views the moving image in the three-dimensional virtual space. The term “pseudo camera work” refers to determining at least one of the viewpoint position, the line-of-sight direction, and the visual field area of a person who views a moving image projected on a virtual screen, for example. In other words, pseudo camera work refers to determining the orientation of the virtual camera, the width of the field of view, and the like. Here, the virtual camera refers to a virtual camera that determines a drawing area in an image frame constituting a moving image. The pseudo camera work can be reproduced by the camera work data (hereinafter referred to as “pseudo camera work data”). For example, one pseudo camera work data does not necessarily indicate the display range in all image frames included in the reproduction time from the reproduction start to the reproduction end of the moving image data. That is, one pseudo camera work data may indicate a display range in an image frame included in a partial time range in the reproduction time. For example, the client 2 displays the moving image according to the display range indicated by the acquired pseudo camera work data while receiving the content by streaming.

The distribution server 1 can be connected to the storage device 3. The storage device 3 is configured by, for example, a hard disk drive (HDD). The storage device 3 is provided in the distribution server 1. Alternatively, the storage device 3 may be provided in a server different from the distribution server 1. The storage device 3 stores Web page data transmitted to the client 2 in response to a request from the client 2. The storage device 3 further includes a moving image data storage area 31a, an audio data storage area 31b, a work file storage area 31c, and a scene information storage area 31d.

The moving image data storage area 31a stores a plurality of moving image data. The moving image data stored in the moving image data storage area 31 a can be shared among a plurality of clients 2 that can access the distribution server 1. A plurality of audio data is stored in the audio data storage area 31b. The audio data stored in the audio data storage area 31 b can be shared among a plurality of clients that can access the distribution server 1.

Next, the work file is stored in the work file storage area 31c in association with each content. The work file stores content titles, pseudo camera work data, and the like. In some cases, the pseudo camera work data is given scene information. The scene information is information indicating the characteristics of a moving image scene. Here, the moving image scene indicates a range that is a segment of a certain operation on the moving image. Such scenes include, for example, a scene where a singer sings, a scene played by an actor, or a scene that shows the state of an event. The scene information includes, for example, text information indicating a subject such as a singer or an actor appearing in the scene, identification information for identifying the subject, and the like. For example, a work ID for identifying the work file is given to the work file. Note that the work file may include a content ID for identifying the content. The work file stored in the work file storage area 31c includes, for example, a work file uploaded from the client 2. One content may be associated with a plurality of work files. For example, the pseudo camera work data indicating the display range designated by the pseudo camera work performed by each of a plurality of users is associated with a certain content.

In the scene information storage area 31d, a scene information database is stored in association with each content. The scene information database is a database in which scene information can be registered in association with each partial area and each reproduction position in an image frame constituting a moving image. A partial area is an area composed of one or more pixels in the image frame. The partial area is determined by the reproduction position of the moving image and the position on the image frame at the reproduction position. The scene information registered in the scene information database is an example of second search information indicating the characteristics of the area. The scene information database may be configured so that information indicating features other than the scene of the moving image is registered in association with the partial area. Note that the partial area may be one pixel in one image frame, but in this case, the load for managing scene information increases. Therefore, the video is divided into three-dimensional blocks of “vertical pixels (plural pixels) H × horizontal pixels (plural pixels) W × playback time (multiple frames) F”, and scene information is associated with each block and registered. It is desirable that Such a block is hereinafter referred to as a “moving image block”. FIG. 2 is a conceptual diagram illustrating an example in which a three-dimensional moving image space is divided into a plurality of moving image blocks. In the example of FIG. 2, each moving image block is assigned a unique block ID. Each moving image block is associated with a coordinate position on the image frame F constituting the moving image and a reproduction position. This playback position is, for example, the playback position of the first image frame among a plurality of image frames included in the moving image block. The image frame in this case is an image frame divided as shown in FIG. In the scene information database, for example, scene information given to the pseudo camera work data uploaded from the client 2 is registered in association with the block ID. The scene information in this case is scene information posted by the user of the client 2. The same or similar scene information may be posted by a plurality of users for the same moving image block. Therefore, as shown in FIG. 2, the number of postings of scene information is registered in the scene information database in association with the block ID. As shown in FIG. 2, scene information and the number of postings of scene information are not necessarily registered in all moving image blocks. Further, the scene information database may be generated and updated on the system operator side, for example, without depending on the scene information provided from the client 2.

And the delivery server 1 is provided with the control part 11 and the interface part 12, as shown in FIG. The control unit 11 includes a CPU, ROM, RAM, and the like as a computer. The control unit 11 is an example of a reception unit, a first determination unit, and a second determination unit of the present disclosure. The control unit 11 performs content transmission or reception control, pseudo camera work data transmission or reception control, and the like. When the control unit 11 receives a request for pseudo camera work data from the client 2, the control unit 11 determines pseudo camera work data to be returned to the client 2 based on the request. The method for determining the pseudo camera work data will be described later.

Next, as shown in FIG. 1, the client 2 includes a control unit 21, a storage unit 22, a video RAM 23, a video control unit 24, an operation processing unit 25, an audio control unit 26, an interface unit 27, a bus 28, and the like. Configured. These components are connected to the bus 28. A display unit 24 a including a display is connected to the video control unit 24. The display is an example of display means. An operation unit 25 a is connected to the operation processing unit 25. Examples of the operation unit 25a include a mouse, a keyboard, and a remote controller. A touch panel serving both as the display unit 24a and the operation unit 25a may be applied. The control unit 21 receives an operation instruction from the operation unit 25 a by the user via the operation processing unit 25. The user can perform the above-described pseudo camera work operation using the operation unit 25a. A speaker 26 a is connected to the audio control unit 26. The interface unit 27 is connected to the network NW.

The control unit 21 includes a CPU, ROM, RAM, and the like as a computer. The control unit 21 has a timer function. The storage unit 22 is configured by, for example, a hard disk drive (HDD). The storage unit 22 stores an OS (Operating System), player software, and the like. The player software is a program for playing back content. Note that the player software may be downloaded from a predetermined server connected to the network NW, for example. Alternatively, the player software may be recorded on a recording medium and read via a drive of the recording medium, for example.

The control unit 21 functions as a player that reproduces content by executing player software. The control unit 21 sequentially acquires the contents streamed from the distribution server 1 by the function of the player and reproduces the contents. The RAM in the control unit 21 is provided with a buffer memory. In the buffer memory, for example, moving image data included in the content streamed from the distribution server 1 is temporarily stored. The buffer memory temporarily stores, for example, pseudo camera work data distributed from the distribution server 1. The control unit 21 outputs moving image data from the buffer memory to the video RAM 23. In the video RAM 23, a frame buffer is provided in the RAM. For example, in the frame buffer, image data of a portion corresponding to the display range indicated by the pseudo camera work data in the image frame constituting the moving image reproduced by the moving image data is written. In accordance with the control signal from the control unit 21, the video control unit 24 displays the image data written in the frame buffer by drawing it on the corresponding screen. For example, audio data may be included in the content held in the buffer memory from the distribution server 1. In this case, the control unit 21 reproduces the audio data from the buffer memory and outputs it to the audio control unit 26. The voice control unit 26 generates an analog voice signal from the voice data, and outputs the generated analog voice signal to the speaker 26a.

Further, the control unit 21 receives an instruction of a display range of a part of the image frame constituting the moving image being displayed on the display screen by the user's pseudo camera work operation. For example, the control unit 21 receives an instruction to change the display range displayed in the image frame constituting the moving image being displayed on the screen. The control unit 21 changes and displays the display range displayed on the screen in response to the change instruction. In other words, the user can change the display range of the moving image being displayed on the screen by changing at least one of the viewpoint position, the line-of-sight direction, and the visual field area by operating the pseudo camera work. For example, assume that image frames constituting a moving image are projected onto a virtual screen arranged in a three-dimensional virtual space, and that there is a person who views the moving image in the three-dimensional virtual space. The viewpoint position is a position where the person is watching the moving image. The line-of-sight direction is the direction of the line of sight of the person's moving image. The visual field area is, for example, the area of a region in the range of the visual field of the person on the virtual screen arranged in the three-dimensional virtual space. The visual field area may be the range of the visual field of the person. Then, the display range of the moving image is determined on the virtual screen by the operation of the pseudo camera work. That is, an image in the range determined as the display range is cut out from the virtual screen, and the cut out image is displayed on the screen.

3A to 3C are diagrams showing an example of a virtual screen and an example of a display range for the virtual screen. In the example of FIG. 3A, a screen SC1 is defined as a virtual screen. The screen SC1 is a rectangular flat screen, and a moving image is projected on the rectangular plane. The display range R1 on the screen SC1 is defined by, for example, an X coordinate, a Y coordinate, a width, and a height. For example, the upper left vertex of the screen SC1 is set as the origin in the coordinate system of the screen SC1. The X coordinate and the Y coordinate define the viewpoint position. The X coordinate is the horizontal coordinate of the upper left vertex of the display range R1, and the Y coordinate is the vertical coordinate of the upper left vertex of the display range R1. For example, a point that is a predetermined distance away from the screen SC1 in the three-dimensional virtual space may be assumed as the viewpoint. For example, a line passing through the viewpoint and perpendicularly intersecting the screen SC1 is defined as a line of sight. The point where the line of sight and the screen SC1 intersect is the center of the display range R1. The width and height define the viewing area. The width and the height are the horizontal length and the vertical length of the display range R1. The line-of-sight direction is determined in advance.

In the example of FIG. 3B, a screen SC2 is defined as a virtual screen. The screen SC2 is a cylindrical screen, and a moving image is projected on the side surface of the cylinder. The side surface of the cylinder is an example of a virtual solid surface. For example, a cylindrical panoramic video is projected on the screen SC2. The panoramic video is, for example, an omnidirectional video. The panoramic video may be a partial orientation video with a viewing angle narrower than 360 degrees. The display range R2 on the screen SC2 is defined by, for example, an azimuth angle, a horizontal viewing angle, and a height. The azimuth determines the viewing direction. For example, let the midpoint of the central axis of the cylinder of the screen SC2 be the viewpoint. The viewpoint is the origin in the coordinate system of the three-dimensional virtual space, and the center axis of the screen SC2 is the Z axis. The X axis passes through the origin and is perpendicular to the Y axis and the Z axis. The Y axis passes through the origin and is perpendicular to the X axis and the Z axis. The azimuth determines the direction of the line of sight from the viewpoint. The line of sight is, for example, perpendicular to the Z axis. The azimuth angle is, for example, an angle between the X axis and the line of sight. The horizontal viewing angle and height define the viewing area. The horizontal viewing angle is an angle indicating the range of the horizontal visual field centered on the direction of the line of sight. The height is the vertical length of the display range R2. Based on the azimuth angle, the horizontal viewing angle, and the height, a quadrangular pyramid indicating a viewing range in the three-dimensional virtual space is defined. This square pyramid is the view volume. A view volume refers to a range that is subject to projection transformation in a three-dimensional virtual space. Although the actual view volume is a quadrangular pyramid, a quadrangular pyramid is used for convenience of explanation. The vertex of the view volume is the viewpoint, and the line of sight passes through the center of the bottom surface of the view volume. Of the side surfaces P21 to P24 of the view volume, the angle formed by the side surface P21 and the side surface P22 parallel to the Z axis is the horizontal viewing angle. The length in the vertical direction of the surface where the view volume and the screen SC2 intersect is the height. The surface where the view volume and the screen SC2 intersect is the display range R2. The viewpoint position is determined in advance.

In the example of FIG. 3C, a screen SC3 is defined as a virtual screen. The screen SC3 is a spherical screen, and a moving image is displayed on a spherical surface. For example, a spherical panoramic video is displayed on the screen SC3. The display range R3 on the screen SC3 is defined by, for example, an azimuth angle, an elevation angle, a horizontal viewing angle, and a vertical viewing angle. The azimuth angle and the elevation angle determine the line-of-sight direction. In the three-dimensional virtual space, for example, the viewpoint is located within a range surrounded by the screen SC3. For example, the viewpoint is the center of the sphere of the screen SC3. The viewpoint is the origin in the coordinate system of the three-dimensional virtual space, and the vertical coordinate axis is the Z axis. The X axis passes through the origin and is perpendicular to the Y axis and the Z axis. The Y axis passes through the origin and is perpendicular to the X axis and the Z axis. The azimuth angle is, for example, an angle formed by the XZ plane and the line of sight. The elevation angle is, for example, an angle formed by the XY plane and the line of sight. The horizontal viewing angle and the vertical viewing angle define the viewing area. The horizontal viewing angle is an angle indicating the range of the horizontal visual field centered on the direction of the line of sight. The vertical viewing angle is an angle indicating the range of the vertical visual field around the direction of the line of sight. A line on the XY plane that passes through the origin and intersects the line of sight perpendicularly is defined as a vertical rotation axis of the line of sight. A line that passes through the origin and perpendicularly intersects the line of sight and the vertical rotation axis is defined as the horizontal rotation axis of the line of sight. Based on the azimuth angle, the elevation angle, the horizontal viewing angle, and the vertical viewing angle, a quadrangular pyramid indicating a viewing range in the three-dimensional virtual space is defined. This square pyramid is the view volume. The vertex of the view volume is the viewpoint, and the line of sight passes through the center of the bottom surface of the view volume. Of the side surfaces P31 to P34 of the view volume, the angle formed by the side surface P31 parallel to the Z axis and the side surface P32 is the horizontal viewing angle. The angle formed by the side surface P33 and the side surface P34 is the vertical viewing angle. The surface where the view volume and the screen SC3 intersect is the display range R3. The viewpoint position is determined in advance. The perspective transformation converts the three-dimensional coordinates of the display range on the virtual screen into two-dimensional coordinates based on the viewpoint position, the line-of-sight direction, and the visual field area. Based on the converted two-dimensional coordinates, for example, it is possible to specify which part of the image frame constituting the panoramic video is within the display range. When at least one of the azimuth angle and the elevation angle changes to change the viewing direction, the display range R3 changes according to the viewing direction. In addition, when at least one of the vertical viewing angle and the horizontal viewing angle changes to change the viewing area, the display range R3 changes according to the viewing area. That is, the display range R3 is a range corresponding to the line-of-sight direction and the visual field area. In general, the screen SC3 may be a solid that completely covers the viewpoint, and may be a screen having a cubic shape, for example.

Note that which of the screens SC1 to SC3 is used may be determined according to the type of moving image data, for example. For example, the screen SC1 is determined for a video other than a panoramic video, the screen SC2 is determined for a cylindrical panoramic video, and the screen SC3 is determined for a spherical panoramic video. Good.

In addition, the control unit 21 generates pseudo camera work data indicating the display range of the moving image being displayed on the screen. 4A to 4C are examples of pseudo camera work data generated by the control unit 21. FIG. FIG. 4A shows an example in which the virtual screen is a rectangular screen SC1. FIG. 4B shows an example in which the virtual screen is a cylindrical screen SC2. FIG. 4C shows an example in which the virtual screen is a spherical screen SC3. In the example of FIG. 4A, for example, when the aspect ratio is determined to be 16: 9, if one of the width and the height is determined, the other is determined. And any one of height may be included. In the examples of FIGS. 4A to 4C, pseudo camera work data indicating display ranges in image frames at respective reproduction positions such as 0 milliseconds, 16 milliseconds, 33 milliseconds, and 49 milliseconds are shown. ing. Note that 16 milliseconds is comparable to the display refresh rate (60 Hz).

[2. Operation of communication system S]
Next, the operation of the communication system S of the present embodiment will be described.

(2-1. Pseudo camera work data upload operation)
First, an operation when pseudo camera work data is uploaded to the distribution server 1 during content reproduction in the client 2 will be described.

(2-1-1. Processing of Client 2)
With reference to FIG. 5, the process of the client 2 in the upload operation of the pseudo camera work data will be described. FIG. 5A is a flowchart showing main processing in the control unit 21 of the client 2. FIG. 5B is a flowchart showing an example of the input process in step S4 shown in FIG.

For example, when the player software is activated in the client 2, the client 2 transmits a page request to the distribution server 1. Then, the client 2 receives the Web page transmitted from the distribution server 1 in response to the page request and displays it on the display in the display unit 24a. On this Web page, for example, content information is displayed in a selectable manner. The content information displayed on the Web page is information on a part of the plurality of content uploaded to the distribution server 1. This corresponds to, for example, content information recommended for the user or content information searched based on a keyword input by the user. The content information includes, for example, information such as the content title.

When the user operates the operation unit 25a to select content information to be played back, the main process shown in FIG. 5A is started. When the main process is started, the control unit 21 initializes the work file and starts playing the selected content (step S1). Thereby, the moving image reproduced by the moving image data included in the content streamed from the distribution server 1 is displayed on the main screen of the display. The work file is a file for uploading pseudo camera work data. The work file is stored in the storage unit 22 in advance. With the initialization of the work file, the title of the content that has been played back is set in the work file. A content ID may be set in the work file.

In the example of FIG. 5A, it is assumed that the operation input of the pseudo camera work for the moving image displayed on the main screen is set to “manual”. Thereby, the display range of the moving image displayed on the main screen does not depend on, for example, the pseudo camera work data acquired from the distribution server 1 and is changed by the user's pseudo camera work operation.

Next, the control unit 21 stores pseudo camera work data indicating the moving image display range at the current reproduction position in the work file (step S2). Next, the control unit 21 determines whether or not there is scene information input by the user at the current playback position (step S3).

FIGS. 5C and 5D are diagrams showing examples of screens for inputting scene information. On the screen shown in FIG. 5C, a reproduced moving image is displayed, and buttons corresponding to a plurality of persons appearing in the scene of the moving image being displayed are displayed so as to be selectable. This person is a singer, for example, and is an example of a subject. These buttons are buttons for inputting scene information (hereinafter referred to as “scene information input buttons”). Each scene information input button is displayed, for example, when the user gives an instruction to display the scene information input button via the operation unit 25a during content reproduction. Each scene information input button is associated with an ID (identification information) for identifying a person. This association is performed, for example, when content is generated on the distribution server 1 side. In the example of FIG. 5C, an ID for identifying a person is associated with the scene information input button. However, in the case of a moving image in which subjects such as animals other than people or buildings are displayed, IDs for identifying these subjects are associated with the scene information input buttons. On the other hand, on the screen shown in FIG. 5D, a reproduced moving image is displayed and a text input field for inputting text information such as a character string is displayed. The text input field is displayed, for example, when the user gives an instruction to display the text input field via the operation unit 25a during content reproduction.

For example, in the display state of the screen shown in FIG. 5C, when the scene information input button corresponding to the person the user likes is selected, the control unit 21 determines that there is scene information input (step S3: YES). ), The process proceeds to the input process of step S4. The selection of the scene information input button is performed, for example, by clicking the scene information input button with a mouse or tapping the scene information input button with a finger or a pen. Further, for example, when the user designates a text input field in the display state of the screen shown in FIG. 5D, the control unit 21 determines that there is an input of scene information (step S3: YES), and in step S4 Proceed to input processing. The designation of the text input field is performed, for example, by clicking the text input field with the mouse or tapping the text input field with a finger or a pen. On the other hand, if it is determined that no scene information is input by the user (step S3: NO), the process proceeds to step S6.

In the input process of step S4, as shown in FIG. 5B, the control unit 21 determines whether or not the input of the scene information is “ID input” by the scene information input button. For example, when the scene information input button is selected in the display state of the screen shown in FIG. 5C, it is determined that “ID input” is made by the scene information input button (step S41: YES), and the process proceeds to step S42. On the other hand, for example, when the text input field is designated in the display state of the screen shown in FIG. 5D, it is determined that it is not “ID input” by the scene information input button (step S41: NO), and the process proceeds to step S43.

In step S42, the control unit 21 stores the ID corresponding to the selected scene information input button as scene information, and displays reproduction position information indicating the reproduction position of the moving image when the scene information input button is selected. It is stored in association with the scene information. On the other hand, in step S43, the control unit 21 determines that the input is text, and stores playback position information indicating the playback position of the moving image when the user inputs the first character from, for example, a keyboard. It should be noted that the playback position information indicating the playback position of the moving image at any point in time from the second character to the completion of text input is stored instead of the playback position of the moving image when the first first character is input. It may be configured. Then, the control unit 21 stores the text information input in the text input field by the user as scene information in association with the reproduction position information stored in step S43 (step S44).

Returning to the processing shown in FIG. 5A, in step S5, the control unit 21 uses the pseudo camera work data stored in step S2 to store the scene information and the reproduction position information stored in the processing in FIG. 5B. Store in a work file in association with. Thereby, scene information and reproduction position information are added to the pseudo camera work data. Next, the control unit 21 determines whether or not there is an instruction to upload pseudo camera work data by the user (step S6). When it is determined that there is an upload instruction (step S6: YES), the process proceeds to step S7. On the other hand, when it is determined that there is no upload instruction (step S6: NO), the process proceeds to step S9.

In step S7, the control unit 21 transmits at least a work file storing pseudo camera work data to the distribution server 1 via the network NW. Thereby, a part of the pseudo camera work data from the start to the end of the playback of the moving image is uploaded. Next, the control unit 21 clears the contents of the work file (step S8) and returns to step S2. As a result, an empty work file remains in the storage unit 22.

In step S9, the control unit 21 determines whether or not the content has been played to the end position of the content. If it is determined that the content has not been played back to the end position (step S9: NO), the process returns to step S2 and the above process is repeated. On the other hand, when it is determined that the content has been played to the end position (step S9: YES), the process proceeds to step S10.

In step S10, the control unit 21 displays an upload button on the screen together with a message asking the user whether to upload the pseudo camera work data. Next, the control unit 21 determines whether there is an instruction to upload pseudo camera work data from the user (step S11). For example, when the user selects the upload button, the control unit 21 determines that there is an upload instruction (step S11: YES), and proceeds to step S12. On the other hand, when it is determined that there is no upload instruction (step S11: NO), the processing shown in FIG. In step S12, the work file storing at least the pseudo camera work data is transmitted to the distribution server 1 via the network NW, and the process shown in FIG.

(2-1-2. Processing of Distribution Server 1)
With reference to FIG. 6, the process of the delivery server 1 in the upload operation of the pseudo camera work data will be described. FIG. 6 is a flowchart showing processing in the control unit 11 of the distribution server 1. The process shown in FIG. 6 is started when the distribution server 1 receives a work file from the client 2. When the process shown in FIG. 6 is started, the control unit 11 stores the received work file in the work file storage area 31c (step S101). Next, the control unit 11 specifies the moving image data of the content from the moving image data storage area 31a from the content title set in the received work file (step S102).

Next, the control unit 11 specifies the pseudo camera work data to which the scene information and the reproduction position information are given from the pseudo camera work data stored in the received work file (step S103). Next, the control unit 11 is assigned to the coordinate position and reproduction position associated with the moving image block constituting the moving image data identified in step S102, the display range indicated by the identified pseudo camera work data, and the pseudo camera work data. Based on the playback position information, one or more moving image blocks are specified (step S104). For example, the reproduction position information given to the specified pseudo camera work data in which the coordinate position is included in the display range indicated by the specified pseudo camera work data among the plurality of video blocks constituting the moving picture data A moving image block including a part of the image frame at the reproduction position indicated by is identified.

Next, the control unit 11 determines whether or not text information is included in the scene information given to the pseudo camera work data specified in step S103 (step S105). When it is determined that text information is included in the scene information (step S105: YES), the process proceeds to step S106. On the other hand, when it is determined that the text information is not included in the scene information (step S105: NO), the process proceeds to step S108. In this case, for example, an ID for identifying a person is included in the scene information.

In step S106, the control unit 11 parses text information included in the scene information and extracts word information. Thereby, for example, when the text information is a sentence, one or more words (words) are extracted from the sentence. Next, the control unit 11 extracts a keyword from the word information extracted in step S106 as scene information to be registered using, for example, a keyword dictionary stored in advance (step S107).

In step S108, the control unit 11 already associates the scene information given to the pseudo camera work data or the scene information that is the same as or similar to the scene information extracted in step S107 with the moving image block specified in step S104. And whether it is registered in the scene information database. Here, this scene information database is a scene information database corresponding to the moving image data specified in step S102. The scene information database is specified from the scene information storage area 31d before the process of step S108. Further, whether or not the scene information is similar is determined by using, for example, a previously stored synonym dictionary or synonym dictionary. For example, scene information that is synonymous or synonymous with the scene information extracted in step S107 is determined as similar scene information. If it is determined that the moving image block identified in step S104 is already associated with the moving image block and is not registered in the scene information database (step S108: NO), the process proceeds to step S109. On the other hand, if it is determined that it is already associated with the moving image block specified in step S104 and registered in the scene information database (step S108: YES), the process proceeds to step S110.

In step S109, the control unit 11 registers the scene information added to the pseudo camera work data or the scene information extracted in step S107 in the scene information database in association with the moving image block specified in step S104. The process shown in FIG. As described above, the scene information input by the user is registered in the scene information database in association with the moving image block in which the scene corresponding to the scene information is displayed. Therefore, it is possible to efficiently generate a scene information database that reflects the preferences of a plurality of users. In step S110, the control unit 11 increments the posting number of the scene information registered in association with the moving image block specified in step S104 by 1, and ends the process shown in FIG. That is, “1” is added to the number of postings associated with the moving image block identified in step S104 and is updated and registered in the scene information database. As described above, the posting number of the scene information input by the user is registered in the scene information database in association with the moving image block in which the scene corresponding to the scene information is displayed. Therefore, it is possible to efficiently generate a scene information database that can determine which moving image block is a moving image block in which a scene with a high degree of attention posted with more scene information from the user is displayed.

(2-2. Pseudo camera work data request operation)
Next, the operation when pseudo camera work data is requested from the client 2 to the distribution server 1 will be described.

(2-2-1. Processing of Client 2)
With reference to FIG. 7, the processing of the client 2 in the request operation for pseudo camera work data will be described. FIG. 7A is a flowchart showing processing in the control unit 21 of the client 2.

The processing shown in FIG. 7A is started, for example, when the user gives a search instruction for pseudo camera work data via the operation unit 25a during execution of the player software. When the process shown in FIG. 7A is started, the control unit 21 determines whether or not there is a search key input by the user (step S21). The search key is, for example, a key for searching for a scene in which a user likes a person. The search key is an example of first search information used for searching a partial area of a moving image.

FIGS. 7B and 7C are diagrams showing examples of screens for inputting a search key. On the screen shown in FIG. 7B, a reproduced moving image is displayed, and buttons corresponding to a plurality of persons appearing in the scene of the moving image being displayed are selectable. These buttons are buttons for inputting search keys (hereinafter referred to as “search key input buttons”). Each search key input button is displayed, for example, when the user gives an instruction to display the search key input button via the operation unit 25a during content reproduction. Each search key input button is associated with an ID for identifying a person, like the scene information input button shown in FIG. On the other hand, on the screen shown in FIG. 7C, a reproduced moving image is displayed and a search keyword input field for inputting text information including the search keyword is displayed. The search keyword input field is displayed, for example, when the user gives an instruction to display the search keyword input field via the operation unit 25a during content reproduction.

For example, when the search key input button corresponding to the person the user likes is selected in the display state of the screen shown in FIG. 7B, the control unit 21 determines that there is a search key input by the user (step S21). : YES), the process proceeds to step S22. The selection method of the search key input button is the same as that of the scene information input button. Further, for example, when the user designates a search keyword input field in the display state of the screen shown in FIG. 7C, the control unit 21 determines that the search key is input by the user (step S21: YES), and step S22. Proceed to The method for specifying the search keyword input field is the same as that for the text input field. On the other hand, if it is determined that the user has not entered a search key (step S21: NO), the process proceeds to step S28.

In step S22, the control unit 21 determines whether or not the input of the search key is “ID input” by the search key input button. For example, when the search key input button is selected in the display state of the screen shown in FIG. 7B, it is determined that “ID input” is made by the search key input button (step S22: YES), and the process proceeds to step S23. On the other hand, for example, when the search keyword input field is specified in the display state of the screen shown in FIG. 7C, it is determined that it is not “ID input” by the search key input button (step S22: NO), and the process proceeds to step S24. .

In step S23, the control unit 21 stores the ID corresponding to the selected search key input button as a search key. Here, the reproduction position information indicating the reproduction position of the moving image when the search key input button is selected may be stored in association with the search key. In step S24, the control unit 21 stores the text information input in the search keyword input field by the user as a search key. Here, for example, playback position information indicating the playback position of the moving image when the user inputs the first character from a keyboard or the like may be stored in association with the search key. It should be noted that the playback position information indicating the playback position of the moving image at any point in time from the second character to the completion of text input is stored instead of the playback position of the moving image when the first first character is input. It may be configured.

Next, the control unit 21 transmits a request for pseudo camera work data to the distribution server 1 via the network NW (step S25). This request is a request for pseudo camera work data. The request for pseudo camera work data includes, for example, the title of the content selected by the user and the search key stored in step S23 or step S24. The request for pseudo camera work data may include a content ID for identifying the content. Further, the request for pseudo camera work data may be configured to include reproduction position information associated with the search key. And the control part 21 receives the work file transmitted from the delivery server 1 according to the request | requirement of pseudo camera work data (step S26). Next, the control unit 21 displays a moving image on a sub-screen different from the main screen according to the pseudo camera work data stored in the received work file (step S27). Here, a plurality of work files may be received from the distribution server 1. In this case, a plurality of sub screens are displayed as thumbnail screens. With this thumbnail screen, a list of pseudo camera work data can be displayed.

FIG. 7D is a diagram showing an example of the main screen MV and the sub screens SV1 to SV5. In the example of FIG. 7D, moving images are displayed on the sub screens SV1 to SV5 according to the respective pseudo camera work data received from the distribution server 1. The display range indicated by the received pseudo camera work data includes scenes included in the moving image block searched based on the search key described above. Further, the playback positions of the moving images displayed on the sub screens SV1 to SV5 are the same. That is, the image frames displayed on the sub screens SV1 to SV5 are the same, but the display ranges in the image frames are different from each other. This means that, for example, the angle of the virtual camera and the field of view are different. The user can see the scene he / she wants to see during the content reproduction by the moving images displayed on the sub-screens SV1 to SV5. When any of the moving images displayed on the sub screens SV1 to SV5 is instructed to be selected by the user, the moving image being displayed on the main screen MV is switched to the selected moving image.

In step S28, for example, it is determined whether or not there is a player termination instruction from the user. If it is determined that the player has instructed to end (step S28: YES), the processing shown in FIG. 7A ends. On the other hand, if it is determined that there is no instruction to end the player (step S28: NO), the process returns to step S21 and the above process is continued.

(2-2-2. Processing of Distribution Server 1)
With reference to FIG. 8, the processing of the distribution server 1 in the request operation for pseudo camera work data will be described. FIG. 8 is a flowchart showing processing in the control unit 11 of the distribution server 1. The process illustrated in FIG. 8 is started when the distribution server 1 receives a request for pseudo camera work data from the client 2. When the process illustrated in FIG. 8 is started, the control unit 11 acquires a search key from the received request (step S111). Here, the request may include text information composed of sentences. In this case, the control unit 11 acquires a search keyword as a search key from the sentence by parsing the text information. The request may include playback position information. In this case, the control unit 11 acquires a search key and reproduction position information from the request.

Next, the control unit 11 specifies the scene information database associated with the title or content ID of the content included in the request from the scene information storage area 31d (step S112). Next, the control unit 11, based on the search key acquired in step S <b> 111 and the scene information registered in the scene information database specified in step S <b> 112, a moving image in which scene information corresponding to the search key is associated. A block is searched from the scene information database (step S113). Here, the scene information corresponding to the search key is scene information that matches the search key. When there are a plurality of search keys, the scene information corresponding to the search key is scene information including all the search keys. Note that playback position information may be acquired in step S111. In this case, the control unit 11 refers to the scene information database, and from among the video blocks in the image frame at the playback position indicated by the acquired playback position information, the video block associated with the scene information corresponding to the search key Search for.

Next, the control unit 11 determines whether or not a moving image block has been searched as a result of the search in step S113 (step S114). And when it determines with a moving image block not being searched (step S114: NO), it progresses to step S115. In step S115, the control unit 11 notifies the client 2 of information indicating that there is no scene to be searched, and ends the process illustrated in FIG. On the other hand, when it is determined that a moving image block has been searched (step S114: YES), the process proceeds to step S116.

In step S116, the control unit 11 determines the moving image block searched in step S113 as a moving image block including a scene to be searched. For example, a scene including a moving image block in which a subject such as a person indicated by the search key appears is determined. Next, the control unit 11 acquires pseudo camera work data indicating the display range including the moving image block determined in step S116 from the work file storage area 31c (step S117). That is, pseudo camera work data that passes through the moving image block determined in step S116 is acquired.

Next, the control unit 11 determines whether or not there is a predetermined number or more of the pseudo camera work data acquired in Step S117 (Step S118). Here, the predetermined number is set to the number of sub-screens in the client 2, for example. And when it determines with the pseudo camera work data acquired by step S117 not being more than predetermined number (step S118: NO), it progresses to step S119. On the other hand, when it is determined that there is a predetermined number or more of the pseudo camera work data acquired in step S117 (step S118: YES), the process proceeds to step S120.

In step S119, the control unit 11 determines the pseudo camera work data acquired in step S117 as pseudo camera work data to be provided to the user of the client 2. In step S120, the control unit 11 ranks the pseudo camera work data acquired in step S117 based on the number of postings of scene information associated with the moving image block determined in step S116. For example, the control unit 11 ranks the pseudo camera work data acquired in step S117 in descending order of the total number of postings associated with each moving image block including the display range indicated by the pseudo camera work data. Then, the control unit 11 determines the upper predetermined number of pseudo camera work data having a higher ranking as pseudo camera work data to be provided to the user of the client 2 (step S121). Here, for example, the upper predetermined number may be set based on the number of sub-screens that can be displayed by the client 2 (five in the example of FIG. 7D). In this case, the request described above includes information indicating the number of sub screens. Thereby, the pseudo camera work data passing through the moving image block with a high degree of attention where more scene information is posted can be determined as the pseudo camera work data to be provided to the user of the client 2.

Next, the control unit 11 transmits the work file storing the pseudo camera work data determined in step S119 or step S121 to the client 2 (step S122), and ends the process shown in FIG.

In addition, in the request operation of the pseudo camera work data, the case where the search key is input by the user when the moving image is displayed on the main screen is shown as an example. However, the input of the search key by the user and the transmission of the request from the client 2 to the distribution server 1 may be performed when a moving image is not displayed on the main screen. For example, a request including a search key input from the mobile terminal of the user of the client 2 may be transmitted to the distribution server 1. In this case, after receiving the request, the control unit 11 of the distribution server 1 does not transmit the work file storing the pseudo camera work data determined in step S119 or step S121 to the mobile terminal that has transmitted the request. The control unit 11 of the distribution server 1 stores the work file in the work file storage area 31c in association with the user ID for identifying the client 2 and the user of the mobile terminal. Thereafter, when the client 2 accesses the distribution server 1 and, for example, the user is logged in using the user ID, the distribution server 1 transmits the work file stored in association with the user ID to the client 2. To do. Thereby, the client 2 displays a moving image on a sub-screen different from the main screen according to the pseudo camera work data stored in the received work file.

As described above, according to the above-described embodiment, when the distribution server 1 receives a request for pseudo camera work data from the client 2, the distribution server 1 is stored in advance in association with the partial area in the image frame constituting the moving image. Based on the scene information, a partial area associated with the scene information corresponding to the search key included in the request is determined, and pseudo camera work data indicating a display range including the determined partial area is transmitted to the user of the client 2 It is determined as pseudo camera work data to be provided. Therefore, the pseudo camera work data indicating the display range corresponding to the pseudo camera work desired by the user who inputs the search key can be efficiently retrieved and provided to the user. For example, pseudo camera work data indicating a display range including a scene in which a subject such as a person the user wants to see can be provided to the user.

In addition, when the request includes playback position information indicating the playback position of the moving image when the search key is input by the user, the distribution server 1 determines whether the playback position indicated by the playback position information is within the partial area in the image frame. Then, the partial area associated with the scene information corresponding to the search key included in the request is determined. Therefore, the pseudo camera work data indicating the display range at the timing desired by the user in the entire playback time of the moving image can be efficiently searched and provided to the user.

In step S117 shown in FIG. 8, the control unit 11 generates pseudo camera work data indicating the display range including the moving image block determined in step S116, based on the scene information database specified in step S112. You may comprise so that it may acquire. For example, the control unit 11 selects a plurality of upper video blocks that are centered on the video block with the highest number of postings or the highest number of postings among the postings associated with the video block in the scene information database described above. A display range centered on the center of gravity is determined for each playback position of the moving image block, and pseudo camera work data indicating the determined display range for each playback position is generated.

FIG. 9A is a diagram showing a display range when a partial region in one image frame constituting a moving image is a pixel. FIG. 9B is a diagram illustrating a display range when a partial area in one image frame constituting a moving image is a moving image block. In the example of FIG. 9A, an area including 50% of the total number of postings of the entire one image frame F is determined as the display range R21 with the pixel P having the highest number of postings as the center. This 50% is a display ratio. For example, if the total number of postings for the entire image frame F is “30”, the total number of postings in the display range R21 is “15”. When there are a plurality of pixels with the highest number of postings, the center of gravity of these pixels is set as the center of the display range. On the other hand, in the example of FIG. 9B, the display range R22 is an area including 50% of the total number of postings of the entire image frame F, centering on the center of the moving image block B1 having the highest number of postings. When there are a plurality of moving image blocks with the highest number of postings, the center of gravity obtained from the center of these moving image blocks is set as the center of the display range. In FIG. 9B, a part of the moving image blocks B2 to B9 adjacent to the moving image block B1 is included in the display range R22. For example, if 30% of the area of the moving image block B2 is included in the display range R22, 30% of the number of postings of the moving image block B2 is included in the number of postings of the display range R22. That is, the adjacent moving image blocks B2 to B9 are divided so that the number of postings in the display range R22 is 50% of the total number of postings in the entire image frame F.

If the number of postings in the display range R21 or R22 is determined to be 50% or more of the total number of postings for the entire image frame F, pseudo camera work data that displays more highly posted scenes is generated. can do. However, the display ratio is not limited to 50% of the total number of postings of the entire image frame F. For example, the display ratio may be determined so that the number of postings in the display range R21 or R22 is 40% or more of the total number of postings in the entire image frame F. In the case where the virtual screen is a rectangular screen SC1, for example, the size of the display range is adjusted by fixing the aspect ratio to 16: 9 or the like.

By the way, when the display range is determined for each image frame F by the automatic generation process of the pseudo camera work data, the moving image may not be continuous and difficult to see. For this reason, the display range in the image frame F may be determined for each unit reproduction time, and complemented so that the interval continuously changes. FIG. 9C is a conceptual diagram showing an example in which the display ranges R31 to R33 determined every 5 seconds are complemented so as to continuously change. In the example of FIG. 9C, each of a unit playback time range of “0 to 5 seconds”, a unit playback time range of “5 to 10 seconds”, and a unit playback time range of “10 to 15 seconds”. The display range is determined in FIG. The display range R31 in the unit playback time range of “0 to 5 seconds” is determined as the display range at the playback position of 2.5 seconds in the middle of the unit playback time range. The display range R32 in the unit reproduction time range of 5 seconds to 10 seconds is determined as the display range at the reproduction position of 7.5 seconds in the middle of the unit reproduction time range. The display range R33 in the unit playback time range of “10 seconds to 15 seconds” is determined as the display range at the playback position of 12.5 seconds in the middle of this unit playback time range. The display range in the image frame F positioned between the playback position of 2.5 seconds and the playback position of 5 seconds, and the image frame positioned between the playback position of 5 seconds and the playback position of 7.5 seconds The display range in F is complemented so as to change continuously. Such complementation does not have to be linear as shown in FIG. 9C, and it is desirable to make it a smooth curve.

FIG. 10A is a flowchart showing an automatic generation process of pseudo camera work data in the control unit 11 of the distribution server 1. This flowchart considers the example shown in FIG. The automatic generation process of pseudo camera work data shown in FIG. 10A is executed, for example, every time the scene information database is updated or every predetermined number of times (for example, 10 times). Or you may comprise so that the automatic generation process of pseudo camera work data may be performed regularly. Or you may perform at arbitrary timings by judgment of an operator etc. When the automatic generation process of pseudo camera work data is started, new pseudo camera work data is generated. At this stage, the generated pseudo camera work data is empty pseudo camera work data. Next, the control unit 11 registers the display range at the reproduction position “0” as the first element in the new pseudo camera work data in association with the reproduction position “0” (step S201). The display range at the reproduction position “0” is determined, for example, for the entire image frame F at the reproduction position “0”.

Next, the control unit 11 sets “0” to the variable i (step S202). Next, the control unit 11 determines a set of image frames F at each reproduction position included in the above-described unit reproduction time range “T × i to T × (i + 1)” as a processing target (step S203). Here, “T” is the time length of one unit playback time range. Next, the control unit 11 refers to the scene information database and determines the center of the partial area having the highest number of postings as the center of the display range among the processing targets determined in step S203 (step S204).

Next, the control unit 11 refers to the scene information database and determines the display range in the image frame F so that the display ratio is, for example, 0.5 (step S205). This display ratio is calculated, for example, by dividing the total number of postings for the entire playback time of the moving image by the total number of postings to be processed determined in step S203. Next, the control unit 11 determines the reproduction position of the display range determined in step S205 as “T × i + T / 2)” (step S206). For example, in the unit playback time range of 0 to 5 seconds, the playback position “T × i + T / 2)” is determined as 2.5 seconds.

Next, the control unit 11 at each reproduction position from the last registered reproduction position to the reproduction position “T × i + T / 2) among the reproduction positions registered in the new pseudo camera work data. The display range is determined by complementary calculation (step S207). In the complementary calculation, for example, an image located between the reproduction position “0” and the reproduction position “2.5 seconds” based on the display range at the reproduction position “0” and the display range at the reproduction position “2.5 seconds”. The display range in the frame F is calculated so as to change continuously. In addition, since such a complementary calculation itself can apply a well-known method, detailed description is abbreviate | omitted.

Next, the control unit 11 registers the display range determined in step S205 and the display range determined in step S207 in new pseudo camera work data in association with the respective reproduction positions (step S208). Next, the control unit 11 increments the variable i by 1 (step S209). Next, the control unit 11 determines whether “T × i” is greater than the entire playback time of the moving image (step S210). If it is determined that “T × i” is not greater than the entire playback time of the moving image (step S210: NO), the process returns to step S203. As a result, the same processing as described above is executed in the next unit reproduction time range. If it is determined that “T × i” is greater than the entire playback time of the moving image (step S210: YES), the automatic generation process of pseudo camera work data is terminated.

Next, FIG. 11 is a conceptual diagram showing another example of automatic generation processing of pseudo camera work data. In the example of FIG. 11, when the partial area in the image frame F at a certain reproduction position is a moving image block, the state until the display range is determined is shown. The numerical value shown in FIG. 11 is the number of postings of each moving image block. In the automatic generation process of the pseudo camera work data, first, as shown in FIG. 11A, the moving image block B1 having the largest number of postings is selected. The display ratio at this time is 20% (= 6/30), assuming that the total number of postings for the entire image frame F is “30”. The selection of the moving image block is repeated until the display ratio exceeds 50%, for example. Next, as shown in FIG. 11B, the video block B3 having the largest number of postings is selected from the video blocks adjacent to the selected video block B1. The display ratio at this time is 33% (= 10/30). Next, as shown in FIG. 11C, the video block B5 having the largest number of postings after the video block B3 is selected from the video blocks adjacent to the selected video block B1. The display ratio at this time is 43% (= 13/30). Next, as shown in FIG. 11D, among the moving image blocks adjacent to the selected moving image block B1, the moving image block B8 having the largest number of postings after the moving image blocks B3 and B5 is selected. The display ratio at this time is 53% (= 16/30). When the display ratio exceeds 50% in this way, as shown in FIG. 11E, the minimum area including all the selected moving image blocks B1, B2, B4, and B8 is determined as the display range R41. The When the aspect ratio is fixed, the determined display range R41 is adjusted to a display range corresponding to the aspect ratio.

FIG. 10B is a flowchart showing another example of automatic generation processing of pseudo camera work data. This flowchart considers the example shown in FIG. The start condition of the automatic generation process of pseudo camera work data shown in FIG. 10B is the same as the start condition of the automatic generation process of pseudo camera work data shown in FIG. Note that the processing in steps S211 to S213 shown in FIG. 10B is the same as the processing in steps S201 to S203 shown in FIG.

In step S214 shown in FIG. 10B, the control unit 11 refers to the scene information database and selects a moving image block having the largest number of postings among the processing targets determined in step S213. Next, the control unit 11 refers to the scene information database, and in the image frame F in which the moving image block selected in step S214 is arranged, the control unit 11 posts most of the moving image blocks adjacent to the moving image block selected in step S214. A moving image block with a large number of times is selected (step S215). Note that the moving image block selected in step S215 is set as a non-selection target in the subsequent step S216.

Next, the control unit 11 refers to the scene information database and determines whether or not the display ratio is, for example, 0.5 or more (step S216). Here, the display ratio is the same as in the case of the process of step S205 shown in FIG. For example, when it is determined that the display ratio is not 0.5 or more (step S216: NO), the process returns to step S215. Thereby, for example, a moving image block having the next largest number of postings is selected from the moving image blocks adjacent to the moving image block selected in step S214. On the other hand, when it is determined that the display ratio is 0.5 or more (step S216: YES), the process proceeds to step S217. In step S217, the control part 11 determines the minimum area | region containing all the moving image blocks selected by step S214 and step S215 as a display range, and transfers to step S218. The processes in steps S218 to S222 shown in FIG. 10B are the same as the processes in steps S206 to S210 shown in FIG. As described above, the pseudo camera work data generated by the automatic generation process of the pseudo camera work data is transmitted to the client 2 in step S122 shown in FIG.

As described above, according to the automatic generation process of the pseudo camera work data, the pseudo camera work data indicating the display range including the scene with a high degree of attention that is posted more is automatically used by using the scene information database. And the generated pseudo camera work data can be determined as appropriate pseudo camera work data recommended to the user.

In the above embodiment, the client 2 has been shown to receive content and pseudo camera work data of the content from the distribution server 1. However, the present disclosure can also be applied to a case where the client 2 receives content and pseudo camera work data of the content from another client 2 in a hybrid type or peer type peer-to-peer network. In this case, the client 2 functions as the information processing apparatus of the present disclosure. In the above embodiment, the client 2 may be connected to the storage device 3. In this case, the client 2 reproduces the content acquired from the storage device 3. Then, the client 2 displays a moving image according to the pseudo camera work data acquired from the storage device 3.

DESCRIPTION OF SYMBOLS 1 Distribution server 2 Client 11 Storage device 21 Control part 22 Storage part S Communication system S

Claims

The display range displayed by the display means in the image frame constituting the video is a request for display range information indicating the display range of the video that can be changed according to the playback position of the video for each playback position, and Receiving means for receiving, from a terminal device, the request including first search information used for searching for an area composed of one or more pixels in the image frame of a moving image;
Storage means for storing second search information associated with each of the plurality of areas, the second search information indicating characteristics of the areas;
Based on the second search information stored in the storage unit, the storage unit is configured to associate the region with the second search information corresponding to the first search information received by the reception unit. First determining means for searching and determining from;
Second determination means for determining display range information indicating the display range including the area determined by the first determination means as display range information to be provided to the terminal device;
An information processing apparatus comprising:
The second search information associated with the area indicates a subject appearing in a display range of the moving image including the area,
The receiving means receives the request including the first search information indicating a subject;
The first determining unit determines the region including a display range in which a subject represented by the first search information received by the receiving unit appears based on the second search information associated with the region. And
The said 2nd determination means determines the said display range information which shows the said display range including the said area | region determined by the said 1st determination means as display range information provided to the said terminal device. The information processing apparatus according to 1.
The receiving means includes the request including reproduction position information indicating a reproduction position of the moving image when the first search information is input while the moving image is displayed on the terminal device, and the first search information. Receive
The first determining means includes the first search information corresponding to the first search information received by the receiving means from the region in the image frame at the reproduction position indicated by the reproduction position information received by the receiving means. Determining the region associated with the search information of 2;
The said 2nd determination means determines the said display range information which shows the said display range including the said area | region determined by the said 1st determination means as display range information provided to the said terminal device. The information processing apparatus according to 1 or 2.
An information processing method performed by a computer,
The display range displayed by the display means in the image frame constituting the video is a request for display range information indicating the display range of the video that can be changed according to the playback position of the video for each playback position, and A receiving step of receiving from the terminal device the request including first search information used for searching for an area composed of one or more pixels in the image frame of a moving image;
Based on the second search information stored in storage means for storing second search information that is associated with each of the plurality of regions, and that stores second search information indicating the characteristics of the regions, A first determination step of searching and determining the region associated with the second search information corresponding to the first search information received by the reception step;
A second determination step of determining display range information indicating the display range including the region determined in the first determination step as display range information to be provided to the terminal device;
An information processing method comprising: