JP2015198332A - Terminal and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a terminal and a program which allows even a user of insufficient skill to create a virtual camera work, that can reproduce a good-looking camera work.SOLUTION: A client terminal identifies a variation per a predetermined time of a region specified by pun and tilt, or a variation per a predetermined time of a zoom magnification specified by zoom. The client terminal corrects each virtual camera data associated, respectively with the reproduction time, determined that the variation is not less than a predetermined value, out of the recorded virtual camera data, by using conversion processing for converting to a linear or n(n is a natural number) degree curve.

Description

  The present invention relates to a technical field such as an apparatus for creating virtual camera data representing the orientation of a virtual camera and the angle of view of a virtual camera in an image frame of moving image data composed of a plurality of image frames.

  Conventionally, there is a case where pseudo camera work is performed on predetermined moving image data by a user's operation of a virtual camera. In such camera work, a specific pixel area in the image frame is displayed on the display screen by, for example, the user specifying the direction of the virtual camera or the angle of view of the virtual camera in the image frame constituting the moving image data. The Such virtual camera data representing the orientation of the virtual camera and the angle of view of the virtual camera can be uploaded from the terminal device to the distribution server. Patent Document 1 discloses a communication system that can share pseudo camera work by other users without editing uploaded video data itself.

JP, 2014-011623, A

  By the way, in order to specify a specific pixel area in a plurality of image frames of moving image data, for example, a user who does not have sufficient camerawork skills operates a virtual camera many times to set a specific pixel area in an image frame. May be specified. In this case, an object that is not displayed in the pixel area displayed on the display screen may be searched for in the image frame and displayed on the display screen. In this case, the trajectory of the change in the display area in the image frame from the start of operation of the virtual camera to the end of the operation is often not a straight line but a detour. With the virtual camera data created at this time, it is difficult to reproduce a camera work having a good appearance.

  The present invention provides a terminal device and a program capable of creating a virtual camera work that can reproduce a good-looking camera work even if the user has insufficient skills, for example.

  In order to solve the above-mentioned problem, the invention according to claim 1 is an image frame composed of a plurality of pixels, and in moving image data composed of a plurality of the image frames, a specific pixel in the image frame A first determination step for determining whether or not a virtual camera for specifying a region is operated; and when the first determination step determines that the virtual camera is operated, the virtual camera is operated Virtual camera data representing at least one of the time direction, the position when the virtual camera is operated, and the angle of view when the virtual camera is operated, and the moving image data when the virtual camera is operated A storage step of associating the reproduction time with each other and storing them in the storage means along a time series, and a pixel region specified by the virtual camera data A second determination step of determining whether the amount of change per fixed time or the amount of change of the zoom magnification specified by the virtual camera data per predetermined time is less than or equal to a predetermined value; and the virtual stored in the storage means In the camera data, a conversion process is used to convert each of the virtual camera data associated with the reproduction time determined that the change amount is not equal to or less than a predetermined value into a linear or n (n is a natural number) degree curve. A program for causing a computer to execute a correction step for correcting the image.

  According to a second aspect of the present invention, in the program according to the first aspect, the program associates the virtual camera data corrected by the correction step and the reproduction time in a time series to the server apparatus. The uploading step is further executed by the computer.

  According to a third aspect of the present invention, there is provided a virtual camera for specifying a specific pixel region in the image frame in moving image data including a plurality of the image frames, the image frame including a plurality of pixels. When the virtual camera is determined to be operated by the first determination step for determining whether the virtual camera is being operated, and when the virtual camera is operated, The change amount per predetermined time of the pixel area specified by the virtual camera data representing at least one of the position when operated and the angle of view when the virtual camera is operated, or by the virtual camera data A second determination step of determining whether the amount of change of the specified zoom magnification per predetermined time is equal to or less than a predetermined value; and the second determination When the change amount per predetermined time of the pixel area specified by the virtual camera data or the change amount per predetermined time of the zoom magnification specified by the virtual camera data is determined by the step The first storage step of storing the virtual camera data and the reproduction time when the pixel area is displayed on the display screen in association with each other in a time series, and the second determination step, When it is determined that the change amount per predetermined time of the pixel area specified by the virtual camera data or the change amount per predetermined time of the zoom magnification specified by the virtual camera data is not less than a predetermined value, the virtual camera data And a reproduction time when the pixel area is displayed on the display screen are associated with each other in time series and stored in the second storage means. And storing at least a part of the virtual camera data associated with each of the reproduction times stored in time series in the storage step and in the second storage means into a linear or n (n is a natural number) degree curve This is a program for causing a computer to execute a correction step for correcting using a conversion process.

  According to a fourth aspect of the present invention, in the program according to the third aspect, in the program, the amount of change per predetermined time in the pixel area specified by the virtual camera data is less than or equal to a predetermined value in the second determination step. If it is determined, the computer further executes a third determination step for determining whether the virtual camera data and the reproduction time are stored in association with each other in the second storage unit, and the third determination When it is determined in the step that the virtual camera data and the reproduction time are stored in the second storage unit in association with each other, the correction step includes the virtual storage stored in the second storage unit. It is characterized by correcting camera data.

  According to a fifth aspect of the present invention, in the program according to the third or fourth aspect, the program stores the virtual camera data corrected by the correction step and the first storage unit. The computer is caused to execute a registration step of registering the camera work data generated using the virtual camera data as the camera work data of the moving image data.

  According to a sixth aspect of the present invention, in the program according to the fifth aspect, the registration step stores the main camera work data in a third storage means.

  A seventh aspect of the present invention is the program according to the fifth aspect, wherein the registration step uploads the main camera work data to a server device.

  The invention according to claim 8 is a virtual camera for specifying a specific pixel region in the image frame in moving image data including a plurality of the image frames, the image frame including a plurality of pixels. When the virtual camera is determined to be operated by a first determination unit that determines whether or not the virtual camera is operated by the first determination unit, the direction when the virtual camera is operated, The virtual camera data representing at least one of the position when operated and the angle of view when the virtual camera is operated is associated with the playback time of the moving image data when the virtual camera is operated Storage means for storing in time series, a change amount per predetermined time of the pixel area specified by the virtual camera data, or the virtual camera data A second determination unit that determines whether or not a change amount of the zoom magnification specified by the predetermined time is less than or equal to a predetermined value; and the change amount is less than or equal to a predetermined value among the virtual camera data stored in the storage unit Correction means for correcting each of the virtual camera data respectively associated with the reproduction time determined to be non-linear using a conversion process for converting the virtual camera data into a linear or n (n is a natural number) degree curve. And

The invention according to claim 9 is a virtual camera for specifying a specific pixel region in the image frame in moving image data including a plurality of the image frames, the image frame including a plurality of pixels. First determination means for determining whether or not is operated;
When it is determined by the first determination means that the virtual camera is operated, the direction when the virtual camera is operated, the position when the virtual camera is operated, and the virtual camera is operated The amount of change per predetermined time of the pixel area specified by the virtual camera data representing at least one of the angles of view at the time when the zoom is performed, or the amount of change per predetermined time of the zoom magnification specified by the virtual camera data is predetermined. A second determination unit that determines whether the value is equal to or less than a value; and a change amount per predetermined time of the pixel area specified by the virtual camera data by the second determination unit, or a zoom magnification specified by the virtual camera data When it is determined that the amount of change per predetermined time is not more than a predetermined value, the virtual camera data and the pixel area are displayed on the display screen. A first storage unit that stores the reproduction time in association with each other in a time series, and a change amount per predetermined time of the pixel area specified by the virtual camera data by the second determination unit, or the virtual If it is determined that the amount of change in the zoom magnification specified by the camera data per predetermined time is not less than or equal to the predetermined value, the virtual camera data and the playback time during which the pixel area is displayed on the display screen are time-sequentially Second storage means for storing the data in association with each other, and at least a part of the virtual camera data associated with each of the reproduction times stored in time series in the second storage means in a linear or n ( n is a natural number) and correction means for correcting using a conversion process for conversion to a quadratic curve.

  According to the first, third, sixth, eighth, and ninth aspects of the present invention, it is possible to create camerawork data that can reproduce a good-looking camerawork even if the user has insufficient camerawork skills.

  According to the second and seventh aspects of the present invention, camera work data that can reproduce a good-looking camera work can be shared among a plurality of client terminals.

It is a figure which shows the example of a schematic structure of the communication system S of this embodiment. It is a figure which shows a mode that a pixel area | region changes in time series in an image frame by operation of a virtual camera. 4 is a flowchart showing camera work data recording processing in a client terminal 2. It is a conceptual diagram which shows a mode that camera work data is recorded in the recording process of camera work data. It is a conceptual diagram which shows a mode that each of five virtual camera data is converted into a straight line.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[1. Overview of communication system configuration and operation]
First, with reference to FIG. 1, the structure and operation | movement outline | summary of the communication system S of embodiment of this invention are demonstrated. FIG. 1 is a diagram illustrating a schematic configuration example of a communication system S of the present embodiment. As shown in FIG. 1, the communication system S includes a distribution server 1 and a client terminal 2. The client terminal 2 is an example of a terminal device. The number of client terminals shown in FIG. 1 is an example, and the number is not limited to this number. The distribution server 1 and the client terminal 2 are each connected to the network NW. The network NW is configured by, for example, the Internet.

  The distribution server 1 transmits content to the client terminal 2 in response to a content request from the client terminal 2. The content includes moving image data. Note that the content may include audio data. A content ID is assigned to the content. The content ID is identification information for identifying the content. The content is transmitted by streaming distribution via the network NW, for example. The client terminal 2 receives the content distributed by streaming from the distribution server 1. The client terminal 2 reproduces the moving image data included in the received content. The moving image data is composed of a plurality of image frames. Each image frame is composed of a plurality of pixels.

  Then, the client terminal 2 displays on the display screen a pixel area specified from a plurality of pixels in the image frame of the reproduced moving image data. This pixel area corresponds to a drawing area drawn on the display screen within one image frame. In other words, the pixel area is a display range cut out from the image frame. This pixel region can be specified by, for example, an operation of pseudo camera work (hereinafter referred to as “camera work”). The camera work operation is performed by operating the virtual camera in the image frame. That is, the client terminal 2 displays the pixel area specified from the plurality of pixels on the display screen according to the operation of the virtual camera in the image frame along with the output of the moving image data. Here, the virtual camera refers to a viewpoint virtually set for a moving image projected on a virtual screen in a two-dimensional plane or a three-dimensional virtual space. An example of a moving image is a panoramic moving image. A panoramic video is a video in which a subject is shot by a camera equipped with a lens capable of shooting a wide range, for example, with a high-resolution camera. Examples of lenses capable of photographing a wide range include a wide lens, a fisheye lens, and a 360 degree lens. A panoramic video may be created by combining a plurality of camera videos. The virtual screen includes a flat screen, a cylindrical screen, and a spherical screen. For example, when the lens of a camera that has captured a moving image is a fisheye lens or a 360-degree lens, the virtual screen is a spherical screen.

  The virtual camera operation includes, for example, a virtual camera pan (left / right swing) operation, a virtual camera tilt (up / down swing) operation, and a virtual camera zoom (enlargement or reduction) operation. The direction (angle) of the virtual camera is determined by the pan and tilt operations of the virtual camera. The panning and tilting operations of the virtual camera can be realized by, for example, a mouse dragging operation by the user or a flicking operation by the user with a finger or a pen. The flick operation is, for example, an operation in which the user quickly slides a contact object such as a finger or a pen on a touch panel display screen. Further, the angle of view of the virtual camera is determined by the zoom operation of the virtual camera. The zoom operation of the virtual camera can be realized, for example, when the user presses a button provided on an operation unit such as a mouse or a button displayed on a touch panel display screen. Alternatively, the zoom operation of the virtual camera can be realized by, for example, a pinch operation for pinching a touch-panel display screen with two user fingers. The pinch operation includes pinch-in that reduces the pixel area and pinch-out that enlarges the pixel area.

  The client terminal 2 records camera work data during the reproduction of moving image data. The camera work data includes virtual camera data representing at least one of the position of the virtual camera displaying the pixel area on the display screen, the orientation of the virtual camera, and the angle of view of the virtual camera, and the pixel area displayed on the display screen. Are included in association with each other along the time series. The pixel area is specified by the position of the virtual camera or the direction of the virtual camera. On the other hand, the zoom magnification (display magnification) of the pixel area is specified by the angle of view of the virtual camera. The reproduction time indicates the reproduction time corresponding to the reproduction position from the reproduction start of the moving image. In other words, the reproduction time is a reproduction time during which the pixel area specified by the virtual camera data is output. The camera work data of the present embodiment includes, for example, parameters of pan, tilt, and zoom as virtual camera data. The pan and tilt parameters are an example of data representing the position of the virtual camera or the direction of the virtual camera. The zoom parameter is an example of data representing the zoom magnification (display magnification). When the virtual screen is a flat screen, the position of the virtual camera in the image frame is defined by data representing (x, y) coordinates. In this case, the (x, y) coordinates indicate the center coordinates of the virtual camera, for example.

  Here, the relationship between pan and tilt and the pixel region is expressed by, for example, the following equations (1) and (2).

pan = x '+ width / 2 (1)
tilt = y '+ height / 2 (2)

  Here, (x ′, y ′) indicates, for example, the upper left corner coordinates of the pixel area when the upper left corner coordinates of the image frame are set to the origin (0, 0). width indicates the width of the pixel region. height indicates the height of the pixel region. Moreover, width and height are respectively represented by the following formulas (3) and (4), for example.

width = W / α (3)
height = H / α (4)

  Here, W indicates the width of the image frame. H indicates the height of the image frame. α is a value that varies depending on zoom. For example, if the zoom magnification is 1, α = 1/4. If the zoom magnification is 2, α = ½. The relationship between α and the zoom magnification is defined on a program, for example.

  From the above formulas (1) to (4), the pixel region in the image frame can be specified from the pan, the tilt, and the zoom.

  According to the camera work data, it is possible to define the movement of a virtual camera that virtually captures all or part of a moving image projected onto a virtual screen in a two-dimensional plane or a three-dimensional virtual space. Thereby, camerawork can be reproduced. For example, a plurality of camera work data is recorded every predetermined time in the reproduction time from the start of reproduction of the moving image data to the end of reproduction. For example, the client terminal 2 associates the virtual camera data when the virtual camera is operated with the reproduction time of the moving image data when the virtual camera is operated, and stores them in a storage unit such as a RAM along the time series. To do. Here, the recording of the camera work data includes a temporary recording and a main recording. Temporary recording refers to temporarily storing data in a predetermined temporary storage area of the RAM or the storage unit 22, for example. The temporary storage area is an example of a first storage unit. The main recording refers to storing data in a main storage area different from the temporary storage area of the RAM or the storage unit 22, for example. This storage area is an example of the second storage means. The first storage unit and the second storage unit may be separate storage devices. The recorded data is stored, for example, in a file.

  The client terminal 2 specifies the amount of change per predetermined time of the pixel area specified by panning and tilting, or the amount of change per predetermined time of the zoom magnification specified by zooming. Here, the amount of change per predetermined time of the pixel region is, for example, the amount of change per predetermined time between the center coordinates of the pixel region. Note that it may be a change amount per predetermined time between coordinates of a predetermined pixel in the pixel region. The client terminal 2 determines whether the amount of change per predetermined time is equal to or less than a predetermined value. Then, the client terminal 2 performs linear or n (n is a natural number) for each of the virtual camera data respectively associated with the reproduction time determined that the change amount is not less than or equal to the predetermined value in the recorded virtual camera data. ) Correction is performed using a conversion process for converting to a quadratic curve. Note that, for example, a case where the zoom is operated a plurality of times per predetermined time and the final zoom magnification is determined or there is a sudden change in the zoom magnification is assumed. In such a case, since a predetermined pixel area is suddenly enlarged or reduced, there is a possibility that smooth camera work will not be achieved. For this reason, in this embodiment, the zoom magnification associated with the reproduction time determined that the change amount of the zoom magnification per predetermined time is not less than or equal to the predetermined value is also corrected. Here, a playback section including a playback time for which it is determined that the amount of change is not less than a predetermined value is referred to as a “specific playback section”.

  FIG. 2 is a diagram illustrating a state in which the pixel region in the image frame changes in time series by the operation of the virtual camera. In the example of FIG. 2, the drawing area displayed in the image frame changes from the display range 1 to the display range 4. For example, no object appears in the first display range 1 displayed. Therefore, the user operates the virtual camera and searches for the object until it appears. As a result, the display range moves in the direction of the arrow as in the display range before correction shown in FIG. The movement of the position of the display range in the image frame is not a straight line but a detour. The camera work data recorded during the movement of the display range is a detoured movement. However, as described above, according to the corrected camera work data, it is possible to move linearly like the movement of the corrected display range shown in FIG. In FIG. 2, when the user operates the virtual camera by, for example, a drag operation, the mouse pointer is moved in the direction opposite to the arrow direction shown in FIG.

[2. Configuration of each device]
Next, the configuration of each device included in the communication system S of the present embodiment will be described with reference to FIG. As shown in FIG. 1, the distribution server 1 includes a control unit 11, a storage unit 12, an interface unit 13, and the like. These components are connected to the bus 14. The interface unit 13 is connected to the network NW. The control unit 11 includes a CPU (Center Processing Unit) as a computer, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The control unit 11 controls transmission / reception of contents and transmission / reception of camera work data. The storage unit 12 is configured by, for example, a hard disk drive. The storage unit 12 stores an OS, a server program, and the like. For example, the content received from the photographing apparatus is stored in the storage unit 12 in association with the content ID. The storage unit 12 stores a plurality of camera work data received from the client terminal 2 in association with the content ID.

  Next, as shown in FIG. 1, the client terminal 2 includes a control unit 21, a storage unit 22, a video RAM 23, a video control unit 24, an operation processing unit 25, an audio control unit 26, an interface unit 27, and the like. Is done. These components are connected to the bus 28. A display unit 24 a including a display is connected to the video control unit 24. The control unit 21 includes a CPU (Center Processing Unit) as a computer, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The control unit 21 is an example of a first determination unit, a second determination unit, and a correction unit of the present invention. It is an example. An operation unit 25 a is connected to the operation processing unit 25. Examples of the operation unit 25a include a mouse, a keyboard, and a remote controller. A touch panel serving both as the display unit 24a and the operation unit 25a may be applied. The control unit 21 receives an operation instruction from the operation unit 25 a by the user via the operation processing unit 25. The user can operate the virtual camera described above using the operation unit 25a. A speaker 26 a is connected to the audio control unit 26. The interface unit 27 is connected to the network NW. The storage unit 22 is configured by, for example, a hard disk drive (HDD). The RAM or the storage unit 12 is an example of a third storage unit of the present invention. The third storage unit may be a different storage device different from the first storage unit and the second storage unit. The storage unit 22 stores an OS, player software, and the like. The player software is a program that causes the CPU to execute content reception and playback processing, camera work data recording processing, and camera work data transmission / reception processing.

  The control unit 21 functions as a player that reproduces the received content by executing the player software. Moreover, a display screen is displayed on the display in the display unit 24a by the function of the player. The display screen is also called a window. A moving image is displayed on the display screen according to the operation of the virtual camera. This moving image is a moving image corresponding to a pixel region specified from a plurality of pixels in the image frame of the received moving image data. Further, the RAM in the control unit 21 is provided with a buffer memory. The buffer memory temporarily holds moving image data included in the received content. The control unit 21 outputs moving image data from the buffer memory to the video RAM 23. The video RAM 23 is provided with a frame buffer into which drawing data is written. The video control unit 24 displays the drawing data written in the frame buffer by drawing on the display screen according to the control signal from the control unit 21. In some cases, audio data is included in the content held in the buffer memory. In this case, the control unit 21 reproduces the audio data from the buffer memory and outputs it to the audio control unit 26. The voice control unit 26 generates an analog voice signal from the voice data, and outputs the generated analog voice signal to the speaker 26a.

[3. Operation of communication system S]
Next, the operation of the communication system S of the present embodiment will be described with reference to FIGS. FIG. 3 is a flowchart showing the camera work data recording process in the client terminal 2. The camera work data recording process is executed by the control unit 21 or the CPU of the client terminal 2. FIG. 4 is a conceptual diagram showing how camera work data is recorded in the camera work data recording process. For example, the control unit 21 starts the process illustrated in FIG. 3 when the user gives an activation instruction. When the process shown in FIG. 3 is started, the control unit 21 activates the player (step S1). When the player is activated, for example, reception and playback processing of content selected by the user is started. The content reception and playback processing is performed in parallel with the processing shown in FIG. 3 by the multitask function of the OS, for example. In the content reception and reproduction process, the control unit 21 accesses the distribution server 1 and transmits a request for content. Then, in response to the content request, the control unit 21 receives the content distributed by streaming from the distribution server 1 and stores it in the buffer memory. And the control part 21 reproduces | regenerates the moving image data contained in the content memorize | stored in the buffer memory. The control unit 21 displays a moving image on the display screen based on the reproduced moving image data.

  In step S2 shown in FIG. 3, the control unit 21 sets the camerawork determination flag to OFF. The in-camera determination flag is a flag indicating whether or not the user is operating the virtual camera. If the camera work determination flag is OFF, it indicates that the virtual camera is not being operated. Next, the control unit 21 waits for a preset first time (step S3). The first time is a time for determining the recording interval of the camera work data. The first time is set to 0.5 seconds, for example. Next, the control unit 21 determines whether or not there is an end instruction from the player (step S4). If the control unit 21 determines that there is no instruction to end the player (step S4: NO), the control unit 21 proceeds to step S5.

  In step S5, the control unit 21 determines whether or not the camerawork determination flag is ON. If the controller 21 determines that the camerawork determination flag is not ON (step S5: NO), the controller 21 proceeds to step S6. In step S6, the control unit 21 determines whether or not camera work data is temporarily recorded. When it is determined that the camera work data is not temporarily recorded (step S6: NO), the control unit 21 proceeds to step S7.

  In step S7, the control unit 21 determines whether or not the virtual camera is being operated. That is, the control unit 21 determines whether or not a virtual camera for specifying a specific pixel area in the image frame is operated in the moving image data being reproduced. If it is determined that the virtual camera is not being operated (step S7: NO), the control unit 21 proceeds to step S8. In step S8, the control unit 21 performs main recording of camera work data. That is, the control unit 21 performs main recording of camera work data in which virtual camera data that identifies the pixel area currently displayed on the display screen is associated with a reproduction time during which the pixel area is displayed, and step S3 Return to. Note that the main recording in the present embodiment means that the camera work data is finally determined and stored in the storage unit 22. Thus, as shown in FIG. 4, the camera work data (pan1, tilt1, zoom1, time1) and the like are sequentially recorded until the operation of the virtual camera is started. And if operation of a virtual camera is started by the user before determination of step S7, the control part 21 will determine with operation of a virtual camera being in step S7 (step S7: YES). In this case, the control unit 21 sets the camerawork in-progress determination flag to ON (step S9), and proceeds to step S10.

  In step S10, the control unit 21 temporarily records camera work data. That is, the control unit 21 temporarily records the camera work data in which the virtual camera data specifying the pixel area displayed on the display screen according to the operation of the virtual camera and the reproduction time in which the pixel area is displayed are associated with each other, Return to step S3. Thus, camera work data (pan4, tilt4, zoom4, time4) are temporarily recorded as shown in FIG. Here, every time camerawork data is temporarily recorded, the control unit 10 adds 1 to a count value indicating the number of camerawork data. This count value is stored in a predetermined area of the RAM. If the control unit 21 determines in step S5 that the in-camera determination flag is ON (step S5: YES), the control unit 21 proceeds to step S11.

  In step S11, the control unit 21 determines whether or not the virtual camera is being operated. If it is determined that the virtual camera is being operated (step S11: YES), the control unit 21 proceeds to step S12. In step S12, the control unit 21 specifies the amount of change per predetermined time. Here, the predetermined time is the same as the first time, for example. As described above, the change amount per predetermined time is, for example, the change amount per predetermined time of the center coordinates of the pixel region specified by panning and tilting, or the change per predetermined time of the zoom magnification specified by zooming. Amount. The pixel area is specified by the control unit 21 using, for example, the above-described equations (1) to (4). Thereby, for example, the four corner coordinates of the pixel region are specified. The control unit 11 calculates center coordinates from the four corner coordinates of the specified pixel region.

  Next, the control unit 21 determines whether or not the amount of change specified in step S12 is equal to or less than a threshold value (predetermined value) (step S13). Note that the control unit 21 may determine whether the amount of change is less than a threshold value. If the control unit 21 determines that the amount of change per predetermined time is not less than or equal to the threshold (step S13: NO), the control unit 21 proceeds to step S10. In step S10, the control unit 21 temporarily records the camera work data as described above, and returns to step S3. That is, the control unit 21 temporarily records the virtual camera data in which the pixel area is displayed on the display screen and the reproduction time in which the pixel area is displayed in association with each other in time series. While the change amount is larger than the threshold value by operating the virtual camera, the temporary recording of the camera work data is repeated. During this time, it is considered that the user is operating the virtual camera to search for an object outside the display screen, for example.

  On the other hand, if the control unit 21 determines that the amount of change per predetermined time is equal to or less than the threshold (step S13: YES), the control unit 21 proceeds to step S14. In this case, for example, it is assumed that the amount of change per predetermined time has become equal to or less than the threshold as a result of loosening the operation of the virtual camera by operating the virtual camera and finding an object that was searched for with a change amount greater than the threshold. Is done. In step S14, the control unit 21 refers to the count value indicating the number of camera work data, and determines whether or not there is a predetermined number of temporary camera work data recordings. That is, the control unit 21 determines whether the virtual camera data and the reproduction time are stored in the temporary storage area in association with each other. Here, the predetermined number is set to three or more, which is a number necessary for correcting the camera work data. When it is determined that the amount of change per predetermined time is equal to or less than the threshold (step S13: YES), the camerawork data is recorded in step S8. That is, in this case, the virtual camera data in which the pixel area is displayed on the display screen and the reproduction time in which the pixel area is displayed are associated and recorded in time series. When it is determined that there is a predetermined number of temporary recordings of camera work data (step S14: YES), the control unit 21 proceeds to step S15.

  In step S15, the control unit 21 corrects each of the plurality of temporarily recorded camera work data using a conversion process that converts the data into linear or n (n is a natural number) degree curve. That is, the control unit 21 corrects each of the virtual camera data associated with each playback time included in the specific playback section described above using a conversion process that converts the virtual camera data into a linear or n (n is a natural number) degree curve. To do. Note that the playback time included in the playback time included in the specific playback section described above may be corrected. In the example of FIG. 4, five pieces from (pan4, tilt4, zoom4, time4) to (pan8, tilt8, zoom8, time8) are temporarily recorded camera work data. Here, (pan4, tilt4, zoom4, time4) is camera work data at the start point of a specific playback section. Further, (pan8, tilt8, zoom8, time8) is camera work data at the end point of a specific playback section.

  FIG. 5 is a conceptual diagram showing how each of the five virtual camera data is converted into a straight line. In the example of FIG. 5, the five virtual camera data (pan, tilt, zoom) described above are used for correction calculation as three-dimensional coordinates (x, y, z). Here, the coordinates of P1 are (x1, y1, z1), the coordinates of P2 are (x2, y2, z2), the coordinates of P3 are (x3, y3, z3), and the coordinates of P4 are (x4, y4, z4) The coordinates of P5 are (x5, y5, z5). In this case, the straight line shown in FIG. 5A can be obtained by the following equation (1).

  (x−x1) / (x5−x1) = (y−y1) / (y5−y1) = (z−z1) / (z5−z1) (1)

  Here, x, y, and z are variables. When the above formula (1) is modified, y = {(y5 + y1) / (x5-x1)} x + (− x1y5 + x5y1) / (x5−x1), and y is a linear function of x. The same applies to x and z. In this example, the straight line is an example of a linear shape.

  In this case, the control unit 21 defines and defines a straight line connecting the coordinates (x1, y1, z1) of P1 and the coordinates (x5, y5, z5) of P5, for example, by the calculation formula (1) above. The corrected P2a coordinates (x2a, y2a, z2a), P3a coordinates (x3a, y3a, z3a), and P4a coordinates (x4a, y4a, z4a) are allocated on the straight line. In other words, the control unit 21 corrects the three virtual camera data between the virtual camera data at the start point of the specific playback section and the virtual camera data at the end point of the specific playback section. In this case, P2a, P3a, and P4a are complementary points between P1 and P5. The virtual camera data at the start point of the specific playback section and the virtual camera data at the end point of the specific playback section may be corrected. In this case, the reproduction time associated with each of the five virtual camera data is not corrected. In this case, as shown in FIG. 5B, the amount of change per predetermined time from P1 to P5 through P1a, P2a, and P3a is constant.

  However, as shown in FIG. 5C, the control unit 21 may correct the change amount per predetermined time from P1 to P5 through P1a, P2a, and P3a. In this case, for example, the control unit 21 calculates a time difference TD between a reproduction time T1 corresponding to P1 included in a specific reproduction section and a reproduction time T2 corresponding to P2. Next, the control unit 21 multiplies the calculated time difference TD by a predetermined coefficient k1 to calculate a correction time difference TD2a. This coefficient k1 is set to 1.8, for example. Next, the control unit 21 calculates a new reproduction time T2a by adding the calculated correction time difference TD2a to the reproduction time T1. Next, the control unit 21 associates the new playback time T2a with the virtual camera data corresponding to P2a. Next, the control unit 21 multiplies the calculated time difference TD by a predetermined coefficient k2 to calculate a correction time difference TD3a. This coefficient k2 is set to 1.0, for example. Next, the control unit 21 calculates a new reproduction time T3a by adding the calculated correction time difference TD3a to the new reproduction time T2a. Next, the control unit 21 associates the new playback time T3a with the virtual camera data corresponding to P3a. Next, the control unit 21 multiplies the calculated time difference TD by a predetermined coefficient k3 to calculate a correction time difference TD4a. This coefficient k3 is set to 0.8, for example. Next, the control unit 21 calculates a new reproduction time T4a by adding the calculated correction time difference TD4a to the new reproduction time T3a. Next, the control unit 21 associates the new playback time T4a with the virtual camera data corresponding to P4a. As a result, as shown in FIG. 5C, the amount of change per predetermined time for displacement from P1 to P5 via P1a, P2a, and P3a is initially large and can be corrected so as to gradually decrease.

  In the example shown in FIG. 5, an example is shown in which the coordinates of the three corrected P2, P3, and P4 are corrected to the same number of complementary points (P2a, P3a, and P4a). However, the coordinates of the three corrected P2, P3, and P4 need not be corrected to the same number of complementary points. For example, the coordinates of the three corrected P2, P3, and P4 may be corrected to more complementary points than this number. In the above example, the virtual camera data is corrected using a linear function that makes the change in the position between the central coordinates in the specific pixel region a straight line. As another example, the control unit 21 may correct each of a plurality of temporarily recorded virtual cameras using a conversion process that converts the virtual camera into an n-order curve. An example of an nth order curve is a Bezier curve. In this case, for example, the control unit 21 defines a Bezier curve from the coordinates of P1, P3, and P5 as shown in FIG. 5D, and the corrected P2a coordinates (x2a, y2a) are defined on the defined Bezier curve. , z2a), P3a coordinates (x3a, y3a, z3a), and P4a coordinates (x4a, y4a, z4a).

  Next, the control unit 21 performs main recording of the camera work data corrected in step S15 or the temporarily recorded camera work data (step S16). When the process goes through step S15, the control unit 21 generates main camera work data using the camera work data corrected in step S15 and the camera work data recorded in step S8. Next, the control unit 21 resets the temporarily recorded camera work data and the OFF time (step S17). That is, the temporarily recorded camera work data is deleted from the temporary recording area. Further, the OFF time recorded in step S19 described later is reset to zero. Then, after resetting the camera work data and the OFF time, the control unit 21 proceeds to step S8. In this case, in step S8, the camera work data is fully recorded, and the process returns to step S3. Thereafter, when the operation of the virtual camera is stopped, the control unit 21 determines in step S11 that the virtual camera is not being operated (step S11: NO), and proceeds to step S18. In step S18, the control unit 21 resets the temporarily recorded camera work data and the OFF time. Next, the time reset in step S18 is recorded as the OFF time (step S19), and the process proceeds to step S10. In this case, in step S10, camera work data is temporarily recorded, and the process returns to step S3. And the control part 21 shall transfer to step S5 through step S3 and step S4. In this case, the control unit 21 determines in step S5 that the camerawork in-progress determination flag is not ON (step S5: NO), and proceeds to step S6.

  By the way, for example, after the user starts operating the virtual camera, the virtual camera may be operated with a change amount equal to or less than a threshold value. In this case, the control unit 21 proceeds to step S14 without determining in step S13 that the amount of change per predetermined time is not less than or equal to the threshold value. Therefore, in this case, only one camera work data is temporarily recorded. However, in the example of FIG. 3, since the process of step S10 is performed after the process of step S9, one camera work data is always temporarily recorded. If the control unit 21 determines that the predetermined number of camera work data is not temporarily recorded (step S14: NO), the control unit 21 proceeds to step S16. In step S16 in this case, the control unit 12 performs the main recording of the temporarily recorded camera work data. That is, in this case, as shown in FIG. 4, the temporarily recorded camera work data (pan11, tilt11, zoom11, time11) is not corrected. Next, the control unit 21 resets the temporarily recorded camera work data and the OFF time (step S17), and proceeds to step S8. In addition, the control part 21 may transfer to step S3 after the process of step S9. In this case, camera work data may not be temporarily recorded at the time of determination in step S14. In this case, the control part 11 will transfer to step S8 from step S14, for example.

  In addition, after the camera work determination flag is set to ON in step S9, the operation of the virtual camera may be stopped before the process of step S11. In other words, this is a case where the user has operated the virtual camera for a moment, for example. In this case, in step S11, the control unit 21 determines that the virtual camera is not being operated (step S11: NO), and proceeds to step S18. In step S18, the control unit 21 resets the temporarily recorded camera work data and the OFF time. Next, the time reset in step S18 is recorded as the OFF time (step S19), and the process proceeds to step S10. Then, it is assumed that the control unit 21 proceeds to step S5 through step S10, step S3, and step S4. In this case, the control unit 21 determines in step S5 that the camerawork in-progress determination flag is not ON (step S5: NO), and proceeds to step S6.

  If the control unit 21 determines in step S6 that there is temporary recording of camera work data (step S6: YES), the control unit 21 proceeds to step S20. In step S20, the control unit 21 determines whether or not the second time has elapsed from the OFF time of the camera work work determination flag. For example, the second time is set to be longer than the time until three or more camera work data are temporarily recorded. When it is determined that the second time has not elapsed since the OFF time of the camera work work determination flag (step S20: NO), the control unit 21 proceeds to step S7. On the other hand, when it is determined that the second time has elapsed from the OFF time of the camera work work determination flag (step S20: YES), the control unit 21 proceeds to step S21. In addition, the process of step S21-S23 is the same as the process of step S15-S17.

  Then, when the user gives an instruction to end the player, the control unit 21 determines in step S4 that there is an instruction to end the player (step S4: YES), and proceeds to step S24. In step S24, as in step S6, the control unit 21 determines whether or not camera work data is temporarily recorded. When it is determined that the camera work data is not temporarily recorded (step S24: NO), the control unit 21 proceeds to step S29. On the other hand, when it is determined that there is temporary recording of camera work data (step S24: YES), the control unit 21 proceeds to step S25. Note that the processing in steps S25 to S28 is the same as the processing in steps S14 to S17. In this case, the control unit 21 proceeds to step S29 after performing the process of step S28.

  In step S29, the control unit 21 registers the main camera work data stored in the main storage area as the camera work data of the reproduced moving image data. For example, the control unit 21 stores the camera work data and the content ID of the reproduced moving image data in the storage unit 22 in association with each other. In addition, the control unit 21 uploads the camera work data stored in the storage area and the content ID of the reproduced moving image data to the distribution server 1. When the camera work data is corrected in the camera work data recording process, the corrected camera work data is uploaded to the distribution server 1. That is, in this case, the control unit 21 uploads the corrected virtual camera data and the reproduction time to the distribution server 1 in association with each other in time series. The distribution server 1 stores the camera work data uploaded from the client terminal 2 in the storage unit 12 in association with the content ID. A URL (Uniform Resource Locator) indicating the location of the camera work data stored in this way is disclosed on a predetermined website, for example. Thereby, the uploaded camera work data can be shared among a plurality of client terminals. In addition, the control part 21 complete | finishes the process shown in FIG. 3, after performing the process of step S29.

  As described above, according to the above-described embodiment, the client terminal 2 changes the amount of change in the pixel area specified by panning and tilting per predetermined time or the change in the zoom magnification specified by zooming per predetermined time. Specify the amount. Then, the client terminal 2 performs linear or n (n is a natural number) for each of the virtual camera data respectively associated with the reproduction time determined that the change amount is not less than or equal to the predetermined value in the recorded virtual camera data. ) Correction is performed using a conversion process for converting to a quadratic curve. Therefore, even if the user does not have sufficient camerawork skills, the client terminal 2 can create camerawork data that can reproduce a smooth camerawork that looks good.

DESCRIPTION OF SYMBOLS 1 Distribution server 2 Client terminal S Communication system

Claims (9)

First, it is determined whether or not a virtual camera for specifying a specific pixel region in the image frame is operated in moving image data including a plurality of the image frames. 1 determination step;
When it is determined by the first determination step that the virtual camera is operated, the direction when the virtual camera is operated, the position when the virtual camera is operated, and the virtual camera is operated A storage step of associating the virtual camera data representing at least one of the angles of view at the time when the virtual camera is operated with the reproduction time of the moving image data on which the virtual camera is operated, and storing them in a storage unit in time series;
A second determination step of determining whether the amount of change per predetermined time of the pixel area specified by the virtual camera data or the amount of change per predetermined time of the zoom magnification specified by the virtual camera data is equal to or less than a predetermined value. When,
Of the virtual camera data stored in the storage means, each of the virtual camera data associated with a reproduction time determined that the change amount is not less than a predetermined value is linear or n (n is a natural number) A correction step for correcting using a conversion process for converting to a quadratic curve;
A program that causes a computer to execute. The program is
2. The program according to claim 1, further causing the computer to further execute a step of associating the virtual camera data corrected by the correction step and the reproduction time in a time series and uploading them to a server device. . First, it is determined whether or not a virtual camera for specifying a specific pixel region in the image frame is operated in moving image data including a plurality of the image frames. 1 determination step;
When it is determined by the first determination step that the virtual camera is operated, the direction when the virtual camera is operated, the position when the virtual camera is operated, and the virtual camera is operated The amount of change per predetermined time of the pixel area specified by the virtual camera data representing at least one of the angles of view at the time when the zoom is performed, or the amount of change per predetermined time of the zoom magnification specified by the virtual camera data is predetermined. A second determination step of determining whether the value is equal to or less than a value;
The amount of change per predetermined time of the pixel area specified by the virtual camera data or the amount of change per predetermined time of the zoom magnification specified by the virtual camera data is less than or equal to a predetermined value by the second determination step. A first storage step of storing the virtual camera data and the reproduction time when the pixel area is displayed on the display screen in association with each other in a time series when the determination is made;
In the second determination step, it is determined that the change amount per predetermined time of the pixel area specified by the virtual camera data or the change amount per predetermined time of the zoom magnification specified by the virtual camera data is not less than a predetermined value. A second storage step of storing the virtual camera data and the reproduction time in which the pixel area is displayed on the display screen in association with each other in time series in the second storage unit;
A conversion process for converting at least a part of the virtual camera data associated with each of the reproduction times stored in time series in the second storage unit into a linear or n (n is a natural number) degree curve is used. Correction steps to correct
A program that causes a computer to execute. The program is
When it is determined by the second determination step that the amount of change per predetermined time of the pixel area specified by the virtual camera data is equal to or less than a predetermined value, the virtual memory data and the reproduction time are stored in the second storage unit. And further causing the computer to execute a third determination step for determining whether or not is stored in association with each other,
When it is determined by the third determination step that the virtual camera data and the reproduction time are stored in the second storage unit in association with each other, the correction step is stored in the second storage unit. The program according to claim 3, wherein the virtual camera data is corrected. The program is
The camera work data generated by using the virtual camera data corrected by the correction step and the virtual camera data stored in the first storage unit is registered as camera work data of the moving image data. The program according to claim 3 or 4, wherein the registration step is executed by the computer.   6. The program according to claim 5, wherein the registration step stores the main camera work data in a third storage unit.   6. The program according to claim 5, wherein the registration step uploads the camera work data to a server device. First, it is determined whether or not a virtual camera for specifying a specific pixel region in the image frame is operated in moving image data including a plurality of the image frames. 1 determination means;
When it is determined by the first determination means that the virtual camera is operated, the direction when the virtual camera is operated, the position when the virtual camera is operated, and the virtual camera is operated Storage means for storing virtual camera data representing at least one of the angle of view at the time when the virtual camera is operated and a reproduction time of the moving image data in which the virtual camera is operated in association with each other in time series;
Second determination means for determining whether a change amount per predetermined time of the pixel area specified by the virtual camera data or a change amount per predetermined time of the zoom magnification specified by the virtual camera data is equal to or less than a predetermined value. When,
Of the virtual camera data stored in the storage means, each of the virtual camera data associated with a reproduction time determined that the change amount is not less than a predetermined value is linear or n (n is a natural number) Correction means for correcting using a conversion process for converting to a quadratic curve;
A terminal device comprising: First, it is determined whether or not a virtual camera for specifying a specific pixel region in the image frame is operated in moving image data including a plurality of the image frames. 1 determination means;
When it is determined by the first determination means that the virtual camera is operated, the direction when the virtual camera is operated, the position when the virtual camera is operated, and the virtual camera is operated The amount of change per predetermined time of the pixel area specified by the virtual camera data representing at least one of the angles of view at the time when the zoom is performed, or the amount of change per predetermined time of the zoom magnification specified by the virtual camera data is predetermined. Second determination means for determining whether the value is equal to or less than a value;
The amount of change per predetermined time of the pixel area specified by the virtual camera data or the amount of change per predetermined time of the zoom magnification specified by the virtual camera data by the second determination unit is less than or equal to a predetermined value. A first storage unit that stores the virtual camera data and the reproduction time in which the pixel area is displayed on the display screen in association with each other in a time series when the determination is made;
The second determination means determines that the change amount per predetermined time of the pixel area specified by the virtual camera data or the change amount per predetermined time of the zoom magnification specified by the virtual camera data is not less than a predetermined value. A second storage means for storing the virtual camera data and the reproduction time when the pixel area is displayed on the display screen in association with each other in time series;
A conversion process for converting at least a part of the virtual camera data associated with each of the reproduction times stored in time series in the second storage unit into a linear or n (n is a natural number) degree curve is used. Correction means for correcting
A terminal device comprising: