JP5550248B2 - Program, information storage medium, image processing apparatus, image processing method, and data structure - Google Patents

Description

  The present invention relates to a program, an information storage medium, an image processing apparatus, an image processing method, and a data structure.

  While playing the game, the game execution status such as operation data (operation log) indicating the timing when the user operated the controller, operated buttons, or position data indicating the position of an object such as a character in each frame is displayed. There is a game device that generates replay data for reproduction. Patent Document 1 discloses a game device capable of replaying a game. Such game devices include one that generates a replay moving image of a game based on the replay data and displays it on a display or the like.

US Pat. No. 6,724,385

  When a replay moving image of a game is displayed on a display or the like, and a moving image representing a state in which the user is playing the game is also displayed, the viewer of the replay moving image can play the game along with the replay of the game. You can also enjoy watching the game and being enthusiastic.

  In general image processing devices as well as game devices, when replay data and a moving image generated by an imaging means such as a camera are linked in a period corresponding to the replay data, the generated moving image is interesting. Is expected to improve.

  The present invention has been made in view of the above problems, and is a program, an information storage medium, and image processing capable of linking replay data and a moving image generated by an imaging unit in a period corresponding to the replay data. An object is to provide an apparatus, an image processing method, and a data structure.

  In order to solve the above problems, a program according to the present invention includes replay data acquisition means for acquiring replay data for reproducing the execution status of the program, and a reproduction period in which the execution status of the program is reproduced by the replay data. A computer as an imaging moving image acquisition means for acquiring an imaging moving image generated by an imaging means during an imaging period including at least a part, a means for causing the holding means to hold the replay data and the imaging moving image in association with each other It is characterized by functioning.

  The image processing apparatus according to the present invention includes replay data acquisition means for acquiring replay data for reproducing the execution status of a program, and at least a part of a reproduction period in which the execution status of the program is reproduced by the replay data. An imaging moving image acquisition unit that acquires an imaging moving image generated by the imaging unit in an imaging period including the image data, and a holding unit that holds the replay data and the imaging moving image in association with each other. And

  The image processing method according to the present invention includes a replay data acquisition step for acquiring replay data for reproducing a program execution status, and at least a part of a reproduction period during which the program execution status is reproduced by the replay data. Including an imaging moving image acquisition step of acquiring an imaging moving image generated by the imaging means in an imaging period including the step of causing the holding means to hold the replay data and the imaging moving image in association with each other. It is characterized by.

  Further, the data structure according to the present invention includes replay data for reproducing the execution status of the program, and a captured moving image generated by the imaging means in at least a part of a period corresponding to the replay data, The replay data and the captured moving image are associated with each other.

  According to the present invention, since the replay data and the captured moving image are held in association with each other, the replay data and the captured moving image generated by the imaging unit during the period corresponding to the replay data can be linked.

  In the program according to an aspect of the present invention, the computer is further caused to function as a reproduction moving image generation unit that generates a reproduction moving image based on the replay data and the captured moving image. To do. In this way, since a playback moving image is generated based on the replay data and the captured moving image, a moving image in which the replay data and the captured moving image generated by the imaging unit in the period corresponding to the replay data are linked. An image can be generated.

  In the program according to this aspect, the reproduction moving image generation unit may generate the reproduction moving image by reproducing a moving image based on the replay data and the captured moving image. In this way, the reproduced moving image can be used for generating a moving image for reproduction.

  Further, in the program according to one aspect of the present invention, the reproduction moving image generation unit generates the reproduction moving image based on the replay moving image generated based on the replay data and the captured moving image. Is generated. In this way, a moving image based on the replay data can be used to generate a moving image for reproduction.

  In the program according to this aspect, the playback moving image generation means may generate the playback moving image by combining the replay moving image and the captured moving image according to a given rule. In this way, it is possible to generate a playback moving image in which the replay moving image and the captured moving image are combined according to a given rule.

  In the program according to this aspect, the reproduction moving image generation means combines the replay moving image and the captured moving image according to a rule determined based on the replay moving image to generate a reproduction moving image. It may be generated. In this way, it is possible to generate a playback moving image in which the replay moving image and the captured moving image are combined according to a rule determined based on the replay moving image.

  In the program according to this aspect, the computer may further function as a rule changing unit that changes a rule for combining the replay moving image and the captured moving image in accordance with a user operation. In this way, the user can control the synthesis rule between the replay moving image and the captured moving image.

  In the program according to an aspect of the present invention, the moving image for reproduction is output to the moving image output server for outputting the moving image in association with the user's identifier in response to a request from the user. The computer is further caused to function as output means. In this way, a user who can access the moving image output server can view the reproduction moving image output to the moving image output server.

  In the program according to one aspect of the present invention, the replay data indicates at least the content of an operation by the user. By doing so, it is possible to generate a moving image for reproduction based on the content of the operation by the user.

  In the program according to one embodiment of the present invention, the imaging period is a period longer than the reproduction period. By doing this, it is possible to generate a moving image for reproduction based on a captured moving image captured during an imaging period that is not included in the reproduction period.

  Each of the above programs can also be stored in a computer-readable information storage medium.

It is a hardware block diagram which shows an example of the hardware configuration of the entertainment system used as an image processing apparatus which concerns on this embodiment. It is a detailed block diagram which shows an example of the detail of MPU. It is a perspective view which shows an example of a controller. It is an upper side view showing an example of a controller. It is a functional block diagram which shows an example of the function of the entertainment system which concerns on this embodiment. It is a figure which shows an example of a game screen. It is a figure which shows an example of virtual three-dimensional space. It is a figure which shows an example of the flow of the process performed with the entertainment system which concerns on this embodiment. It is a figure which shows an example of the frame image for reproduction | regeneration.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating a hardware configuration of an entertainment system (image processing apparatus) 10 according to the present embodiment. As shown in FIG. 1, an entertainment system 10 includes an MPU (Micro Processing Unit) 11, a main memory 20, an image processing unit 24, a monitor 26, an input / output processing unit 28, an audio processing unit 30, and a speaker. 32, an optical disk reading unit 34, an optical disk 36, a hard disk drive 38, interfaces (I / F) 40, 44, a controller 42, a camera unit 46, and a network interface 48. System.

  FIG. 2 is a diagram illustrating the configuration of the MPU 11. As shown in FIG. 2, the MPU 11 includes a main processor 12, sub-processors 14 a, 14 b, 14 c, 14 d, 14 e, 14 f, 14 g, 14 h, a bus 16, a memory controller 18, and an interface (I / F) 22. And comprising.

  The main processor 12 is an operating system stored in a ROM (Read Only Memory) (not shown), for example, a program and data read from an optical disc 36 such as a DVD (Digital Versatile Disk) -ROM, and a program supplied via a communication network. Based on the data and the like, various information processing is performed, and the sub-processors 14a to 14h are controlled.

  The sub-processors 14a to 14h perform various types of information processing in accordance with instructions from the main processor 12, and each part of the entertainment system 10 is read from the optical disk 36 such as a DVD-ROM or the like via a communication network. Control based on supplied programs and data.

  The bus 16 is used for exchanging addresses and data among the various parts of the entertainment system 10. The main processor 12, the sub processors 14a to 14h, the memory controller 18, and the interface 22 are connected to each other via the bus 16 so as to be able to exchange data.

  The memory controller 18 accesses the main memory 20 in accordance with instructions from the main processor 12 and the sub processors 14a to 14h. Programs and data read from the optical disk 36 and hard disk drive 38 and programs and data supplied via the communication network are written to the main memory 20 as necessary. The main memory 20 is also used for work of the main processor 12 and the sub processors 14a to 14h.

  An image processing unit 24 and an input / output processing unit 28 are connected to the interface 22. Data exchange between the main processor 12 and the sub processors 14 a to 14 h and the image processing unit 24 or the input / output processing unit 28 is performed via the interface 22.

  The image processing unit 24 includes a GPU (Graphical Processing Unit) and a frame buffer. The GPU renders various screens in the frame buffer based on the image data supplied from the main processor 12 and the sub processors 14a to 14h. The screen formed in the frame buffer is converted into a video signal at a predetermined timing and output to the monitor 26. As the monitor 26, for example, a home television receiver is used.

  An audio processing unit 30, an optical disk reading unit 34, a hard disk drive 38, and interfaces 40 and 44 are connected to the input / output processing unit 28. The input / output processing unit 28 controls data exchange between the main processor 12 and the sub-processors 14a to 14h, and the audio processing unit 30, the optical disk reading unit 34, the hard disk drive 38, the interfaces 40 and 44, and the network interface 48. .

  The sound processing unit 30 includes an SPU (Sound Processing Unit) and a sound buffer. The sound buffer stores various audio data such as game music, game sound effects and messages read from the optical disc 36 and the hard disk drive 38. The SPU reproduces these various audio data and outputs them from the speaker 32. As the speaker 32, for example, a built-in speaker of a home television receiver is used.

  The optical disk reading unit 34 reads programs and data stored on the optical disk 36 in accordance with instructions from the main processor 12 and the sub processors 14a to 14h. Note that the entertainment system 10 may be configured to be able to read a program or data stored in a computer-readable information storage medium other than the optical disk 36.

  The optical disk 36 is a general optical disk (computer-readable information storage medium) such as a DVD-ROM. The hard disk drive 38 is a general hard disk device. Various programs and data are stored in the optical disk 36 and the hard disk drive 38 so as to be readable by a computer.

  The interfaces (I / F) 40 and 44 are interfaces for connecting various peripheral devices such as the controller 42 and the camera unit 46. For example, a USB (Universal Serial Bus) interface is used as such an interface.

  The controller 42 is general-purpose operation input means, and is used by the user to input various operations (for example, game operations). The input / output processing unit 28 scans the state of each unit of the controller 42 every predetermined time (for example, 1/60 seconds), and supplies the operation state representing the result to the main processor 12 and the sub processors 14a to 14h. The main processor 12 and the sub processors 14a to 14h determine the content of the operation performed by the user based on the operation state. The entertainment system 10 is configured so that a plurality of controllers 42 can be connected, and the main processor 12 and the sub-processors 14a to 14h execute various processes based on an operation state input from each controller 42. ing.

  The camera unit 46 includes, for example, a known digital camera, and inputs black and white, grayscale, or color photographed images every predetermined time (for example, 1/60 seconds). The camera unit 46 in the present embodiment is configured to input a captured image as image data in JPEG (Joint Photographic Experts Group) format. The camera unit 46 is installed on the monitor 26 with the lens facing the user, for example, and is connected to the interface 44 via a cable. In the present embodiment, the camera unit 46 includes a microphone and inputs external sound as sound data. The network interface 48 is connected to the input / output processing unit 28 and the network, and relays that the entertainment system 10 performs data communication with other entertainment systems 10 via the network.

  FIG. 3A is a perspective view illustrating an example of the controller 42. FIG. 3B is an upper side view showing an example of the controller 42. As shown in FIG. 3A, the controller 42 is connected to the entertainment system 10 by a controller cable 62, and a direction button group 60 (a left button 60L, a right button 60R, an up button 60U, and a down button on the left side of the surface 42a. A direction button 60D) and a left operation stick 54 are provided, and a button group 58 (a left button 58L, a right button 58R, an upper button 58U, and a lower button 58D) and a right operation stick 56 are provided on the right side. Further, as shown in FIG. 3B, on the back side surface of the controller 42, a first left button 50 </ b> L and a first right button 50 </ b> R are provided on the left and right sides of the front surface 42 a, respectively, A left button 52L and a second right button 52R are provided. When the user holds both sides of the housing of the controller 42 with both hands, the left thumb is positioned on the direction button group 60 and the left operation stick 54, and the right thumb is positioned on the button group 58 and the right operation stick 56. And at least one of the right index finger or middle finger is positioned on the first right button 50R or the second right button 52R, and at least one of the left index finger or middle finger is the first left button 50L or second The left button 52L is located.

  The direction button group 60, the button group 58, the first left button 50L, the first right button 50R, the second left button 52L, and the second right button 52R are configured as pressure sensitive buttons and include a pressure sensor. ing. When the user presses these buttons, a 256-step digital value having a value of 0 to 255 is input to the entertainment system 10 according to the pressing force. That is, in the entertainment system 10, when the digital value input from the controller 42 is 0, the button is not pressed, and when the digital value is 255, the digital button indicates that the button is pressed with the maximum pressing force. It can be judged from the value.

  The left operation stick 54 and the right operation stick 56 are stick-like operation members standing upright on the surface of the housing of the controller 42, and can be tilted by a predetermined angle in all directions from this upright state. As shown in FIG. 3A, the casing longitudinal direction of the controller 42 is the X-axis direction (the right direction in FIG. 3A is the positive direction), and the casing depth direction orthogonal to the X-axis direction is the Y-axis direction (see FIG. 3A). 3A, the direction from the front to the back is the positive direction.) The posture (operation state) of the left operation stick 54 is the inclination (posture data (X, Y)) in the X-axis direction and the Y-axis direction. Digital values of 0 to 255 are input to the entertainment system 10 respectively. Specifically, the vicinity of X = 127 or X = 128 indicates that the left operation stick 54 is not inclined in the X-axis direction. X = 255 indicates that the left operation stick 54 is tilted to the limit in the positive direction of the X axis (rightward in FIG. 3A). Further, X = 0 indicates that the left operation stick 54 has fallen to the limit in the negative direction of the X axis (leftward in FIG. 3A). The same applies to the Y-axis direction. The right operation stick 56 is the same as that of the left operation stick 54. In this way, the entertainment system 10 can grasp the current tilt state (posture) of the left operation stick 54 and the right operation stick 56. The left operation stick 54 and the right operation stick 56 are also configured as pressure-sensitive buttons similar to the direction button group 60 and the button group 58, and can be pressed in the axial direction of the stick.

  The controller 42 includes a built-in vibrator (vibrator). The vibrator vibrates in accordance with an instruction from the MPU 11.

  Here, functions realized by the entertainment system 10 according to the present embodiment will be described. FIG. 4 is a functional block diagram showing an example of functions realized by the entertainment system 10 according to the present embodiment.

  As shown in FIG. 4, the entertainment system 10 according to this embodiment functionally includes a program execution unit 64, a replay data generation unit 66, a replay data acquisition unit 68, a captured moving image acquisition unit 70, and a data holding unit 72. , The playback moving image generation unit 74, the rule change unit 76, and the playback moving image output unit 78. The entertainment system 10 does not have to include all of these elements. The data holding unit 72 is mainly realized by a storage device included in the entertainment system 10 such as the main memory 20 and the hard disk drive 38. Other elements are mainly realized by a control device provided in the entertainment system 10 such as the MPU 11 and the image processing unit 24.

  These elements are realized by executing a program installed in the entertainment system 10, which is a computer, by a control device such as the MPU 11 provided in the entertainment system 10. This program is supplied to the entertainment system 10 via an information transmission medium such as a CD-ROM or DVD-ROM, or via a communication network such as the Internet.

  The program execution unit 64 executes a program such as a game program installed in the entertainment system 10, for example. In the present embodiment, for example, the program execution unit 64 generates the game screen 80 shown in FIG. 5 and displays and outputs it on the monitor 26 or the like. In the present embodiment, the game screen 80 shown in FIG. 5 is a visualization of the virtual three-dimensional space 82 generated by the game program shown in FIG. As shown in FIG. 6, in the virtual three-dimensional space 82, for example, at least one block 84 (each block 84 includes at least one polygon) is arranged. Then, the program execution unit 64 displays and outputs, on the monitor 26, an image showing a state in which the viewing direction 88 is viewed from the viewpoint 86 as the game screen 80. In the present embodiment, the program execution unit 64 changes the position of the viewpoint 86 and the line-of-sight direction 88, for example, according to a user operation. Of course, the program execution unit 64 may execute generation, deformation, movement, deletion, and the like of the block 84 in the virtual three-dimensional space 82 in accordance with a user operation.

  The replay data generation unit 66 generates replay data for reproducing the execution status of the program during execution of the program. In the present embodiment, the replay data generation unit 66 reproduces the game play status (for example, the play progress status of the stage) by the user during the execution of the game program corresponding to the predetermined stage in the game. Generate replay data. In the present embodiment, the replay data generation unit 66 generates replay data including operation data indicating the content of an operation by a user that is accepted during execution of the game program corresponding to the above-described stage. The operation data may be associated with order data indicating the order in which the user performed the operation. The operation data may be associated with operation timing data indicating the timing of the operation by the user (for example, the number of frames that have elapsed since the start of program execution or operation timing data that indicates the elapsed time). Note that the replay data may indicate the position (for example, coordinates) of the object (block 84, character, etc.) at each frame or at a certain time.

  In the present embodiment, the replay data generation unit 66 generates replay data for a period (execution period) from the start to the end of program execution by the program execution unit 64, for example.

  The replay data generation unit 66 also includes an event (for example, an event in the game (more specifically, an event indicating that the puzzle has been solved, a score, etc.) that occurs during the execution of the program by the program execution unit 64. An event indicating that the character in the game has laughed) or an event received from the outside (for example, a sound of a predetermined volume or higher is input from the microphone of the camera unit 46) Replay data including event data indicating the content of the event) may be generated. The event data may be associated with event timing data indicating the timing at which the event has occurred.

  The replay data acquisition unit 68 acquires the replay data generated by the replay data generation unit 66.

  The captured moving image acquisition unit 70 acquires the captured moving image generated by the imaging unit such as the camera unit 46 during the imaging period including at least a part of the reproduction period in which the program execution status is reproduced by the replay data. In the present embodiment, the captured moving image acquisition unit 70 acquires a moving image obtained by capturing an image of the user playing the above-described stage. In the present embodiment, the above-described reproduction period corresponds to a period including the generation timing of each operation data included in the replay data. In the present embodiment, the captured moving image acquisition unit 70 sequentially acquires captured frame images generated by the camera unit 46 every predetermined time (for example, every 1/60 seconds), for example. In this case, the captured moving image includes at least one captured frame image. The captured moving image acquisition unit 70 may write the acquired captured moving image in a storage device such as the main memory 20 or the hard disk drive 38, for example.

  Note that the above imaging period does not have to coincide with the execution period of the program. Specifically, for example, at least a predetermined length of time after the user has finished playing the above-described stage, or a predetermined length of time before the user has started to play the above-described stage. For a period including one, the captured moving image acquisition unit 70 may acquire a moving image captured by the camera unit 46. Thus, the imaging period by the camera unit 46 may be a period longer than the program execution period, including the program execution period.

  The data holding unit 72 holds the replay data and the captured moving image in association with each other.

  The reproduction moving image generation unit 74 generates a reproduction moving image based on the replay data held in the data holding unit 72 and the captured moving image. In the present embodiment, the reproduction moving image generation unit 74 generates a reproduction moving image by reproducing a moving image based on the replay data and the captured moving image.

  In the present embodiment, the reproduction moving image generation unit 74 causes the program execution unit 64 to execute a game program corresponding to the above-described stage. Then, the playback moving image generation unit 74 outputs the replay data held in the data holding unit 72 to the program execution unit 64, for example. Then, the program execution unit 64 acquires the replay data output from the playback moving image generation unit 74. And the program execution part 64 produces | generates the replay frame image corresponding to the above-mentioned game screen 50 about each frame based on this replay data. Specifically, for example, the program execution unit 64 considers that an operation signal corresponding to the replay data has been received (accepted as a pseudo operation signal), and generates a replay frame image for each frame. In this way, a replay moving image (reproduced moving image) that reproduces a state in which the user is playing the above-described stage including at least one replay frame image is generated. The reproduction moving image generation unit 74 may give an instruction to the program execution unit 64 to change the image generation mode. And the program execution part 64 may produce | generate the replay frame image from which the display mode differs from the image which the game screen 50 shows (for example, the viewpoint 86 and the gaze direction 88 in the virtual three-dimensional space 82 differ). Specifically, for example, in the case where the program execution unit 64 is executing a soccer game program, the player who has shot the shot corresponds to the game screen 50 showing a state where the program execution unit 64 looks down on the player object. You may make it produce | generate the replay frame image zoomed up to the object.

  The reproduction moving image generation unit 74 generates a reproduction moving image based on a moving image generated based on the replay data (for example, a replay moving image (reproduction moving image)) and a captured moving image. Also good. The replay moving image may include at least one replay frame image (for example, a replay image (reproduced image) of a game program).

  The rule changing unit 76 changes a rule for combining the replay moving image and the captured moving image in accordance with an operation by the user. Details of the rule changing unit 76 will be described later.

  Here, an example of the flow of processing executed every predetermined time (for example, every 1/60 seconds) in the entertainment system 10 in the present embodiment will be described with reference to the flowchart shown in FIG.

  In the present embodiment, the start timing of the execution period described above corresponds to the start timing of the imaging period described above. The end timing of the execution period described above corresponds to the end timing of the imaging period described above. In the following processing example, an example of a processing flow corresponding to the nth frame (nth frame) from the start of the above-described execution period will be described. In addition, the data holding unit 72 holds replay data and captured moving images in association with each other in advance. In addition, the reproduction moving image generation unit 74 causes the program execution unit 64 to execute a game program corresponding to the stage to be reproduced in advance. The reproduction moving image generation unit 74 outputs the replay data held in the data holding unit 72 to the program execution unit 64.

  First, the playback moving image generation unit 74 causes the program execution unit 64 to generate a replay frame image in the nth frame (that is, a replay frame image at the point of (1/60) × n seconds from the start of the execution period). (S101).

  Then, the playback moving image generation unit 74 holds the captured frame image in the nth frame (that is, the captured frame image at the time when (1/60) × n seconds have elapsed from the start of the imaging period) held in the data holding unit 72 ) Is acquired (S102).

  Then, the playback moving image generation unit 74 combines the replay frame image in the nth frame and the captured frame image in the nth frame to generate a playback frame image 90 (see FIG. 8) in the nth frame (see FIG. 8). S103). FIG. 8 shows an example of the playback frame image 90. Details of the processing shown in S103 will be described later.

  Then, the reproduction moving image generation unit 74 reproduces the reproduction frame image 90 in the nth frame (for example, displays the reproduction frame image 90 on the monitor 26) (S104).

  Then, the playback moving image generation unit 74 records the playback frame image 90 in the nth frame in a storage device such as the main memory 20 or the hard disk drive 38 (S105). The playback frame image 90 is associated with, for example, a frame number corresponding to the elapsed time from the start of playback, data indicating the time from the start of playback, and the like. As described above, the reproduction moving image generation unit 74 may generate the reproduction moving image by recording the reproduction moving image.

  The processes shown in S101 to S105 described above are executed sequentially from the first frame to the last frame. Thus, in the present embodiment, a playback moving image including at least one playback frame image 90 is recorded in a storage device such as the main memory 20 or the hard disk drive 38.

  For example, the reproduction moving image generation unit 74 synthesizes the replay frame image and the captured frame image in accordance with a given rule. Specifically, for example, the reproduction moving image generation unit 74 overwrites a reduced image 92 of the captured frame image on a predetermined region in the replay frame image (for example, an upper right region in the replay frame image), for example. A playback frame image 90 is generated. Further, the playback moving image generation unit 74 may generate the playback frame image 90 by overwriting the reduced image 92 of the captured frame image with reduced image quality in the area of the replay frame image, for example. In addition, the reproduction moving image generation unit 74 arranges the reduced image 92 of the replay frame image in the lower left area of the reproduction frame image 90 and reduces the captured frame image in the upper right area of the reproduction frame image 90, for example. A playback frame image 90 may be generated by arranging the image 92. As described above, the playback moving image generation unit 74 may generate the playback moving image by combining the replay moving image and the captured moving image according to a given rule.

  Note that the playback moving image generation unit 74 specifies a region in the replay frame image where an image related to a program (for example, a game program) is not displayed by a known image processing technique, and the captured frame image is displayed in that region. A playback frame image may be generated by superimposing the reduced images 92. In addition, the reproduction moving image generation unit 74 determines a color determined based on a distribution (for example, a color distribution) of pixels included in the replay frame image, and reduces the captured frame image processed so as to be the color. The playback frame image 90 may be generated by overwriting the area in the replay frame image with the image 92. In addition, the reproduction moving image generation unit 74 determines translucency based on pixel variations (for example, color variations) included in the replay frame image, and reduces the captured frame image processed to have the translucency. The playback frame image 90 may be generated by overwriting the area in the replay frame image with the image 92. As described above, the playback moving image generation unit 74 may generate the playback moving image by combining the replay moving image and the captured moving image in accordance with a rule determined based on the replay moving image.

  When the replay data generation unit 66 generates event data, the playback moving image generation unit 74 applies the playback frame image 90 corresponding to the timing indicated by the event data (for example, event generation timing). An image effect based on the contents of the event may be added. In the above-described case, the reproduction moving image generation unit 74 may increase the area occupied by the reduced image 92 of the captured frame image in the reproduction frame image 90, for example. In the above-described case, the playback moving image generation unit 74 may generate the playback frame image 90 by overwriting the replay frame image with the reduced image 92 of the captured frame image, for example. In the above-described case, the reproduction moving image generation unit 74 may perform processing such as adding a color, adding a balloon, or shifting the reproduction frame image 90, for example. Further, in the above-described case, the playback moving image generation unit 74 generates a playback frame image 90 so as to perform slow playback (for example, generates a playback frame image 90 that is common to a predetermined number of frames). May be.

  Further, the playback moving image generation unit 74 may determine whether an event has occurred based on the replay frame image. Then, when it is determined that an event has occurred, a predetermined image effect may be added to the reproduction frame image 90 corresponding to the replay frame image.

  If the execution period of the program does not correspond to the imaging period by the camera unit 46, there may be no imaging frame image corresponding to the replay frame image or no replay frame image corresponding to the imaging frame image. Conceivable. In this case, the reproduction moving image generation unit 74 complements the non-existent captured frame image or replay frame image according to a given rule (for example, the replay frame image corresponding to the last frame in the program execution period, A playback frame image 90 may be generated (using the captured frame image).

  Further, in the present embodiment, the rule changing unit 76 receives the operation signal indicating that a predetermined button has been pressed in response to an operation by the user (for example, in response to the operation), and the replay moving image and the captured moving image. Change the rules for combining images. Specifically, for example, the rule changing unit 76 generates a playback frame image 90 that includes a reduced image 92 of the captured frame image or includes a reduced image 92 of the captured frame image according to the operation signal. Whether to generate a non-reproducing frame image 90 is switched. In this way, whether or not the user includes the reduced image 92 of the captured frame image in the reproduction frame image 90 while viewing the reproduced moving image during reproduction of the moving image by the reproduction moving image generation unit 74. You can switch between no.

  In addition, the rule changing unit 76 generates a playback frame image 90 that partially includes the reduced image 92 of the captured frame image, or a playback frame image that does not include the reduced image 92 of the captured frame image, according to the operation data. You may make it switch whether 90 is produced | generated. In this way, during execution of the program by the program execution unit 64, the user can switch whether or not to include a part of the image based on the captured moving image in the playback frame image 90. Further, the rule changing unit 76 may change the position and size of the captured frame image arranged in the reproduction frame image 90 in accordance with an operation by the user.

  In response to a request from the user, the playback moving image output unit 78 associates the playback moving image recorded in the storage device with the user identifier (for example, user ID or user name) via the network. It outputs to the moving image output server connected with the entertainment system 10. Specifically, for example, the playback moving image output unit 78 uploads a playback moving image in which a game replay moving image is combined with a captured moving image to a moving image output server that is a moving image distribution site. In this way, the playback moving image output to the moving image output server can be provided to other users, so that other users can enjoy the replay moving image of the game. At this time, the identifier of the user who requests the output is compared with the identifier of the user associated with the playback moving image to control whether to allow the user to view the playback moving image. You may do it.

  Further, the playback moving image output unit 78 may output the playback moving image to the entertainment system 10 of another user. Then, in response to a request from another user, when the entertainment system 10 reproduces the playback moving image, the camera unit 46 captures how the other user is viewing the playback moving image. Then, an appreciation moving image may be generated. Then, the entertainment system 10 may output the viewing moving image or a combined moving image of the viewing moving image and the playback moving image to the entertainment system 10 that is the transmission source of the playback moving image. In this way, it is possible for the user who has created the playback moving image to enjoy the situation where other users are viewing the playback moving image. Note that the entertainment system 10 may transmit and receive a playback moving image, a viewing moving image, and a synthesized moving image via a server.

  The present invention is not limited to the above embodiment.

  For example, the reproduction moving image generation unit 74 may generate a replay moving image based on the replay data, and then generate the reproduction moving image by combining the replay moving image and the captured moving image.

  For example, the replay data may be associated with the captured frame image. Specifically, for example, at least a part of the generated replay data includes a frame number indicating the order of the captured frame images included in the captured moving image (for example, a frame number indicating the number of frames from the start of the imaging period). ) Or the like. Then, the reproduction moving image generation unit 74 sequentially generates the replay frame image of each frame based on the replay data, and generates the replay frame image and the replay data that is the basis for generating the replay frame image. The frame image for reproduction 90 may be generated by synthesizing the associated captured frame image. Further, at least a part of the captured frame image included in the captured moving image may be associated with an identifier of replay data (for example, replay data indicating the position of the character). Then, the playback moving image generation unit 74 sequentially plays back the captured frame images in the order indicated by the frame numbers included in the captured frame images, and plays back the captured frame images and the replays associated with the captured frame images. The replay frame image 90 may be generated by combining the replay frame image showing a state where each character is arranged at a position determined based on the data.

  In addition, for example, the playback moving image generation unit 74 uses the playback frame based on the replay frame image and the captured frame image whose timing is shifted by a predetermined number of frames from the timing corresponding to the replay frame image. The image 90 may be generated.

  For example, the captured moving image acquisition unit 70 may acquire a plurality of captured moving images generated by the plurality of camera units 46. The rule changing unit 76 switches the captured moving image (captured frame image) to be synthesized in response to a request from the user, and the playback moving image generating unit 74 performs the captured moving image (captured frame image). A moving image for reproduction based on the above may be generated.

  Further, the playback moving image generation unit 74 may generate a playback moving image based on the captured moving image or the replay moving image generated by each of the plurality of entertainment systems 10. Specifically, for example, the reproduction moving image generation unit 74 may generate a reproduction moving image by combining a plurality of replay moving images and a plurality of captured moving images based on a given rule.

  10 Entertainment System, 11 MPU, 12 Main Processor, 14a-14h Sub-Processor, 16 Bus, 18 Memory Controller, 20 Main Memory, 22, 40, 44 Interface, 24 Image Processing Unit, 26 Monitor, 28 Input / Output Processing Unit, 30 Audio processing unit, 32 speaker, 34 optical disk reading unit, 36 optical disk, 38 hard disk drive, 42 controller, 42a surface, 46 camera unit, 48 network interface, 50L first left button, 50R first right button, 52L second Left button, 52R second right button, 54 left operation stick, 56 right operation stick, 58 button group, 58L left button, 58R right button, 58U up button, 58D down button, 60 direction Direction button group, 60L Left button, 60R Right button, 60U Up button, 60D Down button, 62 Controller cable, 64 Program execution unit, 66 Replay data generation unit, 68 Replay data acquisition unit, 70 Captured moving image acquisition Part, 72 data holding part, 74 reproduction moving picture generation part, 76 rule changing part, 78 reproduction moving picture output part, 80 game screen, 82 virtual three-dimensional space, 84 blocks, 86 viewpoints, 88 gaze directions, 90 reproductions Frame image, 92 Reduced image.

Claims (15)

Replay data acquisition means for acquiring replay data for reproducing the execution status of the program, which is generated when an image generated in accordance with a user operation received during execution of the program is displayed on the display means;
After obtaining the replay data, a server that is a moving image distribution site by executing the program in a mode different from that for generating the image displayed on the display unit based on the generated replay data A reproduction moving image generating means for generating a reproduction moving image composed of a frame image different from the image displayed on the display means,
An imaging moving image acquisition means for acquiring an imaging moving image generated by an imaging means during an imaging period including at least a part of a reproduction period in which the execution status of the program is reproduced by the replay data;
Means for causing the holding means to hold the replay data and the captured moving image in association with each other;
A program characterized by causing a computer to function. The reproduction moving image generating means generates a reproduction moving image based on the replay data and the captured moving image.
The program according to claim 1 . The reproduction moving image generating means generates the reproduction moving image by reproducing a moving image based on the replay data and the captured moving image.
The program according to claim 2 , wherein: The reproduction moving image generation means generates the reproduction moving image based on the replay moving image generated based on the replay data and the captured moving image.
The program according to claim 2 or 3 , characterized in that. The reproduction moving image generation means generates the reproduction moving image by combining the replay moving image and the captured moving image according to a given rule.
The program according to claim 4 . The reproduction moving image generating means generates the reproduction moving image by combining the replay moving image and the captured moving image in accordance with a rule determined based on the replay moving image;
The program according to claim 5 . The replay moving image includes at least one replay frame image;
The captured moving image includes at least one captured frame image;
The playback moving image includes at least one playback frame image;
The reproduction moving image generation means generates a reproduction frame image by synthesizing a captured frame image corresponding to the replay frame image in an area where an image related to the program in the replay frame image is not displayed.
The program according to claim 6 . The replay moving image includes at least one replay frame image;
The captured moving image includes at least one captured frame image;
The playback moving image includes at least one playback frame image;
The reproduction moving image generation means generates a reproduction frame image by complementing a non-existent image according to a given rule when there is no other to be combined with one of the replay frame image or the captured frame image.
The program according to any one of claims 4 to 6 , wherein: Further causing the computer to function as rule changing means for changing a rule for combining the replay moving image and the captured moving image in accordance with an operation by a user.
The program according to any one of claims 5 to 8 , characterized by: Causing the moving image output server that outputs moving images to further function the computer as reproducing moving image output means for outputting the reproducing moving image in association with an identifier of the user in response to a request from the user;
The program according to any one of claims 2 to 9 , wherein: The replay data indicates at least the content of an operation by the user;
The program according to any one of claims 1 to 10 , wherein: The imaging period is a period longer than the reproduction period;
The program according to any one of claims 1 to 11 , characterized in that: A computer-readable information storage medium storing the program according to any one of claims 1 to 12 . Replay data acquisition means for acquiring replay data for reproducing the execution status of the program, which is generated when a moving image generated according to a user operation received during execution of the program is displayed on the display means; ,
After obtaining the replay data, a server that is a moving image distribution site by executing the program in a mode different from that for generating the image displayed on the display unit based on the generated replay data A reproduction moving image generating means for generating a reproduction moving image composed of a frame image different from the image displayed on the display means,
Imaging moving image acquisition means for acquiring an imaging moving image generated by an imaging means during an imaging period including at least a part of a reproduction period in which the execution status of the program is reproduced by the replay data;
Means for causing the holding means to hold the replay data and the captured moving image in association with each other;
An image processing apparatus comprising: The replay data acquisition unit included in the image processing apparatus reproduces the execution status of the program generated when a moving image generated in response to a user operation received during execution of the program is displayed on the display unit. A replay data acquisition step for acquiring replay data for
The reproduction moving image generation means included in the image processing apparatus is in a mode different from the generation of the image displayed on the display means based on the replay data generated after the replay data is acquired. A reproduction moving image that generates a reproduction moving image composed of a frame image different from the image displayed on the display means, which is a target to be uploaded to a server that is a moving image distribution site by executing the program. An image generation step;
An imaging moving image acquisition step of acquiring an imaging moving image generated by an imaging unit in an imaging period including at least a part of a reproduction period in which the execution status of the program is reproduced by the replay data;
Holding the replay data and the captured moving image in association with each other in a holding unit;
An image processing method comprising:

Images