JP2019140567A - Image processing system - Google Patents

Abstract

To create a nice looking video by creating a video according to the situation, when creating a video from a panoramic image.SOLUTION: When performing creation/display of videos for a prescribed time with regard to one panoramic image, trimming is carried out by sequentially moving the position of trimming region of the panoramic image, and creation/display of videos is performed so that multiple trimming images, subjected to trimming while changing the position, are reproduced sequentially. Trimming of the panoramic image at a prescribed aspect ratio (S906) and trimming of the panoramic image at an aspect ratio different from the prescribed aspect ratio (S907, S908) are changed over according to the size of the panoramic image in a prescribed direction or the aspect ratio (S901). Furthermore, when trimming at an aspect ratio different from the prescribed aspect ratio, the creation/display time of the video is prolonged, according to the size of the panoramic image in the prescribed direction or the aspect ratio.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image processing apparatus that handles panoramic images.

  2. Description of the Related Art Conventionally, it has become common for digital cameras and smartphones to create a series of album videos using captured still images and moving images and enjoy them with family and friends. In order to enjoy album moving images, there are provided elaborate effects such as adding character data to still images and moving images as materials of album moving images and recording transitions such as fades and BGM together.

  Also, various photographing methods are provided in recent digital cameras and smartphones. Among them, panoramic photography that captures a wide range of scenery is also widely used. A panorama image shot by panorama shooting is recorded with an aspect that is significantly different from a still image shot normally. The panoramic image expresses the entire atmosphere at the time of shooting, and is suitable as a material target of the album moving image. It is necessary to convert a panorama image into a movie in order to import it into an album movie. Patent Document 1 describes a method of creating a panning video overlooking the entire panoramic image from the panoramic image.

Japanese Patent Application Laid-Open No. 2011-082917

  Generating and reproducing a panning video from a panoramic image is an effective expression means. However, simply creating a panning video may impair the appearance of the panning video. For example, when a plurality of images including a panoramic image are played back continuously, the playback time of other scenes is determined to be 4 seconds, and if the panning video is much longer than 4 seconds, the overall appearance is improved. It will be lost. Or, despite the fact that BGM is a slow tempo song, the panning video slide speed is fast and the difference from other scenes is conspicuous, so it is not appropriate as one scene of the album video. As a result, the entire album video It will happen that the appearance of will be impaired. In view of the above, the present invention has an object to generate a good-looking moving image by generating a moving image according to the situation when generating a moving image from a panoramic image.

In order to solve the above-described problems, an image processing apparatus according to the present invention provides:
An acquisition unit that acquires a panoramic image and a moving image generation unit that generates a moving image based on the panoramic image, and sequentially trims the position of the trimming area when trimming the panoramic image, and changes the position to perform trimming. Moving image generating means for generating a moving image for a predetermined time in which the plurality of trimmed images are sequentially reproduced, and the moving image generating means determines the panoramic image according to the size or aspect ratio of the panoramic image in a predetermined direction. When the aspect ratio of the panorama image is switched or the panorama image is trimmed with an aspect ratio different from the predetermined aspect ratio, and the aspect ratio different from the predetermined aspect ratio is trimmed, the aspect ratio of the panoramic image or the predetermined direction is selected. Generate videos longer than the specified time depending on the size of And features.

Alternatively, the image processing apparatus of the present invention is
An acquisition unit that acquires a panoramic image, and a display control unit that controls to trim a partial area of the panoramic image and display the trimmed image, in order to display a single panoramic image for a predetermined period. Display control means for sequentially moving and trimming the position of the trimming area of the panoramic image and controlling the trimmed image that has been trimmed by changing the position within a predetermined period, and the display control means Depending on the size or aspect ratio of the panoramic image in a predetermined direction, the panorama image can be trimmed with a predetermined aspect ratio or the panoramic image can be trimmed with an aspect ratio different from the predetermined aspect ratio. When cropping with an aspect ratio different from the Depending on's or aspect ratio, characterized by a period for displaying the one panoramic image and a period longer than the predetermined time period.

  According to the present invention, an appropriate moving image can be generated from a panoramic image.

1 is a configuration block diagram of a digital camera 100. FIG. It is a flowchart of a panoramic shooting mode. It is a figure which shows the relationship between the motion of a digital camera at the time of panoramic image photography, and a picked-up image. It is a figure explaining the image composition at the time of panoramic image photography. It is a flowchart of reproduction | regeneration mode. It is a flowchart of an album creation process. It is a flowchart of an album moving image saving process. It is a flowchart of the process which adds JPEG data to an album moving image. It is a flowchart of the determination process of the trimming method of a panoramic image. It is a schematic diagram for demonstrating trimming of a horizontally long panoramic image. It is a flowchart of the process which adds MP4 data to an album moving image. It is a flowchart of a synthesis process of BGM. It is an example of the screen display of the menu regarding album creation. It is a schematic diagram for demonstrating trimming of a vertically long panoramic image. It is a figure which shows the structure of a scenario.

[Example 1]
In the present embodiment, a digital camera as an image pickup apparatus will be described as an example. Or you may implement | achieve with the information processing apparatus, image processing apparatus, etc. which can acquire the image image | photographed with the imaging device via a recording medium or communication.

  Hereinafter, the main configuration of the imaging apparatus (digital camera) 100 of the present embodiment will be described with reference to FIG.

  In FIG. 1, the control unit 101 includes, for example, a CPU (MPU), a memory (DRAM, SRAM), a nonvolatile memory (EEPROM), and the like. Then, the program is read from the nonvolatile memory (EEPROM), and various processes are executed based on the read program to control each block of the imaging apparatus 100, and to control data transfer between the blocks. Further, the control unit 101 controls each block of the imaging apparatus 100 in accordance with an operation signal from the operation unit 102 that receives an operation from the user. Further, the control unit 101 analyzes an image obtained by an image processing unit 111 described later, and controls each block of the imaging device 100 according to the analysis result.

  The operation unit 102 includes switches for inputting various operations related to shooting such as a power button, a shutter button (SW1, SW2), a moving image recording start button, a zoom adjustment button, and an autofocus button. Further, it includes a menu display button, a determination button, other cursor keys, a pointing device, a touch panel, and the like. When these keys and buttons are operated by the user, an operation signal is transmitted to the control unit 101.

  The bus 103 is a general-purpose bus for sending various data, control signals, instruction signals, and the like to each block of the imaging apparatus 100.

  The non-volatile memory 105 is an electrically erasable / recordable memory, and stores constants, programs, and the like for the operation of the control unit 101.

  The imaging unit 110 controls the light amount of the optical image of the subject captured by the lens using an aperture, converts the optical image into an image signal using an imaging element such as a CCD sensor or a CMOS sensor, performs analog-digital conversion, and performs an image processing unit. 111.

  The image processing unit 111 performs image quality adjustment processing for adjusting white balance, color, brightness, and the like on the input digital image signal based on setting values. The image signal processed by the image processing unit 111 is transmitted by the control unit 101 to the memory 104, the video output unit 150, which will be described later, and the display control unit 131.

  In this embodiment, for example, the imaging unit 110 has an “optical zoom” function and an “optical image stabilization” function, and the image processing unit 111 has an “electronic zoom” function and an “electronic image stabilization” function. ing. Here, the “optical zoom” function and the “electronic zoom” function are functions for enlarging an obtained image in accordance with a user operation. In addition, the “optical image stabilization” function and the “electronic image stabilization” function are functions that prevent image shake due to vibration of the imaging apparatus 100 main body. These functions are used simultaneously, alternately, or independently under the control of the control unit 101.

  Here, the “optical zoom” function is a function for enlarging / reducing the optical image of the captured subject by moving the lens of the imaging unit 110 in response to the user operating the zoom key of the operation unit 102. . In addition, the “electronic zoom” function refers to an enlargement of an image obtained by cutting out a part of the image generated by the imaging unit 110 by the image processing unit 111 in response to the user operating the zoom key of the operation unit 102. This is a process for generating the processed image signal. The “optical image stabilization” function calculates the amount of lens movement based on the value of an acceleration signal from a vibration detection unit (not shown), and prevents the image from being shaken by moving the lens. It is a function to do. Further, the “electronic image stabilization” function prevents image shaking by adjusting the position of the image signal acquired by the imaging unit 110 based on the value of an acceleration signal from a vibration detection unit (not shown). It is a function. The “electronic image stabilization” function can also be realized by shifting the readout position of the image sensor of the imaging unit 110 based on the value of an acceleration signal from a vibration detection unit (not shown). Since these functions are well-known techniques, detailed description is omitted.

  The audio input unit 120 collects (collects) audio around the imaging apparatus 100 using, for example, a built-in omnidirectional microphone or an external microphone connected via an audio input terminal, and analog It performs digital conversion and transmits it to the audio processing unit 121. The sound processing unit 121 performs sound-related processing such as level optimization processing of the input digital sound signal. The audio signal processed by the audio processing unit 121 is transmitted to the memory 104 by the control unit 101. The memory 104 temporarily stores the image signal and the audio signal obtained by the image processing unit 111 and the audio processing unit 121.

  The image processing unit 111 and the audio processing unit 121 read out the image signal and the audio signal temporarily stored in the memory 104 and perform the encoding of the image signal, the encoding of the audio signal, and the like, and the compressed image signal and the compressed audio signal And so on. The control unit 101 transmits the compressed image signal and the compressed audio signal to the recording / reproducing unit 140.

  The recording / reproducing unit 140 records the compressed image signal generated by the image processing unit 111 and the audio processing unit 121, the compressed audio signal, and other control data related to shooting on the recording medium 141. When the audio signal is not compression-encoded, the control unit 101 transmits the audio signal generated by the audio processing unit 121 and the compressed image signal generated by the image processing unit 111 to the recording / reproducing unit 140. Recording is performed on the recording medium 141.

  The recording medium 141 may be a recording medium built in the imaging apparatus or a removable recording medium. The recording medium 141 can record a compressed image signal, a compressed audio signal, an audio signal, various data including BGM data, which will be described later, and the like generated by the imaging apparatus 100, and has a larger capacity than the non-volatile memory 105. Generally used. For example, the recording medium 141 includes all types of recording media such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a nonvolatile semiconductor memory, and a flash memory.

  The recording / reproducing unit 140 reads (reproduces) a compressed image signal, a compressed audio signal, an audio signal, various data, and a program recorded on the recording medium 141. Then, the control unit 101 transmits the read compressed image signal and compressed audio signal to the image processing unit 111 and the audio processing unit 121. The image processing unit 111 and the audio processing unit 121 temporarily store the compressed image signal and the compressed audio signal in the memory 104, decode them in a predetermined procedure, and decode the decoded audio signal to the audio output unit 151. The signal is transmitted to the video output unit 150 and the display control unit 131. When the audio signal is recorded in the recording medium 141 in a non-compressed manner, the control unit 101 transmits the audio signal directly to the audio output unit 151.

  The audio output unit 151 includes an audio output terminal, for example, and transmits an audio signal in order to output audio from a connected earphone or speaker. In addition, the audio output unit 151 may be a speaker that is built in the imaging apparatus 100 and outputs audio related to an audio signal. The video output unit 150 includes, for example, a video output terminal, and transmits an image signal to display a video on a connected external display or the like. The audio output unit 151 and the video output unit 150 may be a single integrated terminal, for example, a terminal such as an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal.

  In addition, the display control unit 131 causes the display unit 130 to display a video based on the image signal transmitted from the image processing unit 111, an operation screen (menu screen) for operating the imaging apparatus 100, and the like. The display unit 130 may be any display device such as a liquid crystal display, an organic EL display, and electronic paper.

  The communication unit 152 performs communication between the imaging device 100 and an external device. For example, the communication unit 152 transmits or receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. Also, it transmits and receives control signals related to shooting, such as shooting start and end commands, and other information. The communication unit 152 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a wireless USB, or a GPS receiver.

  Here, the normal operation of the imaging apparatus 100 of the present embodiment will be described.

  In the imaging apparatus 100 according to the present exemplary embodiment, when the user operates the power button of the operation unit 102, a start instruction is issued from the operation unit 102 to the control unit 101. Upon receiving this instruction, the control unit 101 controls a power supply unit (not shown) to supply power to each block of the imaging apparatus 100.

  When power is supplied, for example, the control unit 101 instructs the operation unit 102 to indicate which mode, for example, the still image shooting mode, the moving image shooting mode, the playback mode, or the like, of the mode switching switch of the operation unit 102. Check by signal.

  In the still image shooting mode, the imaging apparatus 100 captures an image when the user operates a still image recording button of the operation unit 102 in a shooting standby state, and a compressed image signal is recorded on the recording medium 141. And it will be in a photography standby state again. In the moving image shooting mode, the imaging apparatus 100 starts shooting when the user operates the moving image recording start button of the operation unit 102 in a shooting standby state. During that time, the compressed image signal and the compressed audio signal or audio signal are transferred to the recording medium 141. To be recorded. Then, when the user operates the moving image recording end button of the operation unit 102, the photographing is finished, and the photographing standby state is entered again. In the reproduction mode, a compressed image signal, a compressed audio signal, or an audio signal related to a file selected by the user is reproduced from the recording medium 141, an audio signal is output from the audio output unit 151, and an image is displayed on the display unit 130.

  Hereinafter, various processes in the imaging apparatus 100 will be described with reference to flowcharts of FIGS. 2, 5, 6, 7, 8, 9, 11, and 12. The processing of these flowcharts is executed by the control unit 101 based on the program read from the nonvolatile memory 105 by the control unit 101.

  FIG. 2 shows panorama image shooting processing in the panorama shooting mode. 6 is a flowchart illustrating panorama image shooting processing in a panorama shooting mode.

  In the imaging apparatus 100, a panoramic shooting mode for recording a panoramic image can be selected by moving the imaging apparatus 100 from the left to the right while continuing to hold down SW2 by switching a mode change switch included in the operation unit 102. . In the present embodiment, the camera is moved from left to right. However, the image may be moved from right to left or moved up and down.

  FIG. 3 illustrates the operation of the imaging apparatus 100 in the panoramic shooting mode.

  In the panorama shooting mode, continuous shooting is performed while moving the imaging device 100 to the right from 301 to 30N as shown in FIG. Images captured at this time are P (0) to P (N). The panoramic image 301 can be finally recorded by aligning and continuing to combine the images P (n) and P (n−1) before and after the images continuously photographed.

  In the panorama shooting mode, in step S201, the control unit 101 confirms whether SW1 has been pressed. If not, the control unit 101 returns to step S201 again.

  When SW1 is pressed, in step S202, the control unit 101 performs an AF (auto focus) process, an AE (auto exposure) process, an AWB (auto white balance) process, and an EF (flash pre-flash) process. Instruct each block.

  In step S203, the control unit 101 confirms whether or not SW1 is being pressed. If not, the control unit 101 returns to step S201 again.

  In step S204, the control unit 101 confirms whether SW2 is pressed. If the switch is pressed, the control unit 101 returns to step S203.

  In step S205, the control unit 101 initializes a variable n indicating the number of times an image has been captured to 0 when capturing one panoramic image in the panorama capturing mode.

  In step S206, a photographing process is performed, and the control unit 101 stores the image P (n) obtained from the imaging unit 110 in the memory 104 after the image processing unit 111 performs the image processing.

  In step S207, whether n is not 0 is compared. If it is 0, the process proceeds to step S211; otherwise, the process proceeds to step S208. Here, assuming that n is 0, the process proceeds to step S211.

  In step S211, 1 is added to the variable n indicating the number of shots.

  In step S212, it is checked whether n has reached a predetermined number, and if it has reached, the process proceeds to step S213. Otherwise, the process returns to step S206. This predetermined number is the maximum number of times that an image can be taken when one panoramic image is taken, and is set in advance according to the recording size, memory capacity, and the like.

  In step S213, it is confirmed whether or not SW2 is continuously pressed. If it is pressed, the process proceeds to step S206. Otherwise, the process proceeds to step S214.

  Subsequently, the processing of step S208 to step S210 performed when n is 1 or more in step S207 will be described.

  In step S208, the image processing unit 111 performs alignment processing between the latest captured image P (n) stored in the memory 104 and the image P (n-1) captured one time before. .

  FIG. 4 shows an example of the alignment process. First, in the alignment process, 12 blocks B (n) [n = 0 to 11] on P (n) newly photographed are set. B (n) is an area on the end side opposite to the moving direction of the camera in panoramic photography, that is, in this embodiment, from left to right so that an area overlapping P (n−1) is included. Since the image is moved, the region on the right side of P (n) is set. Then, a search is performed on which position on each P (n-1) imaged previously (hereinafter referred to as a corresponding block position).

  As a method for determining the corresponding block position, a method in which the position where the difference in the high-frequency component of the pixel in the block region is the minimum is set as the corresponding block position, or the case where the sum of the absolute difference values of the pixels is the minimum is determined as the corresponding block position. The method to do is used.

  FIG. 4 (2) shows an ideal search result B ′ (n) for the corresponding block position. FIG. 4 (2) shows an ideal search result, and the positional relationship of each block is equally spaced, as is the positional relationship on P (n). However, in general, the subject is not in an ideal shape due to the subject being in motion during shooting or the influence of random noise of the imaging device.

  FIG. 4 (3) shows an actual search result B ″ (n) of the corresponding block position. In such a case, a position where the relative position is greatly shifted is excluded, and positioning is performed using only the bold blocks whose positional relationship is not shifted.

  In step S209, the number of blocks whose relative positional relationship is not shifted among the blocks searched in step S208 is stored in the memory 104 as the composite reliability. In other words, the composition reliability is used for positioning at the time of composition when the composition part between two images to be synthesized is subjected to pattern matching for each block to calculate the similarity, and the similarity is equal to or greater than a predetermined value. It is the number of blocks. It can be determined that as the number of blocks used for positioning in which the positional relationship is not deviated increases, the coupling is correctly performed. Therefore, the number of blocks whose positional relationship used for positioning at the time of composition is not shifted can be considered as a numerical value indicating how accurately the composition processing has been performed.

  In step S210, composition is performed based on the alignment result in step S208, and the result is stored in the memory 104 as P (n-1).

  If it is determined that the SW2 is turned off in S213, the image thus obtained and stored in the memory 104 is recorded on the external recording medium 141 as a JPEG image in step S214. Information on the composition reliability obtained in step S209 is also recorded as image information (attribute information) together with the image.

  As described above, panoramic images are taken.

  Next, the playback mode process will be described with reference to FIG.

  In step S <b> 501, the control unit 101 acquires image information of an image recorded on the recording medium 141 from the recording medium 141 and stores it in the memory 104. Then, the latest image is read from the recording medium 141 and displayed on the display unit 130, and the process proceeds to step S502.

  In step S502, the control unit 101 determines whether the album button of the operation unit 102 has been pressed. If it is determined that the album button has been pressed, the process proceeds to step S503. If it is determined that the album button has not been pressed, the process proceeds to step S507.

  In step S503, the control unit 101 displays an album selection screen such as 1301 in FIG. On the album selection screen, by selecting one of the four album selection icons of “event”, “date”, “person”, and “custom”, it is possible to select the conditions of the images to be reproduced in the album. When “Event” is selected, an image captured for several days is set as a selection condition for the album playback target image based on the shooting date of the image currently displayed on the display unit 130. When “Date” is selected, an image on the same shooting date as the image currently displayed on the display unit 130 is set as a selection condition for the album reproduction target image. When “person” is selected, a person is detected from the images currently displayed on the display unit 130, and an image including the same person as the detected person or an image including a related person is selected as an album playback target image. Condition. When “Custom” is selected, the user is allowed to select a date, an image, a person, and the like, and the selection condition for the album playback target image is determined according to the selected date, image, and person.

  Note that in this embodiment, the album target images that can be the target of the album are still images and moving images that are self-recorded by the imaging apparatus 100, and therefore the album target image is acquired in advance using the image information acquired in S501. It may be determined whether an image exists. When it is determined that the album target image does not exist, the album selection icon on the album selection screen may be grayed out. In addition, when the number of album target images having the same shooting date as the currently displayed image is smaller than a predetermined number, the “date” icon is grayed out, or a predetermined number of album target images having a person included in the currently displayed image are displayed. If the number is smaller, the “person” icon may be grayed out.

  In step S504, the control unit 101 uses the operation unit 102 to determine whether an album to be created is selected on the album selection screen. If an album is selected, the process proceeds to step S505, and if an album is not selected, the process proceeds to S506.

  In step S505, the control unit 101 creates an album and proceeds to step S506. The album creation process will be described with reference to FIG.

  In step S506, the control unit 101 determines whether “End” has been selected on the album selection screen, that is, whether or not to end the album mode. If the album mode is to be ended, the process proceeds to step S507, and if the album mode is not to be ended, the process proceeds to step S503.

  In step S507, the control unit 101 receives an operation for ending the playback mode from the user, and determines whether or not the playback mode is ended. When the playback mode ends, the playback mode ends, and when the playback mode does not end, the process proceeds to step S508.

  In step S508, the control unit 101 performs other reproduction processing instructed by the operation unit 102, and proceeds to step S502. Other processes include, for example, an image sending process for switching an image displayed on the display unit 130, a moving image reproduction process, an image erasing process, an image editing process, an image transmission process, and the like.

  As described above, the playback mode processing is performed.

  Next, the album creation process executed in step S505 will be described with reference to FIG.

  In step S <b> 601, the control unit 101 performs scenario creation processing using the album target image read into the memory 104 and records the created scenario in the memory 104. In the scenario creation process, first, still images and moving images used for the album moving image are selected. As explained before, when creating an album video with “Date”, the images taken on that date are selected, and when creating an album video with “People”, the person was taken as the subject. Select an image recorded in the image information of the image. When creating an album video by “event”, images taken for several days are selected based on the shooting date of the display image. When the user selects “custom” in creating the album moving image, the image selected by the user is selected.

  When the image selection is completed, the playback order (display order) of the images selected as the album is determined, and a scenario as shown in FIG. 15 is created. In the scenario, scene number 1501, file information 1502, chapter 1503, number of frames 1504, image size 1505, panorama image flag 1506, and the like are described in the order of reproduction for each image to be reproduced. A scene number 1501 indicates the playback order of images, and starts from 1. The file information 1502 describes information for specifying an image to be reproduced. In this embodiment, the file path of the image is recorded. Instead of the file path, a handle, identification ID, or the like that can identify the image may be recorded as the file information 1502. The chapter 1503 describes the number of the chapter to be played back when the playback target image is a moving image. It is not described for still images. As described above, for a moving image, the reproduction target is designated for each chapter instead of the entire moving image. The number of frames 1504 describes the number of frames used for reproduction. When a chapter of a moving image is designated as a playback target, the number of frames of that chapter is recorded as the number of frames 1504. In the case of a still image, 60 is set for a normal image, and 120 is set for a panoramic image. In this embodiment, the reproduction and recording of the album moving image is performed at a frame rate of 30 fps. That is, a normal still image is played back for 2 seconds, and a panoramic image is played back for 4 seconds. The vertical and horizontal size 1505 describes the vertical and horizontal sizes of the image to be reproduced. The panorama image flag 1506 describes information indicating whether the image to be reproduced is a panorama image. By creating such a scenario, it is possible to specify the playback order, the playback time, and the chapter to be played back for the moving image for the playback target image. In addition to these pieces of information, information on BGM settings and information on the number of frames in the entire scenario are also recorded as information on the entire scenario.

  In step S602, the control unit 101 causes the display unit 130 to display the album setting change menu 1305 in FIG. 13 (1305).

  In steps S603, S605, S608, and S610, the operation on the operation unit 102 in the album setting change menu is monitored, and one of album moving image creation 1306, BGM setting 1307, preview playback 1308, and end 1309 is selected. Judge that. Until any one is selected, the processes in steps S603, S605, S608, and S610 are repeated. If it is determined in step S603 that the BGM setting 1307 has been selected, the process proceeds to S604.

  In step S604, the control unit 101 performs BGM setting. A BGM to be combined with an album video is selected from a plurality of BGMs recorded in advance on the recording medium 141 of the imaging apparatus 100, or a voice synthesis rate when the audio of the video and the BGM are mixed is selected. The set contents are recorded in the scenario, and then the process returns to step S603. At this time, the attached information of the BGM setting is also recorded in the scenario. The BGM auxiliary information sets what kind of music the BGM has, for example, a slow music, a fast-tempo music, a music playback time, a voice synthesis rate, and the like.

  If it is determined in step S605 that the album moving image storage 1306 has been selected, the process proceeds to step S606.

  In step S606, album moving image storage is performed based on the scenario created in step S601. The album moving image saving process will be described with reference to FIG. When the album moving image storage is completed, the control unit 101 performs control so as to shift to the reproduction mode (step S607), and the album creation process is ended.

  If it is determined in step S608 that the album playback 1308 has been selected, the process proceeds to step S609. In step S609, the album moving image is reproduced based on the scenario.

  If it is determined in step S610 that the end 1309 has been selected, the control unit 101 performs control so as to transition to the playback mode (step S607), and ends the album creation process.

  Next, the album moving image saving process will be described with reference to FIG.

  First, in step S701, the control unit 101 newly creates a moving image file for the album moving image, and causes the recording / playback unit 140 to record the new moving image file on the recording medium 141.

  In step S <b> 702, the control unit 101 reads scenario information having a scene number of 1 from the scenario created in step S <b> 601 and recorded in the memory 104. Then, the processing of steps S703 to S706 is executed based on the read scenario information of the scene number.

  In step S703, the control unit 101 specifies an image file to be reproduced from file information described in the scenario information, and the image file to be reproduced is a JEPG (still image) file or an MP4 (moving image) file. Determine if there is. If the file is a JEPG file, the process proceeds to step S704. If not, the process proceeds to step S705 because it is a moving image MP4 file.

  In step S704, the control unit 101 executes processing for adding the data of the JPEG file to the album moving image file for the JPEG file to be reproduced specified by the file information described in the scenario information, and proceeds to step S706. . The process of adding the JPEG file data to the album moving image file will be described in detail with reference to FIG.

  In step S705, the control unit 101 executes a process of adding the data of the JPEG file to the album video file for the MP4 file to be reproduced specified by the file information described in the scenario information, and the process proceeds to step S706. . The process of adding the MP4 file data to the album moving image file will be described in detail with reference to FIG.

  In this embodiment, album moving image data is generated by sequentially adding image data (still image, moving image) designated as a reproduction target in the scenario to the album moving image file. However, it is also possible to generate moving image data from image data designated as playback targets in the scenario, and combine the generated moving images to create an album moving image file.

  In step S706, the control unit 101 determines whether there is scenario information of the next scene number in the scenario recorded in the memory 104. If it is determined that there is scenario information of the next scene number, the scenario information of the next scene number is read in step S707, and the processes of steps S703 to S705 are executed based on the read scenario information. The processes in steps S703 to S707 are repeated until there is no scenario information for the next scene number. If it is determined that there is no scenario information for the next scene number, the album moving image generation has been completed, and the process advances to step S708.

  In step S708, the control unit 101 determines whether to synthesize BGM according to the information regarding the BGM setting described in the scenario created in step S601 and recorded in the memory 104. If it is determined that BGM is to be combined, the process proceeds to step S709. If it is determined that BGM is not to be combined, the album moving image saving process is terminated.

  In step S709, a BGM synthesis process for synthesizing BGM data to the MP4 file of the album video that has been generated in step S708 is executed, and then the album video storage process ends. The BGM synthesis process will be described in detail with reference to FIG.

  Next, the process of adding the JPEG file data to the album moving image file in step S704 will be described with reference to FIG.

  In step S 801, the JPEG file specified from the file information 1502 is read from the recording medium 141 and temporarily stored in the memory 104.

  In step S802, the image data of the JPEG file held in the memory 104 is decoded into YUV data using the image processing unit 111 and held in the memory 104, and the process proceeds to step S803.

  In step S803, it is determined from the information of the panoramic image flag 1506 whether the image to be reproduced is a panoramic image. In this embodiment, the information of the panorama image flag 1506 is recorded in the scenario. However, without recording the information of 1506 in the scenario, it is determined whether the image is a panoramic image from the image information described in the JPEG file. May be. For example, when information indicating that the image is captured in the panorama shooting mode is described in the shooting mode information included in the image information, it may be determined that the image is a panoramic image. Further, it may be determined whether the image is a panoramic image according to the vertical and horizontal sizes of the image. If it is determined that the image is not a panoramic image, the process proceeds to step S804. If it is determined that the image is a panoramic image, the process proceeds to a step S811.

  In step S804, the YUV data held in the memory 104 in step S802 is trimmed or resized in accordance with the image size of the album moving image using the image processing unit 111 and held in the memory 104, and the process proceeds to step S805. In the present embodiment, the image size of the album moving image is set to full HD (1920 × 1080).

  In step S805, the YUV data that has been resized and stored in the memory 104 is subjected to an effect such as a color filter using the image processing unit 111, and is stored again in the memory 104, and the process proceeds to step S806.

  In step S 806, the YUV data that has been stored in the memory 104 with the effect is converted to H.264 using the image processing unit 111. H.264 is encoded to create moving image data.

  In step S807, sound data obtained by encoding silent PCM data into AAC is generated using the sound processing unit 121, and the process proceeds to step S808.

  In step S808, stream data composed of the moving image data generated by encoding in step S806 and the audio data generated by encoding in step S807 are added to the album moving image file created in the recording medium 141 in step S701. Here, the unit of writing to the recording medium 141 is H.264. It may be written for each H.264 frame number or stream data size, or for every predetermined number of frames and time.

  In step S809, it is determined whether the processing in steps S806 to S808 has been completed for the number of frames specified by the number of frames 1504. If the processing for the designated number of frames has not been completed, since the moving image for the period designated by the number of frames 1504 has not been generated, the processing of S806 to S808 is repeated. Steps S804 to S809 are processing for a still image, so 60 frames, that is, 2 seconds of moving images are generated and added to the album moving image file. When the processing for the designated number of frames is completed, in step S810, the file management information of the album moving image file to which the stream has been added is updated and recorded. Note that the management information is updated so that a moving image generated from one still image is configured as one chapter. In this way, the process of adding the moving image data generated from the still image specified in the scenario to the album moving image file is completed.

  On the other hand, if it is determined that the JPEG file to be played back is a panoramic image, a trimming method for converting a panoramic image (still image) into a panning video is determined in step S811. The determination method of the trimming method will be described later. By this process, the frame in which each frame of the panning video created from the panoramic image is trimmed and created, and the frame described in the scenario. It is decided whether to correct the number. Specifically, first, the aspect ratio of the trimming area when trimming a partial area from the panoramic image is determined. Then, the number of frames described in the scenario is corrected to determine whether to change the playback time of the panning video generated from the panorama image to be played back. Further, based on the determined aspect ratio and the number of frames, the vertical and horizontal sizes and the movement amount of the trimming area are determined. Since the moving amount of the trimming region is determined so as to move from one end portion in the long side direction of the panoramic image to the other end portion, a panning moving image can be generated from a plurality of trimmed images. Subsequent processing is performed according to the method determined in this processing.

  In step S812, for the YUV data held in the memory 104 in step S802, the area determined in step S811 is trimmed using the image processing unit 111, and further resized to fit within the image size for the album moving image. Then, the resized YUV data is held in the memory 104. Here, the album moving image generated from one panoramic image is 120 frames (4 seconds). Although the size of the trimming area is common to 120 frames, the position of the trimming area differs depending on each frame. Therefore, in step S812, trimming is performed at a position corresponding to the frame to be generated. Specifically, when generating the data of the first frame, the end region in the long side direction of the panoramic image is trimmed, and the position of the trimming region at the time of generating the previous frame is stored for the second and subsequent frames. The trimming area is moved by the movement amount determined in step S811. The moving direction of the trimming region is a direction in which the panoramic image moves from the long side end side to the other end side in the long side direction when the first frame is trimmed. That is, the movement direction when the right edge is trimmed in the first frame is from right to left, and the movement direction when the left edge is trimmed in the first frame is from left to right. The movement direction when the upper end is trimmed in the first frame is from top to bottom, and the movement direction when the lower end is trimmed in the first frame is from bottom to top. In this way, the position of the trimming area is sequentially moved for each frame to be generated. For this reason, a panning moving image is generated in which images in a region where the position in the panoramic image has moved are sequentially reproduced (sequentially displayed).

  In step S813, the YUV data that has been resized and stored in the memory 104 is subjected to an effect such as a color filter using the image processing unit 111, is stored again in the memory 104, and the process proceeds to step S815.

  In step S815, it is determined whether the aspect of the resized YUV data is 16: 9 which is the same as the aspect of the album moving image. In S812, since resizing is performed so as to fit within the size of the album moving image, it is actually determined whether the vertical and horizontal sizes of the resized YUV data are the same as the size of the album moving image. If they are the same, the process proceeds to step S816 without performing the process of step S815. If they are different, the process proceeds to step S815.

  In step S815, an extra image (black data) of an insufficient size is added and stored in the memory 104 so that the vertical and horizontal sizes of the resized YUV data are the same as the image size of the album moving image. Extra images are added so as to be equal to the left and right or up and down.

  In step S816, the effected YUV data held in the memory 104 (YUV data with an extra image added when step S815 is executed) is converted into H.264 data using the image processing unit 111. H.264 is encoded to create moving image data.

  In step S817, the silent PCM data is encoded into AAC using the voice processing unit 121 to generate voice data, and the process proceeds to step S818.

  In step S818, stream data composed of the moving image data generated by encoding in step S816 and the audio data generated by encoding in step S817 are added to the album moving image file created in the recording medium 141 in step S701. Here, the unit of writing to the recording medium 141 is H.264. It may be written for each H.264 frame number or stream data size, or for every predetermined number of frames and time.

  In step S819, it is determined whether the processing in steps S812 to S818 has been completed for the number of frames specified by the number of frames 1504. When the number of frames is changed in the process of step S811, it is determined whether the processes of steps S812 to S818 have been completed for the changed number of frames. If processing for the designated number of frames has not been completed, the processing of S812 to S818 is repeated until moving images for the designated period are generated. Since steps S812 to S818 are processing for a panoramic image, normally 120 frames, that is, 4 seconds of moving images are generated and added to the album moving image file. When the number of frames is corrected in the process of step S811, the number of frames is greater than 120, 150, or 180, and a moving image for 5 seconds or 6 seconds is generated. Thus, when the processing for the designated number of frames is completed, in step S810, the file management information of the album moving image file to which the stream has been added is updated and recorded. Note that the management information is updated so that a moving image generated from one still image is configured as one chapter. In this way, the process of adding the moving image data generated from the panoramic image specified in the scenario to the moving image file for album is completed.

Note that moving image data generated from one still image created by the processing of S806 to S808 or S812 to S818 is recorded so as to become one chapter of the album moving image.
Next, determination of a panoramic image trimming method will be described with reference to FIGS.

  First, with reference to FIG. 10, setting of the aspect ratio of the trimming area and generation of a panning video will be described using a horizontally long panoramic image 1001 as an example.

  The panoramic image 1001 is an image having a size of 4200 × 900. When the panorama image 1001 is trimmed in the trimming area having the same aspect ratio 16: 9 as that of the album moving image, the size of the trimming area 1011 is 1600 × 900. Here, when the panning video is generated, the trimming area is moved in the long side direction (horizontal direction) of the panoramic image for each frame, so that the image size in the short side direction (vertical direction) of the panoramic image remains as it is. Is the vertical size. Further, the horizontal size of the trimming area corresponding to the long side direction of the panoramic image can be calculated from the image size of the panoramic image in the short side direction and the aspect ratio of the trimming area. When trimming in the trimming area 1011 having an aspect ratio of 16: 9, the album moving image has the same aspect ratio of 16: 9. Therefore, the album moving image frame 1012 can be generated simply by resizing the trimmed image.

  When panorama image 1001 is trimmed in a trimming area having an aspect ratio of 20: 9, the size of trimming area 1021 is 2000 × 900. In this case, the aspect ratio of the album moving image and the aspect ratio of the trimming area 1021 are different. Therefore, the trimmed image (2000 × 900) is resized to generate a 1920 × 864 image 1023, and extra images are added to the top and bottom of the resized image 1023, respectively. Is generated.

  Further, when the panoramic image 1001 is trimmed in a trimming area having an aspect ratio of 30: 9, the size of the trimming area 1031 is 3000 × 900. In this case, the aspect ratio of the album moving image is different from that of the trimming area 1031. Therefore, the trimmed image (3000 × 900) is resized to generate a 1920 × 576 image 1033, and extra images are added to the top and bottom of the resized image 1033, respectively. Is generated.

  Here, if the trimming is performed with the same 16: 9 aspect ratio as the album image, a panning moving image can be generated without adding an extra image. However, in the present embodiment, the length of the panning video generated from the panoramic image is determined to be 4 seconds of 120 frames. Therefore, in panoramic images that have a very large size in the long side direction relative to the size in the short side direction, when generating a panning video that moves in the entire long side direction, the amount of movement of the trimming position between frames increases. A video with a fast panning speed is generated. Since the image size in the long side direction to be trimmed is larger in the aspect ratio of 20: 9 than in the aspect ratio of 16: 9, the amount of movement of the trimming position between frames is larger than in the case of 16: 9. It can be reduced and the panning speed can be reduced. Further, since the image size in the long side direction to be trimmed is larger at the aspect ratio of 30: 9 than at the aspect ratio of 20: 9, the amount of movement of the trimming position between frames is further reduced and panning is performed. Can reduce the speed.

  Therefore, in this embodiment, in order to adjust the panning speed, the ratio of the image size in the long side direction (horizontal direction) and the short side direction (vertical direction) of the panoramic image, or the image in the long side direction (horizontal direction). The aspect ratio of trimming when generating a panning video from a panoramic image is switched according to the size.

  The trimming method determination process in step S811 will be described with reference to FIG.

  In step S901, the control unit 101 determines whether the horizontal image size (long side length) of the panoramic image is less than the threshold value TH1. If it is less than the threshold value TH1, the process proceeds to step S906. If it is greater than or equal to TH1, the process proceeds to step S902. In the present embodiment, since the panorama image is horizontally long and the vertical image size of the panorama image is fixed at 1080, it is determined whether or not the horizontal image size on the long side is less than the threshold value TH1. However, if the vertical image size on the short side of the panoramic image is variable, the horizontal (long side) image size and the horizontal (short side) image size are acquired, and (horizontal image size) / (vertical image size) It is also possible to determine whether the image size is less than the threshold value TH10.

  In step S906, since the panoramic image is not so long in the long side direction, the control unit 101 sets the aspect ratio of the trimming area for generating the panning video to 16: 9, which is the same as the aspect ratio of the album video. decide.

  In step S902, the control unit 101 determines whether a person is recorded in the subject information included in the image information of the panoramic image. If a person is recorded, the process proceeds to step S906. If not recorded, the process proceeds to step S903. A panoramic image in which a person is recorded has a trimming area aspect ratio of 16: 9 because it is better to reproduce the image without adding an extra image even if the panning speed is increased.

  In step S903, the control unit 101 determines whether or not the BGM tempo is fast based on the BGM attached information described in the above-described scenario. If it is early, the process proceeds to step S906. Otherwise, the process proceeds to step S904. When the BGM tempo is fast, even if the panning speed is increased, the affinity with the BGM is increased and there is no need to reduce the panning speed, so the aspect ratio of the rimming area is set to 16: 9.

  In step S904, the control unit 101 determines whether the event information described in the above scenario is a sports scene. If sport is set, the process proceeds to step S906. Otherwise, the process proceeds to step S905.

  In step S905, the control unit 101 determines whether the horizontal image size (long side length) of the panoramic image is less than the threshold value TH2 (> TH1). If it is less than the threshold value TH2, the process proceeds to step S907, and if it is greater than or equal to the threshold value TH2, the process proceeds to step S908. Also in step S905, as in step S901, it may be determined whether (horizontal image size) / (vertical image size) is less than the threshold value TH20. Here, TH20> TH10.

  In step S907, since the panorama image is a panorama image that is long to some extent in the long side direction, the control unit 101 determines that the aspect ratio of the trimming area for generating the panning moving image is 20: 9. In step S908, since the panoramic image is a panoramic image that is very long in the long side direction, the control unit 101 determines that the aspect ratio of the trimming area for generating the panning moving image is 30: 9.

  When it is determined in step S908 that the aspect ratio is 30: 9, it is further determined whether to change the number of frames described in the scenario, and processing for changing the number of frames is executed in steps S911 to S915. First, in step S911, the control unit 101 determines whether the horizontal image size (long side length) of the panoramic image is greater than or equal to a threshold value TH3 (> TH2). If it is greater than or equal to the threshold value TH3, the process proceeds to step S912, and if it is less than the threshold value TH3, the process proceeds to step S909. Also in step S911, as in steps S901 and S905, it may be determined whether (horizontal image size) / (vertical image size) is greater than or equal to a threshold value TH30. Here, TH30> TH20. In step S912, the control unit 101 determines whether the horizontal image size (long side length) of the panoramic image is greater than or equal to a threshold value TH4 (> TH3). If it is equal to or greater than the threshold TH4, the process proceeds to step S914, and in step S914, the control unit 101 changes the number of frames from 120 to 180 and describes it in the scenario. If it is less than the threshold TH4, the process proceeds to step S913, and in step S913, the control unit 101 changes the number of frames from 120 to 150 and describes it in the scenario. Also in step S912, as in steps S901, S905, and S911, it may be determined whether (horizontal image size) / (vertical image size) is greater than or equal to a threshold value TH40. Here, TH40> TH50. By such processing, when the horizontal image size (long side length) is less than TH3, the number of frames is kept at 120, and when it is greater than TH3 and less than TH4, the number of frames is changed to 150, In the case of TH4 or more, the number of frames is changed to 180. When the horizontal image size (long side length) is greater than TH3, it is possible to make the horizontal image size larger than the 30: 9 aspect ratio of trimming. If this is increased, the area to which the extra image (black data) is added in the generated moving image becomes large, and it becomes difficult to confirm the image. Therefore, in the present embodiment, the panning speed is reduced by increasing the number of frames of the generated moving image without increasing the horizontal image size further than 30: 9.

  Furthermore, when the number of frames is changed to 180 in step S914, in step S915, the control unit 101 further determines whether the horizontal image size (long side length) of the panoramic image is greater than or equal to the threshold value TH5 (> TH4). To do. If it is greater than or equal to the threshold value TH5, the process proceeds to step S916. Also in step S915, as in steps S901, S905, S911, and S912, it may be determined whether (horizontal image size) / (vertical image size) is greater than or equal to a threshold value TH50. Here, TH50> TH40. If the horizontal image size (long side length) is the threshold value TH5, the process proceeds to step S916, and if it is less than TH5, the process proceeds to step S909.

  If the horizontal image size (long side length) of the panoramic image is less than the threshold TH5, the process proceeds to step S909 regardless of the determined trimming aspect ratio and the number of frames.

  In step S909, the control unit 101 determines the size of the trimming area from the aspect ratio of the trimming area determined in steps S906 to S908 and the image size of the panoramic image.

  In step S910, the control unit 101 determines the trimming area from one end in the long side direction of the panoramic image to the other end based on the determined size of the trimming area and the number of frames 1504 described in the scenario. The amount of movement when moving sequentially to the part is determined. When the number of frames is changed in steps S913 and S914, the determination is made based on the changed number of frames. In S910, the amount of movement of the trimming area is determined so that the entire panoramic image is panned by the designated number of frames.

  In step S916, the control unit 101 determines the size of the trimming area as in step S909.

  In step S <b> 917, the control unit 101 determines the movement amount of the trimming area as a fixed value M. In this embodiment, M = TH5 / 180. That is, M is the amount of movement when panning the entire panoramic image with 180 frames when the horizontal image size (long side length) of the panoramic image is TH5. In step S917, since the horizontal image size (long side length) of the panoramic image is greater than or equal to the threshold value TH5, when the horizontal image size is larger than TH5, the entire panoramic image is panned with the movement amount M. I can't. However, if the amount of movement is larger than M, the panning speed increases and it becomes difficult to confirm the image. Therefore, the amount of movement is fixed to M and only a part of the panoramic image is panned. When panning only a part of the panoramic image, the panning start position may be determined in order to designate a panning area in the panoramic image.

  In this way, the vertical and horizontal sizes of the trimming area and the amount of movement of the trimming area between frames are determined. In the present embodiment, the vertical and horizontal sizes of the trimming area and the amount of movement between frames are determined, but the position of the trimming area may be determined individually for each frame. As described above, the size of the area to be trimmed and the movement amount of the trimming area for each frame are determined from the vertical and horizontal sizes of the panoramic image and the information set in the scenario.

  In the present embodiment, the normal number of frames when creating a panning video from a panoramic image is predetermined as 120 frames (4 seconds). For this reason, if the aspect ratio of the trimming region is set to be the same for panoramic images having different long side sizes, the amount of movement when the trimming region is moved between frames increases. As a result, a panoramic image that is long in the long side direction is reproduced as if panned at high speed during reproduction. Therefore, in the present invention, the aspect ratio of the trimming region is appropriately changed by the image size in the long side direction or the ratio of the image size in the short side direction to the image size in the long side direction (aspect ratio), and the panning speed is increased. It is adjusting. On the other hand, when BGM with a fast tempo is set or when the event is a sport, the entire album moving image is configured as a speedy moving image with a lively feeling. For this reason, it is considered appropriate to incorporate a panning video that is created by performing trimming to increase the playback speed as a video provided to the user. Furthermore, if the image size in the long side direction, or the ratio of the image size in the short side direction to the image size in the long side direction (aspect ratio) increases, the frame for creating a movie will not only change the aspect ratio. The number is increasing. This prevents a playback area of the panoramic image from becoming small in the generated moving image.

  In FIG. 9, the trimming method for a horizontally long panoramic image has been described. Also in the case of a vertically long panoramic image, the trimming method (the size of the trimming area and the amount of movement of the trimming area) can be determined by the flowchart of FIG. However, in the case of a vertically long panoramic image, the determination method in steps S901, S905, S911, S912, and S915, and the aspect ratio set in steps S906, S907, and S908 are different from those of a horizontally long panoramic image.

  Differences from the trimming method determination process for a horizontally long panoramic image when the trimming method is determined for a horizontally long panoramic image will be described below. Processing other than that described below is the same as the trimming method determination processing for a horizontally long panoramic image.

  In step S901, the control unit 101 determines whether the image size in the vertical direction (long side) of the panoramic image is less than the threshold value TH6. If it is less than the threshold TH6, the process proceeds to step S908. If it is not less than the threshold TH6, the process proceeds to step S902.

  In step S905, the control unit 101 determines whether the image size in the vertical direction (long side) of the panoramic image is less than a threshold value TH7 (> TH6). If it is less than the threshold value TH7, the process proceeds to step S908. If it is not less than the threshold value TH7, the process proceeds to step S902.

  Similarly, in step S911, the control unit 101 determines that the image size in the vertical direction (long side) of the panoramic image is greater than or equal to the threshold value TH8 (> TH7) or greater than or equal to the threshold value TH9 (> TH8) in step S912. In step S915, it is determined whether or not the threshold value is TH10 (TH9) or more.

  In steps S901 and S905, the determination may be made by comparing the aspect ratio with the threshold value without comparing the image size of the long side with the threshold value. That is, in S901, it is determined whether (vertical image size / horizontal image size) is less than the threshold value TH60. In S905, (vertical image size / horizontal image size) is less than the threshold value TH70 (> TH60). May be determined. In S911, (vertical image size / horizontal image size) is greater than or equal to threshold TH80 (TH80> TH70), is greater than or equal to threshold TH90 (> TH80) in S912, or is greater than threshold TH100 (> TH90 in S912). ) It may be determined whether or not.

  In step S906, since the panoramic image is a panoramic image that is not so long in the long side direction, the control unit 101 sets the aspect ratio of the trimming area to 16: 9, which is the same as the aspect ratio of the album moving image.

  In step S907, since the panoramic image is a panoramic image that is long in the long side direction to some extent, the control unit 101 sets the aspect ratio of the trimming area for generating the panning moving image to 4: 3. In step S908, since the panoramic image is a panoramic image that is very long in the long side direction, the control unit 101 sets the aspect ratio of the trimming area for generating the panning moving image to 1: 1.

  Here, with reference to FIG. 14, setting of the aspect ratio of the trimming region and generation of a panning video will be described using a vertically long panoramic image 1401 as an example.

  The panoramic image 1401 is a panoramic image having a size of 1600 × 3600 and long in the vertical direction. When the panoramic image 1401 is trimmed in the trimming area having the aspect ratio of 16: 9, which is the same as the album moving image, the size of the trimming area 1411 is 1600 × 900. Here, when the panning video is generated, the trimming area is moved in the long side direction (vertical direction) of the panoramic image for each frame, so that the image size in the short side direction (horizontal direction) of the panoramic image remains as it is. The horizontal size of The size of the vertical trimming region corresponding to the long side direction of the panoramic image can be calculated from the image size in the short side direction (horizontal direction) of the panoramic image and the aspect ratio of the trimming region. When trimming in the trimming area 1411 having an aspect ratio of 16: 9, the album moving image has the same aspect ratio of 16: 9. Therefore, the album moving image frame 1412 can be generated simply by resizing the trimmed image.

  When the panorama image 1401 is trimmed in a trimming area having an aspect ratio of 4: 3, the size of the trimming area 1421 is 1600 × 1200. In this case, the aspect ratio of the album moving image and the aspect ratio of the trimming area 1421 are different. Therefore, the trimmed image (1600 × 1200) is resized to generate a 1440 × 1080 image 1423, and extra images are added to the left and right of the resized image 1423, respectively, so that the frame 1422 of the 1920 × 1080 album movie is obtained. Is generated.

  Further, when the panorama image 1401 is trimmed in a trimming area having an aspect ratio of 1: 1, the size of the trimming area 1431 is 1600 × 1600. In this case, the aspect ratio of the album moving image and the aspect ratio of the trimming area 1431 are different. Therefore, the trimmed image (1600 × 1600) is resized to generate a 1080 × 1080 image 1033, and extra images are added to the left and right of the resized image 1433, respectively, so that the frame 1432 of the 1920 × 1080 album movie is added. Is generated.

  Here, if the trimming is performed with the same 16: 9 aspect ratio as the album image, a panning moving image can be generated without adding an extra image. Since the aspect ratio of 4: 3 is larger than the aspect ratio of 16: 9, the image size in the long side direction (vertical direction) to be trimmed is larger, so the trimming position between frames is larger than that of 16: 9. The amount of movement can be reduced, and the panning speed can be reduced. Furthermore, since the aspect ratio of 1: 1 is larger than the aspect ratio of 4: 3, the image size in the long side direction (vertical direction) to be trimmed is further increased, and therefore the amount of movement of the trimming position between frames is further increased. It can be reduced and the panning speed can be reduced.

  As described above, a trimming method can be determined for a vertically long panoramic image as well as a horizontally long panoramic image.

  In the present embodiment, the image size of the generated album movie is set to the full HD (1920 × 1080) size, but may be configured to be able to be changed to another size by the user's operation on the operation unit 102. . In that case, the aspect ratio set in step S906 is made the same as the aspect ratio of the album moving image of the changed size. In step S907, the aspect ratio is set so that the ratio in the long side direction of the panoramic image is larger than the aspect ratio of the album moving image. In step S907, the aspect ratio is set so that the ratio in the long side direction is further increased.

  Next, the process of adding the MP4 file data in step S705 to the album moving image file will be described with reference to FIG.

  In step S <b> 1101, the control unit 101 reads out the chapter data of the chapter number specified by the chapter number 1503 from the MP4 file specified from the file information 1502 from the recording medium 141 and temporarily stores it in the memory 104. Here, in step S1101, the data of the specified chapter number is read in a specific unit (GOP unit or frame unit) in order from the top.

  In step S1102, among the chapter data held in the memory 104 in step S1101, H. The image (moving image) data in the H.264 format is decoded into YUV data using the image processing unit 111, and the process proceeds to step S1103.

  In step S1103, the control unit 101 determines whether voice data is included in the chapter data held in the memory 104 in step S1101. If it is determined that there is audio data, the process proceeds to S1104. If it is determined that there is no audio data, the process proceeds to S1105.

  In step S1104, the audio data in the AAC format held in the memory 104 is decoded into audio data in the PCM format using the audio processing unit 121, and the process proceeds to step S1106.

  On the other hand, in step S1105, silent PCM format audio data is stored in the memory 104 using the audio processing unit 121, and the process proceeds to step S1106.

  In step S1106, image processing effects are applied to the YUV data held in the memory 104 by the image processing unit 111 in the same manner as in steps S805 and S813. If the vertical and horizontal sizes of the YUV data are different from the vertical and horizontal sizes of the album moving image, the resizing process is performed so that the vertical and horizontal sizes of the album moving image are obtained. In step S1107, the YUV data applied in step S1106 is encoded by the image processing unit 111 as in steps S806 and S816, and the H.264 data is encoded. H.264 format image (moving image) data is converted.

  In step S1108, the PCM data created in steps S1104 and S1105 is encoded into AAC using the audio processing unit 121, and the process proceeds to step S1109. At this time, by performing encoding at a bit rate larger than the bit rate of BGM data (AAC data) recorded in the non-volatile memory 105, it is possible to improve the sound quality of the MP4 file recorded on the recording medium 141. is there. The BGM data to be recorded in the non-volatile memory 105 with a capacity restriction can be stored in a plurality of BGM data by reducing the capacity by suppressing the bit rate.

  In step S1109, as in step S808, the stream data composed of the image data and audio data generated in steps S1107 and S1108 is added to the album moving image file created on the recording medium 141 in step S701. In step S1110, the control unit 101 determines whether the processing in steps S1101 to S1108 has been performed for the number of frames specified by the number of frames 1504. If it is determined that the processing for the designated number of frames has been completed, the process proceeds to step S1111. If it is determined that the process has not been completed, the process returns to step S1101, and the process is repeated until the process for the designated number of frames is completed.

  In step S1111, the file management information of the album moving image file to which the stream has been added is updated and recorded. Note that the management information is updated so that the moving image generated from the chapter specified in the scenario is configured as one chapter. In this manner, the album moving image data generated from the moving image chapter specified in the scenario is added to the album moving image file.

  Next, the process of combining BGM with the MP4 file of the album moving image in step S709 will be described using FIG.

  In step S1201, the WAV file selected as the composition BGM data is read from the recording medium 141 and stored in the memory 104, and the process proceeds to step S1202.

  In step S1202, an MP4 file for BGM synthesis is newly created and recorded on the recording medium 141.

  In step S1203, the MP4 file data of the album moving image is read from the recording medium 141 and stored in the memory 104. Here, the MP4 file data is read in order from the top in a predetermined unit (GOP unit, frame unit, time unit).

  In step S1204, the H.264 of the MP4 file held in the memory 104 in step S1203. H.264 format image data is decoded into YUV data. In step S1205, the AAC format audio data of the MP4 file held in the memory 104 in step S1203 is decoded and converted to PCM format audio data.

  In step S1206, the audio processing unit 121 is used to mix the PCM data of the WAV file held in the memory 104 in step S1201 and the PCM data decoded in step S1205. Here, the mixing ratio is set according to one of the synthesis rates 0%, 50%, and 100% described in the scenario, and the process proceeds to step S1207.

  In step S1207, the YUV data generated in step S1204 is obtained. H. Encode to H.264 data. In step S1208, the PCM data mixed with the BGM in step S1206 is encoded into ACC. In step S1209, the H.P. The H.264 format moving image data and the AAC format audio data obtained in step S1208 are recorded in the MP4 file for GM synthesis created in step S1202. If stream data has already been recorded, it is appended. In step S1201, the control unit 101 determines whether the processing in steps S1203 to S1209 has been completed for the data of all frames of the MP4 file of the album moving image for which BGM composition is specified. The processes in steps S1203 to S1209 are repeated until it is determined that the process has been completed. If it is determined that the process has been completed, management information of the MP4 file for BGM synthesis is recorded in step S1211. In step S <b> 1212, the MP4 file before BGM composition that has been designated as the BGM composition target is deleted from the recording medium 141.

  In this way, the BGM synthesis process is executed.

  Next, the preview reproduction in step S609 will be described.

  In the preview reproduction process, an album moving image is created in the same manner as the album moving image storage process in step S606. However, in the preview reproduction process, the generated album moving image is not recorded on the recording medium 141 but is converted to display data by the display control unit and displayed on the display unit 130. Since recording on the recording medium is not performed, the Y.V. Instead of encoding to H.264, it may be immediately converted into display data and displayed on the display unit 130.

  As described above, in the imaging apparatus 100 according to the present embodiment, when an album moving image is generated, recorded, or displayed (reproduced), a panning moving image is generated when a panoramic image is included as a reproduction target. When a panning video is generated from a panoramic image, a trimming method is determined according to the image size or aspect ratio in the long side direction of the panoramic image, the BGM tempo, and the shooting scene. Therefore, an appropriate moving image can be generated from the panoramic image.

  In this embodiment, the album moving image is generated and stored. However, the same scenario moving image may be reproduced next time by storing the generated scenario in the recording medium 141. In that case, the aspect ratio output at the time of reproduction can be changed. Therefore, for example, when outputting an album video to an external display device having a different aspect ratio, an appropriate album video can be generated according to the display size of the external display device or the size of the display area for displaying an image. it can. In this case, in the process described above, the process is executed using the display size instead of the image size of the album moving image.

<Other embodiments>
Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined. Also, when a software program that realizes the functions of the above-described embodiments is supplied from a recording medium directly to a system or apparatus having a computer that can execute the program using wired / wireless communication, and the program is executed Are also included in the present invention. Accordingly, the program code itself supplied and installed in the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included in the present invention. In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS. As a recording medium for supplying the program, for example, a magnetic recording medium such as a hard disk or a magnetic tape, an optical / magneto-optical storage medium, or a nonvolatile semiconductor memory may be used. As a program supply method, a computer program that forms the present invention is stored in a server on a computer network, and a connected client computer downloads and programs the computer program.

Claims (19)

An acquisition means for acquiring a panoramic image;
A moving image generating means for generating a moving image based on the panoramic image, wherein a plurality of trimmed images trimmed by sequentially moving and trimming the position of a trimming area when trimming the panoramic image and changing the position Video generation means for generating a video for a predetermined time to be played,
The moving image generation means trims the panoramic image with a predetermined aspect ratio or trims the panoramic image with an aspect ratio different from the predetermined aspect ratio according to the size or aspect ratio of the panoramic image in a predetermined direction. In addition, when trimming with an aspect ratio different from the predetermined aspect ratio, a video for a time longer than the predetermined time is generated according to the aspect ratio of the panoramic image or the size in a predetermined direction. An image processing apparatus.   The image processing apparatus according to claim 1, wherein the predetermined aspect ratio is an aspect ratio of a moving image generated by the moving image generation unit.   3. The image processing apparatus according to claim 1, wherein the moving image generation unit sequentially moves the position of the trimming area without changing the aspect ratio of the trimming area when the panoramic image is trimmed. 4. .   The image processing apparatus according to claim 3, wherein the moving image generation unit determines a movement amount of the position of the trimming area according to an image size of the panoramic image and an aspect ratio of the trimming area.   The moving image generation means determines the amount of movement of the position of the trimming area according to the image size of the panoramic image, the aspect ratio of the trimming area, and the length of the predetermined time. The image processing apparatus described.   The moving image generating means, when trimming the panoramic image, moves the position of the trimming region from one end portion in the long side direction of the panoramic image to the other end portion in the long side direction. The image processing apparatus according to claim 1.   The moving image generating means generates a moving image by trimming at the predetermined aspect ratio when the long side size of the panoramic image or the long side size / short side size is smaller than a predetermined value, When the size of the long side of the panoramic image or the size of the long side / the size of the short side is larger than the predetermined value, the side corresponding to the long side of the panoramic image is larger than the predetermined aspect ratio. The image processing apparatus according to any one of claims 1 to 6, wherein a moving image is generated by trimming at an aspect ratio that increases in size and adding predetermined data to the trimmed image.   The moving image generating means generates a moving image for a time determined according to a size or an aspect ratio of the panoramic image in a predetermined direction when trimming with an aspect ratio different from the predetermined aspect ratio. The image processing apparatus according to claim 1.   The moving image generation unit generates a moving image for a time longer than the predetermined time when the size of the long side of the panoramic image or the size of the long side / the size of the short side is larger than a predetermined value. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. An acquisition means for acquiring a panoramic image;
Display control means for trimming a partial area of the panoramic image and controlling the trimmed image to be displayed. In order to display a single panoramic image for a predetermined period, Display control means for controlling the trimming by sequentially moving the position, trimming, and trimming the trimmed image by changing the position within the predetermined period;
The display control means trims the panoramic image with a predetermined aspect ratio or trims the panoramic image with an aspect ratio different from the predetermined aspect ratio according to the size or aspect ratio of the panoramic image in a predetermined direction. In addition, when trimming with an aspect ratio different from the predetermined aspect ratio, a period for displaying one panoramic image is longer than the predetermined period according to the size or aspect ratio of the panoramic image in a predetermined direction. An image processing apparatus having a long period.   The image processing apparatus according to claim 10, wherein the predetermined aspect ratio is an aspect ratio corresponding to a display size when the panoramic image is displayed.   12. The display control unit according to claim 10, wherein when the panoramic image is trimmed and displayed, the display control unit sequentially moves the position of the trimming area without changing the aspect ratio of the trimming area. Image processing device.   The image processing apparatus according to claim 12, wherein the display control unit determines a movement amount of a position of the trimming area according to an image size of the panoramic image and an aspect ratio of the trimming area.   11. The display control means determines the amount of movement of the position of the trimming area according to the image size of the panoramic image, the aspect ratio of the trimming area, and the length of the predetermined period. 14. The image processing device according to any one of items 13.   The display control means, when trimming the panoramic image, moves the position of the trimming region from one end portion in the long side direction of the panoramic image to the other end portion in the long side direction. The image processing apparatus according to any one of claims 10 to 14.   The display control means trims and displays the panorama image with the predetermined aspect ratio when the size of the long side of the panoramic image or the size of the long side / the size of the short side is smaller than a predetermined value. When the size of the long side of the image or the size of the long side / the size of the short side is larger than the predetermined value, the size of the side corresponding to the long side of the panoramic image is larger than the predetermined aspect ratio. 16. The method according to any one of claims 10 to 15, wherein trimming is performed with an aspect ratio and control is performed to display the image with the predetermined aspect ratio generated by adding predetermined data to the trimmed image. The image processing apparatus described.   The display control means determines a predetermined period for displaying one panoramic image according to the size or aspect ratio of the panoramic image in a predetermined direction when trimming with an aspect ratio different from the predetermined aspect ratio. The image processing apparatus according to claim 10, wherein:   The display control means has a predetermined period for displaying one panoramic image when the size of the long side of the panoramic image or the size of the long side / the size of the short side is larger than a predetermined value. The image processing apparatus according to claim 10, wherein the image processing apparatus is longer than a predetermined period.   A program for causing a computer to function as each unit of the image processing apparatus according to claim 1.

Images