JP2011146846A - Image reproduction controller, image reproduction control method, and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology capable of reproducing a proper moving image in accordance with the purpose of a reproduction, with a plurality of pieces of moving image data picked up in different frame rates. <P>SOLUTION: A file dividing part 58 receives a file including a stream of a normal frame rate, a stream of a frame rate higher than the normal one about at least a partial scene included in the stream of the normal frame rate, and related information that specifies the scene, and performs division into the stream of the normal frame rate, the stream of the frame rate higher than the normal one, and the related information. An image decoding part 60 decodes at least one of the stream of the normal frame rate and the stream of the frame rate higher than the normal one, to generate a frame for a moving image. Based on the related information, a summary generating part 76 generates, as a summary of a moving image, a frame of the moving image about a scene that is commonly included by the stream of the normal frame rate and the stream of the frame rate higher than the normal one. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image generation control device that decodes and reproduces a plurality of encoded moving images, and an imaging device equipped with the image generation control device.

  In recent years, digital movie cameras that allow general users to easily shoot moving images as well as still images have become widespread. Among them, the frame rate can be changed during shooting, and movies with different frame rates can be recorded. There are some that can be imaged simultaneously (see Patent Document 1). These moving images are recorded on a recording medium or the like and then viewed on, for example, a display monitor, a PC (Personal Computer), a television, or the like mounted on a movie camera.

JP 2007-195050 A

  If the same scene can be shot at different frame rates, for example, important scenes can be shot at a high frame rate with high time resolution. On the other hand, there will be a plurality of moving images of the same scene at the same time, and the user who has captured the moving image may be wondering which moving image to view.

  The present invention has been made in view of such a situation, and an object thereof is to provide a technique capable of reproducing an appropriate moving image from among a plurality of moving image data captured at different frame rates.

  One embodiment of the present invention is an image recording control apparatus. The device includes a normal frame rate stream, a higher frame rate stream for at least some scenes included in the normal frame rate stream, and a file including related information for identifying the scene. A normal frame rate stream, a higher frame rate stream than normal, and a file division unit that divides the stream into related information, the normal frame rate stream, and a higher frame rate stream than normal An image decoding unit that decodes at least one of the streams to generate a moving image frame, and based on the related information, the normal frame rate stream and the higher frame rate stream are commonly used. Videos about the scene including And a summary generator for generating a frame as a summary of the video.

  Another embodiment of the present invention is an imaging apparatus. This apparatus encodes a moving image captured by an imaging unit that acquires a moving image and a moving image captured by the imaging unit, and generates two types of streams, a normal frame rate stream and a higher frame rate stream. An image encoding unit that can be generated, a control unit that generates related information associating the stream of the normal frame rate with the stream of the frame rate higher than the normal generated in parallel with the generation of the stream; An image reproduction control device, and a display device controlled by the image reproduction control device.

  Yet another embodiment of the present invention is an image recording control method. This method relates to the same scene based on related information for identifying the same scene when a stream with a normal frame rate related to the same scene and a stream with a frame rate higher than normal exist simultaneously. Play only the stream.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, and the like are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, a suitable moving image can be reproduced | regenerated among several moving image data imaged with a different frame rate.

1 is a diagram schematically illustrating an internal configuration of an imaging apparatus according to a first embodiment. 3 is a flowchart for explaining a processing flow of the imaging apparatus according to the first embodiment; The normal frame rate frame group to be encoded by the normal frame rate frame encoding unit according to the first embodiment, the first high frame rate frame group to be encoded by the high frame rate frame encoding unit, and the first frame group It is the figure which represented typically an example of the relationship with 2 high frame rate frame groups. It is a figure which shows an example of the positional relationship between a chapter and a frame when the stream in which the chapter concerning Embodiment 1 is expanded is expanded. It is the figure which represented typically an example of the structure of the file which the file connection part concerning Embodiment 1 produces | generates. It is the figure which represented typically the other example of the structure of the file which the file connection part concerning Embodiment 1 produces | generates. It is the figure which represented typically an example of the recording format of the relevant information stored in the file concerning Embodiment 1. FIG. It is the figure which represented typically the other example of the recording format of the relevant information stored in the file concerning Embodiment 1. FIG. FIG. 6 is a diagram schematically illustrating an internal configuration of an image reproduction control apparatus according to a second embodiment. FIG. 10 is a diagram illustrating a temporal relationship between a frame group of 60 fps included in a normal frame rate stream and a frame group of 240 fps included in a high frame rate stream according to the second embodiment. FIG. 6 is a diagram schematically illustrating an internal configuration of an image reproduction control apparatus according to a third embodiment. In the example of FIG. 10, the relationship between the 240 fps synthesized frame group generated by the frame synthesizing unit according to the second embodiment and the 60 fps synthesized frame group generated by the frame synthesizing unit according to the third embodiment. FIG. FIG. 10 is a diagram illustrating another example of switching timing between a 60 fps frame group and a 240 fps frame group when reproducing a 60 fps frame group and a 240 fps frame group according to the fourth embodiment; . FIG. 10 is a diagram illustrating a state in which a 240 fps frame group included in a high frame rate stream according to the fourth embodiment and a 60 fps frame group corresponding to a 240 fps frame group included in the high frame rate stream are divided.

(Embodiment 1)
An outline of the first embodiment of the present invention will be described. In the first embodiment, a moving image frame is generated at a frame rate of 240 fps (frames per second) and stored in a frame buffer. In a normal shooting mode, the frame is thinned out until the frame rate reaches 60 fps, and then the stream is extracted. Generate. While there is an instruction to perform high-speed shooting from the user, in addition to generating a stream at a frame rate of 60 fps, a stream at a frame rate of 240 fps before thinning out the frames is also generated. After shooting, a plurality of generated streams having different frame rates are concatenated and saved as a single file.

  FIG. 1 is a diagram schematically illustrating an internal configuration of the imaging apparatus 200 according to the first embodiment. The imaging device 200 includes a sensor 10, an operation unit 12, a frame generation unit 14, and an image recording control device 100. The image recording control apparatus 100 further includes a frame buffer 16, a frame thinning unit 18, an image encoding unit 20, a control unit 22, a file connection unit 32, and a recording unit 34.

  The frame generation unit 14 generates a video frame acquired by the sensor 10. The sensor 10 is a solid-state imaging device (not shown) such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The frame generation unit 14 outputs analog three primary color signals R, G, and R output from the sensor 10. B is processed and converted into frames of digital luminance signal Y and color difference signals Cr and Cb. The frame buffer 16 temporarily stores the frame generated by the frame generation unit 14.

  The frame thinning unit 18 thins out the frames stored in the frame buffer 16. The frame buffer 16 stores a frame generated by the frame generation unit 14 and having a frame rate of 240 fps. The frame thinning unit 18 thins the frame to a quarter, thereby obtaining a frame having a frame rate of 60 fps. Generate. Here, there may be various combinations of the frame rate of the frame generated by the frame generation unit 14 and the frame rate of the frame generated by the frame thinning unit 18. For example, the frame generation unit 14 may generate a 600 fps frame, and the frame thinning unit 18 may freely change the number of thinnings.

  The image encoding unit 20 further includes two image encoding units, a normal frame rate frame encoding unit 36 and a high frame rate frame encoding unit 38.

  The normal frame rate frame encoding unit 36 encodes the frame generated by the frame thinning unit 18 to generate the normal frame rate stream 24. The normal frame rate frame encoding unit 36 encodes the frame according to a predetermined standard. For the coding of moving images, the ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission) standardized MPEG (Moving Picture Experts Group) series (MPEG-1, MPEG-2 and MPEG-4), an H.264 standardized by ITU-T (International Telecommunication Union-Telecommunication Standardization Sector) which is an international standard organization for telecommunications. 26x series standards (H.261, H.262 and H.263), or H.264, the latest video compression coding standard standardized jointly by both standards organizations. H.264 / AVC (a formal recommendation name in both organizations is MPEG-4 Part 10: Advanced Video Coding and H.264, respectively) H.264 / AVC Annex G, ie H.264. H.264 / SVC (Scalable Video Coding) is used.

  The high frame rate frame encoding unit 38 encodes the frames stored in the frame buffer 16 and generates a stream while receiving a user instruction input to the operation unit 12. Therefore, the stream generated by the high frame rate frame encoding unit 38 is a stream having a higher frame rate (240 fps) than the stream generated by the normal frame rate frame encoding unit 36. Hereinafter, an operation in which a high frame rate frame encoding unit 38 directly encodes a frame stored in the frame buffer 16 to generate a stream in accordance with a user instruction is referred to as a “high-speed shooting mode”. Similar to the normal frame rate frame encoding unit 36, the high frame rate frame encoding unit 38 is also MPEG-2, MPEG-4, H.264, or the like. H.264 / AVC, H.H. Compression encoding is performed according to a standard such as H.264 / SVC.

  The normal frame rate frame encoding unit 36 encodes a series of frames generated by the frame thinning unit 18 to generate one stream (normal frame rate stream 24), whereas the high frame rate frame encoding unit 38 Every time there is an instruction of the high-speed shooting mode from the user, the encoding is performed during the period of the instruction. For example, when the user instructs the first high-speed shooting mode, the high frame rate frame encoding unit 38 generates the first high frame rate stream 26. Thereafter, the high frame rate frame encoding unit 38 generates a stream every time the user gives an instruction for the high-speed shooting mode, and generates the n-th high frame rate stream 28 by the n-th instruction.

  The stream generated by the high frame rate frame encoding unit 38 has a high frame rate and can shoot video with high temporal resolution, but requires a large recording capacity. On the other hand, the normal frame rate stream 24 generated by the normal frame rate frame encoding unit 36 has a reduced temporal resolution because frames are thinned, but the storage capacity is reduced accordingly. Utilizing these properties, the entire video is recorded with the normal frame rate stream 24. For example, important scenes are shot in the high-speed shooting mode, so that important scenes are recorded with high temporal resolution while suppressing storage capacity. It becomes possible to do.

  The control unit 22 generates the related information 30 based on a user instruction input to the operation unit 12. Here, “related information” associates the streams generated by both when the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38 encode the video input to the sensor 10. Information. For example, it is information that specifies a portion that is encoded by the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38, that is, information that specifies a period to be imaged at a high frame rate. Specific examples of the related information will be described later.

  The file linking unit 32 includes the normal frame rate stream 24 generated by the normal frame rate frame encoding unit 36, one or more high frame rate streams generated by the high frame rate frame encoding unit 38, and the association generated by the control unit 22. Information 30 is multiplexed and concatenated to generate a single file. Here, “single file” means a file as a unit of file operation, and is a target of operations such as file duplication and deletion, file transfer, etc. collectively under one file name. As the file, for example, a container file according to the MP4 file format can be used. The container file can include a container describing header information, metadata, time information, and the like of each encoded data.

  The recording unit 34 stores the file generated by the file connection unit 32. As the recording unit 34, for example, a semiconductor memory or a hard disk can be adopted as the built-in memory. In addition, a memory card, a removable hard disk, or an optical disk can be adopted as the removable memory.

  The configuration of the image recording control apparatus 100 can be realized in hardware by an arbitrary processor, memory, and other LSI, and is realized in software by a program loaded in the memory. Depicts functional blocks realized by. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  FIG. 2 is a flowchart for explaining the flow of processing of the imaging apparatus 200 according to the first embodiment. The processing in this flowchart starts when the user presses a power button (not shown) of the imaging apparatus 200, for example.

  When the user selects moving image shooting (S10Y), the frame generation unit 14 generates a frame of the video acquired by the sensor 10 and stores it in the frame buffer 16 (S12). When the user selects the high-speed shooting mode via the operation unit 12 (S14Y), the high frame rate frame encoding unit 38 encodes the high frame rate frame stored in the frame buffer 16 (S16). When the user does not select the high-speed shooting mode (S14N), the high frame rate frame encoding unit 38 does not perform encoding.

  The frame thinning unit 18 thins a predetermined number of frames stored in the frame buffer 16 (S18). The normal frame rate frame encoding unit 36 encodes the normal frame rate frame output by the frame decimation unit 18 (S20). If the moving image shooting is not completed (S22N), the process returns to step S12 to continue the frame encoding. When the shooting of the moving image is finished (S22Y), the control unit 22 generates the related information 30 (S24). Thereafter, the file concatenation unit 32 includes the normal frame rate stream 24 generated by the normal frame rate frame encoding unit 36, one or more high frame rate streams generated by the high frame rate frame encoding unit 38, and the control unit 22 The generated related information 30 is connected to generate one file (S26). If the file linking unit 32 generates a file or does not shoot a moving image (S10N), this process ends.

  In addition, although the case where the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38 execute the encoding process in series has been described in FIG. 2, both may execute the encoding process in parallel. Good.

  Hereinafter, a specific example of the related information 30 generated by the control unit 22 will be described with reference to the drawings.

  FIG. 3 shows a normal frame rate frame group 40 to be encoded by the normal frame rate frame encoding unit 36 and a first high frame rate frame group 42 to be encoded by the high frame rate frame encoding unit 38. FIG. 6 is a diagram schematically illustrating an example of a relationship with a second high frame rate frame group 44. In this example, the frame rate of the normal frame rate frame group 40 is 60 fps, and the frame rates of the first high frame rate frame group 42 and the second high frame rate frame group 44 are 240 fps.

  The time axis 46 with the shooting start time of the normal frame rate frame group 40 as the origin, and the frame number of the first frame of the normal frame rate frame group 40 as 1, and the frame number when the serial number is sequentially assigned thereafter Consider a frame axis 48 to represent. The normal frame rate frame group 40 starts at time 0 and ends at time T5. At this time, the normal frame rate frame group 40 exists from the first frame to the F5th frame, and the total number of frames in the meantime is N0.

  When the user selects the high-speed shooting mode via the operation unit 12 at time T1, the high frame rate frame encoding unit 38 starts encoding the first high frame rate frame group 42. When the user ends the high-speed shooting mode at time T2, the high frame rate frame encoding unit 38 ends the encoding. The first high frame rate frame group is a frame group captured during the period D1 from time T1 to time T2.

  When the user selects the high-speed shooting mode again at time T3, the high frame rate frame encoding unit 38 starts encoding the second high frame rate frame group 44. When the time of D2 elapses and the user ends the high-speed shooting mode at time T4, the high frame rate frame encoding unit 38 ends the encoding.

  In this example, the normal frame rate frame group 40 and the first high frame rate frame group 42 overlap in the period D1 from time T1 to T2. Therefore, a set of times T1 and T2 can be considered as the related information 30 for specifying the overlapping portion between the two. Moreover, the set with D1 which is the period which overlaps with time T1 when both duplication starts can also be made into the relevant information 30 which pinpoints an overlap part. Similarly, the normal frame rate frame group 40 and the second high frame rate frame group 44 overlap in the period D2 from time T3 to T4, so the set of T3 and T4 or the set of T3 and D2 is replaced with the normal frame rate frame. The related information 30 that identifies the overlapping portion between the group 40 and the second high frame rate frame group 44 can be used.

  It is assumed that the normal frame rate frame group 40 is the F1th frame at time T1 when the user selects the high-speed shooting mode. After that, when the user ends the high-speed shooting mode at time T2, it is assumed that the frame group 40 is the F2th frame. At this time, an overlapping portion between the normal frame rate frame group 40 and the first high frame rate frame group 42 can be specified by a set of the frame numbers F1 and F2 of the normal frame rate frame group 40. Also, if there are N1 frames of the normal frame rate frame group 40 between the F1 frame and the F2 frame, the normal frame rate frame group 40 and the first high frame rate frame group 42 overlap. The portion can also be specified by a set of the frame number F1 of the normal frame rate frame group 40 and the number N1 of overlapping portions. Therefore, the set of F1 and F2 or the set of F1 and N1 can be used as the related information 30 that specifies the overlapping portion between the normal frame rate frame group 40 and the first high frame rate frame group 42.

  The same applies to the normal frame rate frame group 40 and the second high frame rate frame group 44. The frame number of the normal frame rate frame group 40 at time T3 is F3, and the normal frame rate frame group 40 at time T4. If the frame number of F4 is F4 and the number of frames between them is N2, the set of F3 and F4 or the set of F3 and N2 is the overlapping portion of the normal frame rate frame group 40 and the second high frame rate frame group 44 Related information 30 for identifying

  Here, instead of the number of frames N1 or N2 of the normal frame rate frame group 40, the number of frames of the corresponding first high frame rate frame group 42 or second high frame rate frame group 44 may be used. In this case, the frame rate of the first high frame rate frame group 42 and the second high frame rate frame group 44 is four times that of the normal frame rate frame group 40 (240 fps / 60 fps = 4). Each is quadrupled.

  FIG. 4 is a diagram schematically illustrating another example of the related information 30. In this example, a chapter is embedded as video segmentation data in a stream.

  In this example, FIG. 4 is a diagram showing an example of the positional relationship between chapters and frames when a stream in which chapters are embedded is expanded. The first chapter 52a exists in the chaptered stream 50, and is embedded immediately before the first high frame rate frame group 42 starts. The second chapter 52b is embedded immediately after the first high frame rate frame group 42 is finished. Therefore, the set of the first chapter 52a and the second chapter 52b can be used as the related information 30 that identifies the overlapping portion between the chaptered stream 50 and the first high frame rate frame group 42. The chapter 52 is embedded by the file linking unit 32 as the related information 30.

  FIG. 5 is a diagram schematically showing an example of the structure of the file 54 generated by the file linking unit 32. The file 54 is stored in the order of the header 56, the related information 30, the normal frame rate stream 24, and the first high frame rate stream 26 from the top of the file, and includes the nth stream 28 at the end. Here, the header 56 stores information on the head position of each stream. Thus, storing each stream in order is advantageous in that the control when the file connection unit 32 generates the file 54 is simplified.

  Note that the header 56 and the related information 30 do not have to be at the beginning of the file, and may be at the end of the file, for example. In this case, since the normal frame rate stream 24 is first, it is advantageous in that the entire video can be reproduced immediately without searching for a file. The related information 30 may be stored as a part of the header 56.

  FIG. 6 is a diagram schematically illustrating another example of the structure of the file 54 generated by the file connection unit 32. This example is the same as the example shown in FIG. 5 in that the header 56 and related information 30 are stored at the beginning, but differs in that each stream file is divided and stored in a distributed manner. In addition, the header 56 stores not only information on the head position of each stream but also information on the head position of each divided stream.

  In FIG. 6, the normal frame rate stream 24 is divided into m units, and the first stream 1 (reference 24a), the first stream 2 (reference 24b), and then the first stream n (reference 24). 24c), and finally the first stream is stored as m (symbol 24d). Between the divided normal frame rate streams 24, the divided second stream 1 (reference numeral 26a) and the second stream 2 (reference numeral 26b) are stored. Similarly, the nth stream is also divided and stored (reference numeral 28a). As described above, the method of dividing and recording each stream is advantageous in that it can efficiently record in the recording area of the recording unit 34, although the control at the time of recording becomes complicated. Even when there is no physically large free area in the recording area and only a small free area exists, if the file can be divided and recorded in small units, it can be recorded in such a small free area. Because.

  Therefore, the “predetermined standard” for dividing the above-described file is a minimum unit of data recording prepared in order to efficiently use the recording area of the recording unit 34. For example, the video recording recorded in the stream A unit delimited by time, data size or chapter 52 is used. These criteria and the minimum unit thereof may be determined experimentally in consideration of the capacity of the recording unit 34, the file system limitation, and the like.

  FIG. 7 is a diagram schematically illustrating an example of a recording format of the related information 30 stored in the file 54. In this example, a case where each time from time T0 to time T5 described above is used as the related information 30 will be described.

  First, the number of the related information 30 to be recorded is stored at the head of the area where the related information 30 is recorded. In this example, since there are six pieces of related information to be recorded from time T0 to time T5, 6 is stored as the number of pieces of related information 30. Next, the start time T0 and the end time T5 of the normal frame rate frame group 40 are stored, and then the start time T1 and the end time T2 of the first high frame rate frame group 42 are stored. . Finally, the start time T3 and the end time T4 of the second high frame rate frame group 44 are stored. If the number of frame groups increases, it may be stored in the same format. Further, since the start time and the end time are always stored as a set, the number of sets may be recorded as the number of related information 30 to be recorded. In the case of this example, since there are three sets of start time and end time, 3 is stored as the number of related information 30.

  Note that as the related information 30, the start time T0 and the end time T5 of the normal frame rate frame group 40 can be omitted.

  FIG. 8 is a diagram schematically showing another example of the recording format of the related information 30 stored in the file 54. In this example, a case where the first frame number of each frame group and the number of frames are used as the related information 30 will be described.

  In the case of this example, unlike the example shown in FIG. 7, the number of related information 30 to be recorded is not stored, but the related information 30 is stored in order from the top. Specifically, first, N0 which is the total number of frames of the normal frame rate frame group 40 is stored. Next, F1, which is the frame number of the normal frame rate frame group 40 corresponding to the first frame of the first high frame rate frame group 42, and the normal frame rate frame group 40 in the section overlapping with the first high frame rate frame group 42 are displayed. N1, which is the number of frames, is stored. Subsequently, F3, which is the frame number of the normal frame rate frame group 40 corresponding to the first frame of the second high frame rate frame group 44, and the normal frame rate frame group 40 in the section overlapping with the first high frame rate frame group 42. N2 which is the number of frames is stored. Finally, a terminator that is an identifier indicating the end of the related information 30 is stored. By reading the information from the head of the related information 30 to the terminator, the related information can be read correctly even if the number of the related information 30 is not stored.

  Note that N0, which is the total number of frames of the normal frame rate frame group 40, can be omitted as the related information 30.

  The operation according to the above configuration is as follows. When a user captures an image desired to be recorded using the imaging apparatus according to the first embodiment, the user performs high-speed shooting for particularly important scenes. When shooting is completed, the entire video is recorded at a frame rate of 60 fps, and for a scene shot at high speed, a video shot at a frame rate of 240 fps is also recorded. As a result, there are a plurality of videos shot at different frame rates, but these are recorded together in a single file.

  As described above, according to the first embodiment, the entire video is shot at a normal frame rate, and important scenes are also shot at a high frame rate, so that the code amount of an unimportant scene can be suppressed. As a result, the storage capacity can be saved. Also, since important scenes are generated as separate streams, it is easy to extract important scenes. Furthermore, since all the streams and the related information 30 are generated as a single file, management of copy, transfer, deletion, etc. of the file becomes easier as compared with the case where a separate file is generated for each stream. . It is also possible to prevent a series of stream files from being all reproduced as a result of erasing a part of the stream file.

  Although not all devices can play back a stream shot at a high frame rate such as 240 fps, according to Embodiment 1, the entire video is shot at a normal frame rate. Video can be played back even on a device that cannot play back a stream shot at a frame rate.

(Embodiment 2)
An outline of the second embodiment will be described. In the second embodiment, when a moving image file including a plurality of streams of a stream shot in the normal shooting mode and a stream shot in the high-speed shooting mode is reproduced, the high-speed shooting mode is used as a summary (digest) of the moving image. Cut out the stream shot in step 1 and play it back as a high frame rate video. This is because the video shot in the high-speed shooting mode is often an important scene and can be a summary of the entire video.

  FIG. 9 is a diagram schematically illustrating an internal configuration of the image reproduction control device 300 according to the second embodiment. The image reproduction control device 300 includes a file division unit 58, a high frame rate stream decoding unit 70, and a summary generation unit 76.

  The file dividing unit 58 receives from the recording unit 34 a single file including the related information 30 and a plurality of streams imaged at different frame rates as components, and divides it into components. The high frame rate stream decoding unit 70 decodes the high frame rate stream among the components divided by the file dividing unit 58.

  The summary generation unit 76 combines a plurality of high frame rate frame groups decoded and generated by the high frame rate stream decoding unit 70 to generate one high frame rate frame group. The display unit 66 displays the frame group generated by the summary generation unit 76.

  The configuration of the image reproduction control device 300 can be realized by an arbitrary processor, memory, or other LSI in terms of hardware, and can be realized by a program loaded in the memory in terms of software. Depicts functional blocks realized by. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  Here, when the 240 fps frame decoded by the high frame rate stream decoding unit 70 is reproduced, the frame is reproduced at 60 fps, which is the same frame rate as the frame at the normal frame rate. Therefore, it seems to be reproduced slowly four times (240 fps / 60 fps = 4 times) compared with the normal frame rate frame reproduction. A high frame rate stream, which is a moving image shot in the high-speed shooting mode, is often an important scene for a user who is a photographer. Therefore, the frame group generated by the summary generation unit 76 is a video that is intended to collect important scenes. It is a summary of.

  FIG. 10 is a diagram illustrating the relationship between a 60 fps frame group included in the normal frame rate stream 24 and a 240 fps frame group included in the high frame rate stream. In this example, there is a first 240 fps frame group from time T6 to time T7, a second 240 fps frame group from time T8 to time T9, and a third 240 fps frame group from time T10 to time T11.

  The high frame rate stream decoding unit 70 decodes the high frame rate stream divided by the file dividing unit 58 to generate a first 240 fps frame group, a second 240 fps frame group, and a third 240 fps frame group. To do. The frame synthesizing unit 72 combines these 240 fps frame groups and connects them to generate one high frame rate frame group. When the frame group generated in this way is played back, the video from time T8 to time T9 is played back at a high frame rate immediately after the video from time T6 to time T7 is played back at a high frame rate. Similarly, immediately after the video from time T8 to time T9 is played back at a high frame rate, the video from time T10 to T11 is played back at a high frame rate.

  As described above, according to the second embodiment, only important scenes shot in the high-speed shooting mode can be provided to the user with a high frame rate video.

(Embodiment 3)
An outline of the third embodiment will be described. In the third embodiment, when playing back a moving image file including a plurality of streams of a stream shot in the normal shooting mode and a stream shot in the high-speed shooting mode, a normal frame in a period shot in the high-speed shooting mode is used. Cut out the rate stream and play it. This is because the video shot in the high-speed shooting mode is often an important scene and can be a summary of the entire video. Hereinafter, the description overlapping with the second embodiment will be omitted as appropriate.

  FIG. 11 is a diagram schematically illustrating an internal configuration of the image reproduction control device 300 according to the third embodiment. The image reproduction control device 300 includes a file division unit 58, a normal frame rate stream decoding unit 68, and a summary generation unit 76. The summary generation unit 76 further includes a frame division unit 62 and a frame synthesis unit 72.

  The normal frame rate stream decoding unit 68 decodes the normal frame rate stream among the components divided by the file dividing unit 58. The summary generation unit 76 refers to the related information 30 and generates a summary from the frame group of the normal frame rate generated by the normal frame rate stream decoding unit 68. Specifically, the frame division unit 62 in the summary generation unit 76 refers to the related information 30 and divides and extracts only frames in a period of high-speed shooting from a group of frames at a normal frame rate. The frame synthesizing unit 72 in the summary generating unit 76 synthesizes and connects the frames divided by the frame dividing unit 62 to generate one normal frame rate frame group. The display unit 66 displays the frame group generated by the frame synthesis unit 72.

  The operation of the third embodiment will be described below using the example of FIG. The normal frame rate stream decoding unit 68 decodes the normal frame rate stream 24 divided by the file dividing unit 58 to generate a frame group of 60 fps. The frame dividing unit 62 refers to the related information 30 among the components divided by the file dividing unit 58, converts the 60 fps frame group into the 60 fps frame group first part from time T6 to time T7, and time from time T8. The frame is divided into three parts: a frame group second part of 60 fps by T9 and a frame group third part of 60 fps from time T10 to time T11. The frame synthesizing unit 72 synthesizes and connects the frames divided by the frame dividing unit 62 to generate one normal frame rate frame group.

  In the example of FIG. 10, the 240 fps synthesized frame group generated by the frame synthesizing unit 72 according to the second embodiment and the 60 fps synthesized frame group generated by the frame synthesizing unit 72 according to the third embodiment are combined. It is the figure which showed the relationship with a frame group. Only the 60 fps frame group captured during the period in which the 240 fps frame group is captured is cut out to form one normal frame rate frame group. Therefore, as in the case of the second embodiment, when the frame group generated by the frame synthesis unit 72 according to the third embodiment is reproduced, immediately after the video between time T6 and time T7 is reproduced at the normal frame rate. The video from time T8 to time T9 is reproduced at a normal frame rate. Immediately after the video from time T8 to time T9 is played back at the normal frame rate, the video from time T10 to T11 is played back at the normal frame rate.

  As described above, according to the third embodiment, only important scenes shot in the high-speed shooting mode can be summarized and provided to the user as a normal frame rate video. Unlike the case of the second embodiment, the summary video is played back at a normal frame rate, so that it takes less time to play compared to the case of playing at a high frame rate, and the whole summary can be viewed quickly. Can do.

(Embodiment 4)
An outline of the fourth embodiment will be described. In the fourth embodiment, when playing back a moving image file including a plurality of streams shot in a normal shooting mode and a stream shot in a high-speed shooting mode, only a stream in a period shot in the high-speed shooting mode is displayed. Although it is reproduced as a summary, the stream to be reproduced can be freely switched between a stream shot in the normal shooting mode and a stream shot in the high-speed shooting mode. The video shot in the high-speed shooting mode is often an important scene, and can be a summary of the entire video. Hereinafter, the description overlapping with the second embodiment or the third embodiment is omitted as appropriate.

  FIG. 13 is a diagram schematically illustrating an internal configuration of the image reproduction control device 300 according to the fourth embodiment. The image reproduction control device 300 includes a file dividing unit 58, an image decoding unit 60, and a summary generation unit 76. The image decoding unit 60 further includes a normal frame rate stream decoding unit 68 and a high frame rate stream decoding unit 70. The summary generation unit 76 further includes a frame division unit 62, a frame selection unit 64, and a frame control unit 74.

  The frame control unit 74 receives the type of video to be reproduced on the display unit 66 from the user via a user interface (not shown). The video type is either a stream shot in the normal shooting mode or a stream shot in the high-speed shooting mode. The frame selection unit 64 receives the type of video to be reproduced from the frame control unit 74, and the normal frame rate stream decoding unit 68 generates a normal frame rate frame decoding unit 68 based on the information, or the high frame rate stream decoding unit. A frame having a frame rate higher than normal generated by 70 is selected. The display unit 66 displays the frame selected by the frame selection unit 64.

  FIG. 14 is a diagram illustrating a state in which a frame group of 240 fps included in the high frame rate stream and a frame group of 60 fps corresponding to the frame group of 240 fps included in the high frame rate stream are divided. Hereinafter, the operation of the fourth embodiment will be described with reference to FIG.

  In this example, the frame control unit 74 receives an instruction from the user to reproduce the video as a summary at a frame rate of 240 fps. The frame selection unit 64 selects a frame with a frame rate of 240 fps generated by the high frame rate stream decoding unit 70 starting from time T6 in accordance with an instruction from the user. After selecting a frame with a frame rate of 240 fps at time T7, the frame selection unit 64 selects a frame with a frame rate of 240 fps at time T8.

  When the frame selection unit 64 selects a frame having a frame rate of 240 fps at time T12, the frame control unit 74 receives an instruction from the user that the video to be summarized should be reproduced at a normal frame rate. Thereafter, the frame selection unit 64 selects a frame having a frame rate of 60 fps at time T12 in accordance with an instruction from the user. After selecting a frame with a frame rate of 60 fps at time T9, the frame selection unit 64 skips frames from time T9 to time T10 and selects a frame with a frame rate of 60 fps at time T10. When the frame with the frame rate of 60 fps at time T11 is selected, the corresponding frame with the frame rate of 240 fps ends, so the frame selection unit 64 ends the frame selection.

  As described above, according to the fourth embodiment, it is possible to reproduce only a scene during a period when the user has taken a picture in the high-speed photography mode as a summary. In Embodiment 4, the frame rate of the video to be summarized can be freely changed, and a particularly important scene can be viewed at a higher frame rate than usual while being played back at the normal frame rate.

  The first to fourth embodiments have been described above. Any combination of these embodiments is also useful as an embodiment of the present invention. For example, if the first embodiment is combined with the second to fourth embodiments and the file dividing unit 58 receives the single file output from the file linking unit 32, the effects of these embodiments can be obtained. In addition to the sum, it is advantageous in that it can be provided to the user from imaging to reproduction consistently.

    The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. is there.

  For example, in the first embodiment, the case where the image encoding unit 20 includes two image encoding units, ie, the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38 has been described. When the calculation speed is sufficiently high, a single image encoding unit may be used. In this case, in the high-speed shooting mode, a normal frame rate video and a high frame rate video are encoded in a time division manner. Since the scale of the circuit can be reduced, it is advantageous in that the manufacturing cost and power consumption can be suppressed.

  In Embodiment 1 described above, it is assumed that the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38 perform calculations independently, but the calculation results may be shared. . For example, if the motion vectors of four frames calculated by the high frame rate frame encoding unit 38 are synthesized, a motion vector for one frame to be calculated by the normal frame rate frame encoding unit 36 is obtained. Sharing this result is advantageous in that the calculation time and power consumption can be suppressed as a whole.

  In the first embodiment, the case where the chapter 52 is recorded as the related information 30 has been described. However, the chapter 52 itself embedded in the stream may have video information of a corresponding high frame rate. In this case, in the above example, the start of the first high frame rate frame group 42 is recorded in the chapter 52a and the end of the first high frame rate frame group 42 is recorded in the chapter 52b as information. In the above example, the chapter 52a and the chapter 52b have been described as the related information 30 of the first high frame rate frame group 42. However, the number of frames, time, and the like of the chapter 52a and the first high frame rate frame group 42 are related information. 30.

  In the first embodiment, the case where the frame generation unit 14 always generates a frame of 240 fps has been described. Normally, a moving image of 60 fps is generated, and a frame of 240 fps is generated only in the high-speed shooting mode. May be. In this case, the calculation amount of the frame generation unit 14 when not in the high-speed shooting mode is reduced, and the calculation of the frame thinning unit 18 becomes unnecessary, which is advantageous in that power consumption can be suppressed. Further, when the frame rate is low, the amount of light received by the sensor 10 can be increased, so that an effect of suppressing noise can be obtained.

  In the second embodiment, the case where the file dividing unit 58 divides a single file into constituent elements has been described. However, the constituent elements are not grouped as a single file but are scattered as a plurality of files. Also good. In this case, a file management unit (not shown) instead of the file division unit 58 associates a plurality of scattered files. This can be realized, for example, by storing related information and a plurality of streams captured at different frame rates in the same folder, or by sharing a naming convention for a plurality of files.

  In the third embodiment, the case where a normal frame rate stream during the period of shooting in the high-speed shooting mode is cut out and combined to generate one composite frame group is generated. However, one composite frame group is generated. Alternatively, the frame may be displayed while being selected. In this case, a frame selection unit corresponding to the frame selection unit 64 of the fourth embodiment is introduced. Since the process of synthesizing the frame can be omitted, it is advantageous in that the process becomes faster.

  Further, in the fourth embodiment, the case where the image decoding unit 60 includes two image decoding units of the normal frame rate stream decoding unit 68 and the high frame rate stream decoding unit 70 has been described. In this case, a single image decoding unit may be used. In this case, the 60 fps frame group and the 240 fps frame group are decoded in a time division manner. Since the scale of the circuit can be reduced, it is advantageous in that the manufacturing cost and power consumption can be suppressed.

  30 related information, 58 file division unit, 60 image decoding unit, 62 frame division unit, 64 frame selection unit, 66 display unit, 68 normal frame rate stream decoding unit, 70 high frame rate stream decoding unit, 72 frame synthesis unit, 74 Frame control unit, 76 summary generation unit, 100 image recording control device, 200 imaging device, 300 image reproduction control device.

Claims (6)

Receiving a file including a normal frame rate stream, a higher frame rate stream for at least some scenes included in the normal frame rate stream, and associated information for identifying the scene; A file dividing unit that divides a normal frame rate stream, a higher frame rate stream, and the related information;
An image decoding unit that decodes at least one of the normal frame rate stream and the higher frame rate stream to generate a moving image frame;
A summary generation unit configured to generate, as a summary of a moving image, a frame of a moving image related to the scene, which includes both the normal frame rate stream and the higher frame rate stream in common based on the related information. An image reproduction control device characterized by the above. The image decoding unit includes a high frame rate stream decoding unit that decodes a stream having a frame rate higher than normal and outputs a frame of a moving image having a frame rate higher than normal,
The summary generation unit generates, as a summary of the moving image, a frame of a moving image having a frame rate higher than the normal with respect to a scene including the common frame rate stream and the higher frame rate stream in common. The apparatus according to claim 1, wherein: The image decoding unit includes a normal frame rate stream decoding unit that decodes the normal frame rate stream and outputs a frame of the normal frame rate moving image,
The summary generation unit includes a frame dividing unit that divides the frames of the moving image having the normal frame rate into frames related to the scene including the common frame based on related information for specifying the scene,
The summary generation unit is configured to extract the frame of the normal frame rate moving image divided by the frame division unit with respect to the scene including the common frame rate stream and the higher frame rate stream in common. The apparatus according to claim 1, wherein the apparatus is generated as a summary of the moving image. The image decoding unit further includes a high frame rate stream decoding unit that decodes a stream with a frame rate higher than normal and outputs a frame of a moving image with a frame rate higher than normal,
The summary generation unit is configured to select either one of a normal frame rate moving image frame and a higher frame rate moving image frame divided by the frame dividing unit based on information of a video frame to be reproduced. 4. The apparatus according to claim 3, wherein a frame is selected. An imaging unit for acquiring a moving image;
An image encoding unit that encodes a moving image captured by the imaging unit to generate two types of streams, a normal frame rate stream and a higher frame rate stream;
A control unit that generates related information that associates the stream with the normal frame rate and the stream with a frame rate higher than the normal generated in parallel with the generation of the stream;
An image reproduction control device according to any one of claims 1 to 4,
An imaging device comprising: a display device whose display is controlled by the image reproduction control device.   When a stream with a normal frame rate related to the same scene and a stream with a frame rate higher than normal exist at the same time, only the stream related to the same scene is played based on the related information identifying the same scene. An image reproduction control method characterized by:

Images