JP2011146847A - Image reproduction controller, image reproduction control method, and imaging device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology for properly switching and reproducing moving image data picked up in different frame rates. <P>SOLUTION: A normal frame rate stream-decoding part 68 decodes a stream of a normal frame rate and outputs a frame for a moving image of a normal frame rate. A high frame rate stream-decoding part 72 decodes a stream of a frame rate higher than a normal one about at least a partial scene included in the stream of the normal frame rate and outputs a frame of a moving image of a frame rate higher than the normal one. When changing a frame rate of a moving image that is currently reproduced, a frame selecting part 62 reproduces the frame rate of a stream of the moving image to be reproduced before and after switching, while performing the gradual switching from a frame rate before the switching to a frame rate after the switching. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image generation control device that decodes and reproduces a plurality of encoded moving images, and an imaging device equipped with the image generation control device.

  In recent years, digital movie cameras that allow general users to easily shoot moving images as well as still images have become widespread. Among them, the frame rate can be changed during shooting, and movies with different frame rates can be recorded. There are some that can be imaged simultaneously (see Patent Document 1). These moving images are recorded on a recording medium or the like and then viewed on, for example, a display monitor, a PC (Personal Computer), a television, or the like mounted on a movie camera.

JP 2007-195050 A

  If the same scene can be shot at different frame rates, for example, important scenes can be shot at a high frame rate with high time resolution. On the other hand, moving images with different frame rates for the same scene exist at the same time, and when both are switched and played back, there may be a sense of discomfort during viewing.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a technique for appropriately switching and reproducing moving image data captured at different frame rates.

  One embodiment of the present invention is an image reproduction control device. The apparatus relates to a normal frame rate stream decoding unit that decodes a normal frame rate stream and outputs a normal frame rate moving image frame, and at least a part of scenes included in the normal frame rate stream. A high frame rate stream decoding unit that decodes a stream with a frame rate higher than normal and outputs a frame with a higher frame rate than normal, and a frame with the normal frame rate or higher than normal A frame selecting unit that selects one of the frames of the rate moving image as a frame of the moving image to be played back, and when the frame rate is changed, the frame selecting unit is photographed in duplicate for the part of the scenes. Said normal frame rate stream A video stream that is reproduced before and after the switching by specifying switching timing between the normal frame rate stream and the higher frame rate stream with reference to related information that associates the high frame rate stream with each other The frame rate is switched stepwise from the frame rate before switching to the frame rate after switching.

Another embodiment of the present invention is an imaging apparatus. This apparatus generates an image capturing unit that acquires a moving image, and a moving image captured by the image capturing unit, and generates two types of streams: a normal frame rate stream and a higher frame rate stream. A possible image encoding unit, a control unit that generates related information associating the normal frame rate stream and the higher frame rate stream generated in parallel with the generation of the stream, and An image reproduction control device; and a display device controlled by the image reproduction control device.
Including.

  Yet another embodiment of the present invention is an image recording control method. In this method, when there are both a normal frame rate moving image stream and a higher frame rate moving image that were shot in duplicate for the same scene, The normal frame rate stream and higher than normal with reference to related information relating the normal frame rate video stream and the higher frame rate video stream for the same scene. The switching timing with the stream of the frame rate is specified, and the frame rate of the moving image stream that is played before and after the switching is reproduced while being gradually switched from the frame rate before the switching to the frame rate after the switching.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, and the like are also effective as an aspect of the present invention.

  According to the present invention, moving image data captured at different frame rates can be appropriately switched and reproduced.

1 is a diagram schematically illustrating an internal configuration of an imaging apparatus according to a first embodiment. 3 is a flowchart for explaining a processing flow of the imaging apparatus according to the first embodiment; The normal frame rate frame group to be encoded by the normal frame rate frame encoding unit according to the first embodiment, the first high frame rate frame group to be encoded by the high frame rate frame encoding unit, and the first frame group It is the figure which represented typically an example of the relationship with 2 high frame rate frame groups. It is a figure which shows an example of the positional relationship between a chapter and a frame when the stream in which the chapter concerning Embodiment 1 is expanded is expanded. It is the figure which represented typically an example of the structure of the file which the file connection part concerning Embodiment 1 produces | generates. It is the figure which represented typically the other example of the structure of the file which the file connection part concerning Embodiment 1 produces | generates. It is the figure which represented typically an example of the recording format of the relevant information stored in the file concerning Embodiment 1. FIG. It is the figure which represented typically the other example of the recording format of the relevant information stored in the file concerning Embodiment 1. FIG. FIG. 6 is a diagram schematically illustrating an internal configuration of an image reproduction control apparatus according to a second embodiment. 10 is a flowchart for explaining a flow of processing of the image reproduction control apparatus according to the second embodiment; It is the figure which showed typically the comparison of the time resolution of the frame of 60 fps and the frame of 240 fps concerning Embodiment 2. FIG. The frame rate concerning each step when the frame rate is changed stepwise from 60 fps to 240 fps is illustrated. It is the first half of a flowchart showing the flow of a frame rate switching process from 60 fps to 240 fps. It is the latter half part of the flowchart which shows the flow of the switching process of the frame rate from 60 fps to 240 fps.

(Embodiment 1)
An outline of the first embodiment of the present invention will be described. In the first embodiment, a moving image frame is generated at a frame rate of 240 fps (frames per second) and stored in a frame buffer. In a normal shooting mode, the frame is thinned out until the frame rate reaches 60 fps, and then the stream is extracted. Generate. While there is an instruction to perform high-speed shooting from the user, in addition to generating a stream at a frame rate of 60 fps, a stream at a frame rate of 240 fps before thinning out the frames is also generated. After shooting, a plurality of generated streams having different frame rates are concatenated and saved as a single file.

  FIG. 1 is a diagram schematically illustrating an internal configuration of the imaging apparatus 200 according to the first embodiment. The imaging device 200 includes a sensor 10, an operation unit 12, a frame generation unit 14, and an image recording control device 100. The image recording control apparatus 100 further includes a frame buffer 16, a frame thinning unit 18, an image encoding unit 20, a control unit 22, a file connection unit 32, and a recording unit 34.

  The frame generation unit 14 generates a video frame acquired by the sensor 10. The sensor 10 is a solid-state imaging device (not shown) such as a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The frame generation unit 14 outputs analog three primary color signals R, G, and R output from the sensor 10. B is processed and converted into frames of digital luminance signal Y and color difference signals Cr and Cb. The frame buffer 16 temporarily stores the frame generated by the frame generation unit 14.

  The frame thinning unit 18 thins out the frames stored in the frame buffer 16. The frame buffer 16 stores a frame generated by the frame generation unit 14 and having a frame rate of 240 fps. The frame thinning unit 18 thins the frame to a quarter, thereby obtaining a frame having a frame rate of 60 fps. Generate. Here, there may be various combinations of the frame rate of the frame generated by the frame generation unit 14 and the frame rate of the frame generated by the frame thinning unit 18. For example, the frame generation unit 14 may generate a 600 fps frame, and the frame thinning unit 18 may freely change the number of thinnings.

  The image encoding unit 20 further includes two image encoding units, a normal frame rate frame encoding unit 36 and a high frame rate frame encoding unit 38.

  The normal frame rate frame encoding unit 36 encodes the frame generated by the frame thinning unit 18 to generate the normal frame rate stream 24. The normal frame rate frame encoding unit 36 encodes the frame according to a predetermined standard. For the coding of moving images, the ISO (International Organization for Standardization) / IEC (International Electrotechnical Commission) standardized MPEG (Moving Picture Experts Group) series (MPEG-1, MPEG-2 and MPEG-4), an H.264 standardized by ITU-T (International Telecommunication Union-Telecommunication Standardization Sector) which is an international standard organization for telecommunications. 26x series standards (H.261, H.262 and H.263), or H.264, the latest video compression coding standard standardized jointly by both standards organizations. H.264 / AVC (a formal recommendation name in both organizations is MPEG-4 Part 10: Advanced Video Coding and H.264, respectively) H.264 / AVC Annex G, ie H.264. H.264 / SVC (Scalable Video Coding) is used.

  The high frame rate frame encoding unit 38 encodes the frames stored in the frame buffer 16 and generates a stream while receiving a user instruction input to the operation unit 12. Therefore, the stream generated by the high frame rate frame encoding unit 38 is a stream having a higher frame rate (240 fps) than the stream generated by the normal frame rate frame encoding unit 36. Hereinafter, an operation in which a high frame rate frame encoding unit 38 directly encodes a frame stored in the frame buffer 16 to generate a stream in accordance with a user instruction is referred to as a “high-speed shooting mode”. Similar to the normal frame rate frame encoding unit 36, the high frame rate frame encoding unit 38 is also MPEG-2, MPEG-4, H.264, or the like. H.264 / AVC, H.H. Compression encoding is performed according to a standard such as H.264 / SVC.

  The normal frame rate frame encoding unit 36 encodes a series of frames generated by the frame thinning unit 18 to generate one stream (normal frame rate stream 24), whereas the high frame rate frame encoding unit 38 Every time there is an instruction of the high-speed shooting mode from the user, the encoding is performed during the period of the instruction. For example, when the user instructs the first high-speed shooting mode, the high frame rate frame encoding unit 38 generates the first high frame rate stream 26. Thereafter, the high frame rate frame encoding unit 38 generates a stream every time the user gives an instruction for the high-speed shooting mode, and generates the n-th high frame rate stream 28 by the n-th instruction.

  The stream generated by the high frame rate frame encoding unit 38 has a high frame rate and can shoot video with high temporal resolution, but requires a large recording capacity. On the other hand, the normal frame rate stream 24 generated by the normal frame rate frame encoding unit 36 has a reduced temporal resolution because frames are thinned, but the storage capacity is reduced accordingly. Utilizing these properties, the entire video is recorded with the normal frame rate stream 24. For example, important scenes are shot in the high-speed shooting mode, so that important scenes are recorded with high temporal resolution while suppressing storage capacity. It becomes possible to do.

  The control unit 22 generates the related information 30 based on a user instruction input to the operation unit 12. Here, “related information” associates the streams generated by both when the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38 encode the video input to the sensor 10. Information. For example, it is information that specifies a portion that is encoded by the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38, that is, information that specifies a period to be imaged at a high frame rate. Specific examples of the related information will be described later.

  The file linking unit 32 includes the normal frame rate stream 24 generated by the normal frame rate frame encoding unit 36, one or more high frame rate streams generated by the high frame rate frame encoding unit 38, and the association generated by the control unit 22. Information 30 is multiplexed and concatenated to generate a single file. Here, “single file” means a file as a unit of file operation, and is a target of operations such as file duplication and deletion, file transfer, etc. collectively under one file name. As the file, for example, a container file according to the MP4 file format can be used. The container file can include a container describing header information, metadata, time information, and the like of each encoded data.

  The recording unit 34 stores the file generated by the file connection unit 32. As the recording unit 34, for example, a semiconductor memory or a hard disk can be adopted as the built-in memory. In addition, a memory card, a removable hard disk, or an optical disk can be adopted as the removable memory.

  The configuration of the image recording control apparatus 100 can be realized in hardware by an arbitrary processor, memory, and other LSI, and is realized in software by a program loaded in the memory. Depicts functional blocks realized by. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  FIG. 2 is a flowchart for explaining the flow of processing of the imaging apparatus 200 according to the first embodiment. The processing in this flowchart starts when the user presses a power button (not shown) of the imaging apparatus 200, for example.

  When the user selects moving image shooting (S10Y), the frame generation unit 14 generates a frame of the video acquired by the sensor 10 and stores it in the frame buffer 16 (S12). When the user selects the high-speed shooting mode via the operation unit 12 (S14Y), the high frame rate frame encoding unit 38 encodes the high frame rate frame stored in the frame buffer 16 (S16). When the user does not select the high-speed shooting mode (S14N), the high frame rate frame encoding unit 38 does not perform encoding.

  The frame thinning unit 18 thins a predetermined number of frames stored in the frame buffer 16 (S18). The normal frame rate frame encoding unit 36 encodes the normal frame rate frame output by the frame decimation unit 18 (S20). If the moving image shooting is not completed (S22N), the process returns to step S12 to continue the frame encoding. When the shooting of the moving image is finished (S22Y), the control unit 22 generates the related information 30 (S24). Thereafter, the file concatenation unit 32 includes the normal frame rate stream 24 generated by the normal frame rate frame encoding unit 36, one or more high frame rate streams generated by the high frame rate frame encoding unit 38, and the control unit 22 The generated related information 30 is connected to generate one file (S26). If the file linking unit 32 generates a file or does not shoot a moving image (S10N), this process ends.

  In addition, although the case where the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38 execute the encoding process in series has been described in FIG. 2, both may execute the encoding process in parallel. Good.

  Hereinafter, a specific example of the related information 30 generated by the control unit 22 will be described with reference to the drawings.

  FIG. 3 shows a normal frame rate frame group 40 to be encoded by the normal frame rate frame encoding unit 36 and a first high frame rate frame group 42 to be encoded by the high frame rate frame encoding unit 38. FIG. 6 is a diagram schematically illustrating an example of a relationship with a second high frame rate frame group 44. In this example, the frame rate of the normal frame rate frame group 40 is 60 fps, and the frame rates of the first high frame rate frame group 42 and the second high frame rate frame group 44 are 240 fps.

  The time axis 46 with the shooting start time of the normal frame rate frame group 40 as the origin, and the frame number of the first frame of the normal frame rate frame group 40 as 1, and the frame number when the serial number is sequentially assigned thereafter Consider a frame axis 48 to represent. The normal frame rate frame group 40 starts at time 0 and ends at time T5. At this time, the normal frame rate frame group 40 exists from the first frame to the F5th frame, and the total number of frames in the meantime is N0.

  When the user selects the high-speed shooting mode via the operation unit 12 at time T1, the high frame rate frame encoding unit 38 starts encoding the first high frame rate frame group 42. When the user ends the high-speed shooting mode at time T2, the high frame rate frame encoding unit 38 ends the encoding. The first high frame rate frame group is a frame group captured during the period D1 from time T1 to time T2.

  When the user selects the high-speed shooting mode again at time T3, the high frame rate frame encoding unit 38 starts encoding the second high frame rate frame group 44. When the time of D2 elapses and the user ends the high-speed shooting mode at time T4, the high frame rate frame encoding unit 38 ends the encoding.

  In this example, the normal frame rate frame group 40 and the first high frame rate frame group 42 overlap in the period D1 from time T1 to T2. Therefore, a set of times T1 and T2 can be considered as the related information 30 for specifying the overlapping portion between the two. Moreover, the set with D1 which is the period which overlaps with time T1 when both duplication starts can also be made into the relevant information 30 which pinpoints an overlap part. Similarly, the normal frame rate frame group 40 and the second high frame rate frame group 44 overlap in the period D2 from time T3 to T4, so the set of T3 and T4 or the set of T3 and D2 is replaced with the normal frame rate frame. The related information 30 that identifies the overlapping portion between the group 40 and the second high frame rate frame group 44 can be used.

  It is assumed that the normal frame rate frame group 40 is the F1th frame at time T1 when the user selects the high-speed shooting mode. After that, when the user ends the high-speed shooting mode at time T2, it is assumed that the frame group 40 is the F2th frame. At this time, an overlapping portion between the normal frame rate frame group 40 and the first high frame rate frame group 42 can be specified by a set of the frame numbers F1 and F2 of the normal frame rate frame group 40. Also, if there are N1 frames of the normal frame rate frame group 40 between the F1 frame and the F2 frame, the normal frame rate frame group 40 and the first high frame rate frame group 42 overlap. The portion can also be specified by a set of the frame number F1 of the normal frame rate frame group 40 and the number N1 of overlapping portions. Therefore, the set of F1 and F2 or the set of F1 and N1 can be used as the related information 30 that specifies the overlapping portion between the normal frame rate frame group 40 and the first high frame rate frame group 42.

  The same applies to the normal frame rate frame group 40 and the second high frame rate frame group 44. The frame number of the normal frame rate frame group 40 at time T3 is F3, and the normal frame rate frame group 40 at time T4. If the frame number of F4 is F4 and the number of frames between them is N2, the set of F3 and F4 or the set of F3 and N2 is the overlapping portion of the normal frame rate frame group 40 and the second high frame rate frame group 44 Related information 30 for identifying

  Here, instead of the number of frames N1 or N2 of the normal frame rate frame group 40, the number of frames of the corresponding first high frame rate frame group 42 or second high frame rate frame group 44 may be used. In this case, the frame rate of the first high frame rate frame group 42 and the second high frame rate frame group 44 is four times that of the normal frame rate frame group 40 (240 fps / 60 fps = 4). Each is quadrupled.

  FIG. 4 is a diagram schematically illustrating another example of the related information 30. In this example, a chapter is embedded as video segmentation data in a stream.

  In this example, FIG. 4 is a diagram showing an example of the positional relationship between chapters and frames when a stream in which chapters are embedded is expanded. The first chapter 52a exists in the chaptered stream 50, and is embedded immediately before the first high frame rate frame group 42 starts. The second chapter 52b is embedded immediately after the first high frame rate frame group 42 is finished. Therefore, the set of the first chapter 52a and the second chapter 52b can be used as the related information 30 that identifies the overlapping portion between the chaptered stream 50 and the first high frame rate frame group 42. The chapter 52 is embedded by the file linking unit 32 as the related information 30.

  FIG. 5 is a diagram schematically showing an example of the structure of the file 54 generated by the file linking unit 32. The file 54 is stored in the order of the header 56, the related information 30, the normal frame rate stream 24, and the first high frame rate stream 26 from the top of the file, and includes the nth stream 28 at the end. Here, the header 56 stores information on the head position of each stream. Thus, storing each stream in order is advantageous in that the control when the file connection unit 32 generates the file 54 is simplified.

  Note that the header 56 and the related information 30 do not have to be at the beginning of the file, and may be at the end of the file, for example. In this case, since the normal frame rate stream 24 is first, it is advantageous in that the entire video can be reproduced immediately without searching for a file. The related information 30 may be stored as a part of the header 56.

  FIG. 6 is a diagram schematically illustrating another example of the structure of the file 54 generated by the file connection unit 32. This example is the same as the example shown in FIG. 5 in that the header 56 and related information 30 are stored at the beginning, but differs in that each stream file is divided and stored in a distributed manner. In addition, the header 56 stores not only information on the head position of each stream but also information on the head position of each divided stream.

  In FIG. 6, the normal frame rate stream 24 is divided into m units, and the first stream 1 (reference 24a), the first stream 2 (reference 24b), and then the first stream n (reference 24). 24c), and finally the first stream is stored as m (symbol 24d). Between the divided normal frame rate streams 24, the divided second stream 1 (reference numeral 26a) and the second stream 2 (reference numeral 26b) are stored. Similarly, the nth stream is also divided and stored (reference numeral 28a). As described above, the method of dividing and recording each stream is advantageous in that it can efficiently record in the recording area of the recording unit 34, although the control at the time of recording becomes complicated. Even when there is no physically large free area in the recording area and only a small free area exists, if the file can be divided and recorded in small units, it can be recorded in such a small free area. Because.

  Therefore, the “predetermined standard” for dividing the above-described file is a minimum unit of data recording prepared in order to efficiently use the recording area of the recording unit 34. For example, the video recording recorded in the stream A unit delimited by time, data size or chapter 52 is used. These criteria and the minimum unit thereof may be determined experimentally in consideration of the capacity of the recording unit 34, the file system limitation, and the like.

  FIG. 7 is a diagram schematically illustrating an example of a recording format of the related information 30 stored in the file 54. In this example, a case where each time from time T0 to time T5 described above is used as the related information 30 will be described.

  First, the number of the related information 30 to be recorded is stored at the head of the area where the related information 30 is recorded. In this example, since there are six pieces of related information to be recorded from time T0 to time T5, 6 is stored as the number of pieces of related information 30. Next, the start time T0 and the end time T5 of the normal frame rate frame group 40 are stored, and then the start time T1 and the end time T2 of the first high frame rate frame group 42 are stored. . Finally, the start time T3 and the end time T4 of the second high frame rate frame group 44 are stored. If the number of frame groups increases, it may be stored in the same format. Further, since the start time and the end time are always stored as a set, the number of sets may be recorded as the number of related information 30 to be recorded. In the case of this example, since there are three sets of start time and end time, 3 is stored as the number of related information 30.

  Note that as the related information 30, the start time T0 and the end time T5 of the normal frame rate frame group 40 can be omitted.

  FIG. 8 is a diagram schematically showing another example of the recording format of the related information 30 stored in the file 54. In this example, a case where the first frame number of each frame group and the number of frames are used as the related information 30 will be described.

  In the case of this example, unlike the example shown in FIG. 7, the number of related information 30 to be recorded is not stored, but the related information 30 is stored in order from the top. Specifically, first, N0 which is the total number of frames of the normal frame rate frame group 40 is stored. Next, F1, which is the frame number of the normal frame rate frame group 40 corresponding to the first frame of the first high frame rate frame group 42, and the normal frame rate frame group 40 in the section overlapping with the first high frame rate frame group 42 are displayed. N1, which is the number of frames, is stored. Subsequently, F3, which is the frame number of the normal frame rate frame group 40 corresponding to the first frame of the second high frame rate frame group 44, and the normal frame rate frame group 40 in the section overlapping with the first high frame rate frame group 42. N2 which is the number of frames is stored. Finally, a terminator that is an identifier indicating the end of the related information 30 is stored. By reading the information from the head of the related information 30 to the terminator, the related information can be read correctly even if the number of the related information 30 is not stored.

  Note that N0, which is the total number of frames of the normal frame rate frame group 40, can be omitted as the related information 30.

  The operation according to the above configuration is as follows. When a user captures an image desired to be recorded using the imaging apparatus according to the first embodiment, the user performs high-speed shooting for particularly important scenes. When shooting is completed, the entire video is recorded at a frame rate of 60 fps, and for a scene shot at high speed, a video shot at a frame rate of 240 fps is also recorded. As a result, there are a plurality of videos shot at different frame rates, but these are recorded together in a single file.

  As described above, according to the first embodiment, the entire video is shot at a normal frame rate, and important scenes are also shot at a high frame rate, so that the code amount of an unimportant scene can be suppressed. As a result, the storage capacity can be saved. Also, since important scenes are generated as separate streams, it is easy to extract important scenes. Furthermore, since all the streams and the related information 30 are generated as a single file, management of copy, transfer, deletion, etc. of the file becomes easier as compared with the case where a separate file is generated for each stream. . It is also possible to prevent a series of stream files from being all reproduced as a result of erasing a part of the stream file.

  Although not all devices can play back a stream shot at a high frame rate such as 240 fps, according to Embodiment 1, the entire video is shot at a normal frame rate. Video can be played back even on a device that cannot play back a stream shot at a frame rate.

(Embodiment 2)
An outline of the second embodiment will be described. In the second embodiment, when playing back a moving image file including a plurality of streams of a normal frame rate stream shot in the normal shooting mode and a higher frame rate stream shot in the high-speed shooting mode, When switching to a stream with a different frame rate, playback is performed by switching the frame rate stepwise so that the frame rate before and after the switching is smoothly connected. This is to reduce visual discomfort when switching between streams with different frame rates.

  FIG. 9 is a diagram schematically illustrating an internal configuration of the image reproduction control device 300 according to the second embodiment. The image reproduction control device 300 includes a file dividing unit 58, an image decoding unit 60, a frame selection unit 62, and a frame control unit 64.

  The file dividing unit 58 receives from the recording unit 34 a single file including the related information 30 and a plurality of streams imaged at different frame rates as components, and divides it into components. The image decoding unit 60 decodes the normal frame rate stream 24 and one or more high frame rate streams among the components divided by the file dividing unit 58. For this reason, the image decoding unit 60 further decodes the normal frame rate stream to generate a normal frame rate frame, and the normal frame rate stream decoding unit 68 to decode the high frame rate stream to increase the frame rate higher than normal. And a high frame rate frame generation unit 70 for generating the frames.

  The high frame rate frame generation unit 70 further interpolates the high frame rate stream decoding unit 72 that decodes the high frame rate stream, and the frame of the high frame rate moving image generated by the high frame rate stream decoding unit 72. And an intermediate frame generation unit 74 for generating a new frame. By generating the intermediate frame, the frame rate of the frame of the moving image generated by the high frame rate stream decoding unit 72 can be further increased. For the interpolation process, a known interpolation process such as linear interpolation or spline interpolation may be used.

  The frame control unit 64 receives the related information 30 and an instruction to change the frame rate of the moving image to be reproduced from the user via a user interface (not shown). The frame selection unit 62 receives the related information 30 and the instruction from the user from the frame control unit 64, and selects a frame to be output to the display unit 66 from the frames output from the image decoding unit 60. A specific method of selecting a frame will be described later. The display unit 66 displays the frame selected by the frame selection unit 62.

  The above-described configuration of the image reproduction control device 300 can be realized in hardware by an arbitrary processor, memory, and other LSI, and is realized in software by a program loaded in the memory. Describes functional blocks realized by collaboration. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  FIG. 10 is a flowchart for explaining a process flow of the image reproduction control apparatus 300 according to the second embodiment. This example is an example when the frame rate is switched from 60 fps to 240 fps, and the processing in this flowchart is started, for example, when a user reproduces a moving image on a display device (not shown) in which the image reproduction control device 300 is mounted.

  The frame selection unit 62 determines whether there is an instruction from the user to switch the frame rate from 60 fps to 240 fps via the frame control unit 64. Specifically, it is checked whether or not a switching signal is transmitted from the frame control unit 64. If there is a switching instruction from the user (S28Y), the frame selection unit 62 checks whether the switching instruction is a switching instruction from 60 fps to 240 fps. In the case of a switching instruction from 60 fps to 240 fps (S30Y), the frame selection unit 62 checks whether a video frame of 240 fps is output from the image decoding unit 60. When a 240 fps moving image frame is output (S32Y), the frame selection unit 62 switches the frame rate in a stepwise manner so as to smoothly connect from 60 fps before switching to 240 fps after switching (S34). After executing the frame rate switching process from 60 fps to 240 fps, the frame selection unit 62 sets a 240 fps frame selection flag (not shown) to ON (S36).

  If it is not a switching instruction from 60 fps to 240 fps, that is, a switching instruction from 240 fps to 60 fps (S30N), the frame selection unit 62 switches the frame rate stepwise from 240 fps to 60 fps (S38). After executing the frame rate switching process from 240 fps to 60 fps, the frame selection unit 62 sets the 240 fps frame selection flag to OFF (S40).

  When there is no switching instruction from the user (S28N), when there is no 240 fps moving image frame (S32N), or after the frame selection unit 62 sets the selection flag, the frame selection unit 62 sets the 240 fps frame. It is checked whether the selection flag is on or off (S42). If the 240 fps frame selection flag is off (S42N), then the frame selection unit 62 selects and outputs a 60 fps frame (S48). If the 240 fps frame selection flag is ON (S42Y), the frame selection unit 62 checks whether a 240 fps moving image frame is output from the image decoding unit 60. When a 240 fps moving image frame is output (S44Y), the frame selection unit 62 selects and outputs a 240 fps frame (S46). When a 240 fps moving image frame is not output (S44N), the frame selection unit 62 selects and outputs a 60 fps frame. When the frame selection unit 62 selects and outputs one of the frames, the processing in this flowchart ends.

  When the 240 fps frame decoded by the high frame rate stream decoding unit 72 is reproduced, the frame is reproduced at 60 fps, which is the same frame rate as the normal frame rate frame. Therefore, it seems to be reproduced slowly four times (240 fps / 60 fps = 4 times) as compared with the normal frame rate frame reproduction. When playback from a frame with a frame rate of 60 fps is changed to playback of a frame with a frame rate of 240 fps, or when playback from a frame with a frame rate of 240 fps is changed to playback with a frame rate of 60 fps, the playback speed increases. Since it is observed as if it is suddenly changed, the viewer may feel uncomfortable.

  Here, before explaining the details of the frame rate switching processing according to the second embodiment, the technology that is the premise thereof will be described first.

  FIG. 11 is a diagram schematically showing a comparison of time resolution between a frame of 60 fps and a frame of 240 fps. In a 60 fps frame, the time interval between adjacent frames 76a and 76b is 1/60 second. On the other hand, in a 240 fps frame, the time interval between adjacent frames 78a and 78b is 1/240 seconds. Therefore, the frame group of 240 fps reaches the frame 78e through the frame 78a, the frame 78b, the frame 78c, and the frame 78d until the frame 76a changes to the frame 76b.

  This is equivalent to playback at 60 fps if playback is performed while skipping three 240 fps frame groups. Specifically, if the frame 78d is displayed next to the frame 78a and then the frame 78i is displayed, this corresponds to displaying the frames 76a, 76b, and 76c, which are 60 fps frames. This is because the time interval between frames is 4/240 = 1/60 seconds.

  Then, if playback is performed while skipping the 240 fps frame group two by two, playback is performed at a time interval of 3/240 = 1/80 seconds, that is, a frame rate of 80 fps. Similarly, playback at a frame rate of 120 fps can be realized by skipping a single image. As a result, the frame rate can be changed stepwise from 60 fps, 80 fps, 120 fps, and 240 fps. Hereinafter, displaying the nth frame by skipping n−1 frames is referred to as “step number n” or “display with step number n”. For example, the step number 4 means that three frames are skipped.

  FIG. 12 illustrates the frame rate at each stage when the frame rate is changed in stages from 60 fps to 240 fps. When the frame rate change is started at time T6 during reproduction at 60 fps, 60 fps reproduction is reproduced by first displaying a 240 fps frame group in step number 4. At time T7, the number of steps is 3, and the frame rate is changed to 80 fps. At time T8, the frame rate is set to 120 fps with the number of steps being 2, and finally, the number of steps is set to 1 at time T9 to shift to the frame rate of 240 fps, and the change of the frame rate is completed. As described above, by selecting a frame group of 240 fps to be displayed while skipping a predetermined number of frames, changing the number of frames to be skipped, the frame rate is smoothly connected from 60 fps to 240 fps via 80 fps and 120 fps. Can be changed as follows.

  Here, the period D60 displayed with the number of steps 4, the period D80 displayed with the number of steps 3, and the period D120 displayed with the number of steps 2 may be changed according to the user's preference, or 240 fps existing at that time It may be determined experimentally so as to reduce the uncomfortable feeling associated with the change in the frame rate, such as using 5% of the number of frames of the moving image.

  The intermediate frame generation unit 74 can change the frame rate in finer steps by generating an intermediate frame of a 240 fps frame group. For example, by generating an average image of adjacent frames of a 240 fps frame group, a frame group corresponding to 480 fps is generated, and the number of steps during reproduction is changed to 8, 7, 6,. think of. Changes in the frame rate at this time are 480/8 = 60 fps, 480 / 7≈69 fps, 480/6 = 80 fps, and thereafter 96 fps, 120 fps, 160 fps, and 240 fps.

  FIG. 13A is the first half of a flowchart showing the flow of the frame rate switching process from 60 fps to 240 fps, and describes step S34 in FIG. 10 in detail.

  60 (fps) is stored in fpsL which is a variable for storing a normal frame rate, and 240 (fps) is stored in fpsH which is a variable for storing a frame rate higher than normal (S50). The least common multiple of fpsL and fpsH is stored in fpsC, which is a variable for storing the least common multiple of fpsL and fpsH (S52). In this example, 240, which is the least common multiple of 60 and 240, is stored.

  A maximum value stepL of the number of steps and a minimum value steppH of the number of steps are set (S54). In this example, stepL = fpsC / fpsC = 240/60 = 4 and stepH = 240/240 = 1. A frame group frameC having a frame rate of fpsC is generated (S56), and the preprocessing of the switching process is terminated.

  In this example, since fpsC = 240, it matches with fpsH. Therefore, the frame group generated by the high frame rate stream decoding unit 72 may be used as it is as frameC. In that case, the intermediate frame generation unit 74 outputs the frame generated by the high frame rate stream decoding unit 72 as it is as an intermediate frame. Therefore, the “intermediate frame” includes a frame generated by the high frame rate stream decoding unit 72.

  If fpsC is larger than fpsH, the intermediate frame generation unit 74 generates an intermediate frame to increase the frame rate. Also, the reason why fpsC, which is the least common multiple of fpsL and fpsH, is stored in step S52 because it is guaranteed that fpsC is divisible by fpsL and fpsH.

  FIG. 13B is the second half of the flowchart showing the flow of the frame rate switching process from 60 fps to 240 fps.

  The value of stepL is substituted into the first loop variable i (S58). In this example, 4 is substituted into the loop variable i. The number num of frames for displaying the loop variable i as the number of steps is set (S62). When the loop variable i = 4, it corresponds to setting the period D60 to be displayed with the number of steps of 4 in FIG. The value of the number num of frames is set as the initial value of the second loop variable j (S64). The frame group frameC with the number of steps i and the frame rate of fpsC is displayed (S68), and the value of the loop variable j is updated to j-1 (S70). Thereafter, while the loop variable j is greater than 0 (S66Y), display of frameC and update of the loop variable j are repeated. When the loop variable j becomes 0 or less (S66N), the process exits the loop and updates the first loop variable i to i-1 (S72). The above processing is repeated while i is larger than stepH (S60Y), and is repeated until loop variable i becomes equal to stepH (S60N). Since the loop variable i corresponds to the number of steps, when the loop variable i becomes equal to stepH, that is, the frame rate switching process is completed. At this point, the processing in this flowchart ends.

  Note that the process of switching the frame rate from 240 fps to 60 fps (step S38) is substantially the same as the process shown in FIGS. 13A and 13B (step S34), and thus the description thereof is omitted. The initial value of the first loop variable i and its update method may be appropriately determined so that the frame rate is 240 fps to 60 fps.

  As described above, according to the second embodiment, it is possible to switch the frame rate step by step when switching and reproducing moving images having different frame rates. Thereby, the uncomfortable feeling resulting from the difference of the time resolution which arises when switching a frame rate can be reduced.

  The first embodiment and the second embodiment have been described above. Any combination of these embodiments is also useful as an embodiment of the present invention. For example, if the first embodiment is combined with the second to fourth embodiments and the file dividing unit 58 receives the single file output from the file linking unit 32, the effects of these embodiments can be obtained. In addition to the sum, it is advantageous in that it can be provided to the user from imaging to reproduction consistently.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. is there.

  For example, in the first embodiment, the case where the image encoding unit 20 includes two image encoding units, ie, the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38 has been described. When the calculation speed is sufficiently high, a single image encoding unit may be used. In this case, in the high-speed shooting mode, a normal frame rate video and a high frame rate video are encoded in a time division manner. Since the scale of the circuit can be reduced, it is advantageous in that the manufacturing cost and power consumption can be suppressed.

  In Embodiment 1 described above, it is assumed that the normal frame rate frame encoding unit 36 and the high frame rate frame encoding unit 38 perform calculations independently, but the calculation results may be shared. . For example, if the motion vectors of four frames calculated by the high frame rate frame encoding unit 38 are synthesized, a motion vector for one frame to be calculated by the normal frame rate frame encoding unit 36 is obtained. Sharing this result is advantageous in that the calculation time and power consumption can be suppressed as a whole.

  In the first embodiment, the case where the chapter 52 is recorded as the related information 30 has been described. However, the chapter 52 itself embedded in the stream may have video information of a corresponding high frame rate. In this case, in the above example, the start of the first high frame rate frame group 42 is recorded in the chapter 52a and the end of the first high frame rate frame group 42 is recorded in the chapter 52b as information. In the above example, the chapter 52a and the chapter 52b have been described as the related information 30 of the first high frame rate frame group 42. However, the number of frames, time, and the like of the chapter 52a and the first high frame rate frame group 42 are related information. 30.

  In the first embodiment, the case where the frame generation unit 14 always generates a frame of 240 fps has been described. Normally, a moving image of 60 fps is generated, and a frame of 240 fps is generated only in the high-speed shooting mode. May be. In this case, the calculation amount of the frame generation unit 14 when not in the high-speed shooting mode is reduced, and the calculation of the frame thinning unit 18 becomes unnecessary, which is advantageous in that power consumption can be suppressed. Further, when the frame rate is low, the amount of light received by the sensor 10 can be increased, so that an effect of suppressing noise can be obtained.

  In the second embodiment, the case where the image decoding unit 60 includes two image decoding units, the normal frame rate stream decoding unit 68 and the high frame rate stream decoding unit 72, has been described. Alternatively, a single image decoding unit may be used. In this case, a normal frame rate stream and a high frame rate stream are decoded in a time division manner. Since the scale of the circuit can be reduced, it is advantageous in that the manufacturing cost and power consumption can be suppressed.

  In the second embodiment, the case where the file dividing unit 58 divides a single file into constituent elements has been described. However, the constituent elements are not grouped as a single file but are scattered as a plurality of files. Also good. In this case, a file management unit (not shown) instead of the file division unit 58 associates a plurality of scattered files. This can be realized, for example, by storing related information and a plurality of streams captured at different frame rates in the same folder, or by sharing a naming convention for a plurality of files.

  In the second embodiment, the case where the intermediate frame generation unit 74 generates a new frame by interpolating the high frame rate moving image frame generated by the high frame rate stream decoding unit 72 has been described. The intermediate frame generation unit 74 can also generate a new frame by duplicating a high frame rate moving image frame. In this case, the interpolation process can be omitted, which is advantageous in that the calculation time and power consumption can be suppressed.

  14 frame generation unit, 20 image encoding unit, 22 control unit, 24 normal frame rate stream, 30 related information, 50 stream, 58 file division unit, 60 image decoding unit, 62 frame selection unit, 68 normal frame rate stream decoding unit 70 high frame rate frame generation unit, 72 high frame rate stream decoding unit, 74 intermediate frame generation unit, 200 imaging device, and 300 image reproduction control device.

Claims (5)

A normal frame rate stream decoding unit that decodes a normal frame rate stream and outputs a frame of a normal frame rate video;
A high frame rate stream decoding unit that decodes a stream having a frame rate higher than normal for at least some scenes included in the stream having the normal frame rate and outputs a frame of a moving image having a frame rate higher than normal; ,
A frame selection unit that selects either a frame of the moving image with the normal frame rate or a frame of the moving image with a frame rate higher than the normal as a frame of the moving image to be played back,
When the frame rate is changed, the frame selection unit obtains related information associating the normal frame rate stream and the higher frame rate stream that have been taken in duplicate for the partial scene. The switching timing between the normal frame rate stream and the higher frame rate stream is specified by referring to, and the frame rate of the video stream played before and after the switching is switched from the frame rate before the switching. An image reproduction control device characterized by switching to a later frame rate step by step.   The frame selection unit, when switching the frame rate in stages, changes the number of frames to skip while selecting a moving image frame having a frame rate higher than normal while skipping a predetermined number of frames. The apparatus according to 1. An intermediate frame generating unit that generates a frame of a new moving image as an intermediate frame by interpolating a frame of a moving image with a frame rate higher than normal;
2. The apparatus according to claim 1, wherein the frame selection unit changes the number of frames to be skipped while selecting the intermediate frames while skipping a predetermined number of frames when switching the frame rate stepwise. An imaging unit for acquiring a moving image;
An image encoding unit capable of encoding a moving image captured by the imaging unit to generate two types of streams, a normal frame rate stream and a higher frame rate stream;
A control unit that generates related information that associates the stream with the normal frame rate and the stream with a frame rate higher than the normal generated in parallel with the generation of the stream;
An image reproduction control device according to any one of claims 1 to 3,
An imaging device comprising: a display device whose display is controlled by the image reproduction control device.   When there are both a normal frame rate moving image stream and a higher frame rate moving image stream recorded for the same scene at the same time, when switching between the two, the same The normal frame rate stream and the higher frame rate stream with reference to related information relating the normal frame rate video stream and the higher frame rate video stream for the scene. Image playback control, characterized by specifying the timing of switching to and moving the frame rate of the video stream played before and after the switching stepwise from the frame rate before switching to the frame rate after switching Method.

Images