JP2015080035A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, capable of efficiently processing compressed moving image data, according to a display frame rate of an external display device.SOLUTION: An image processing device includes: acquisition means for acquiring hierarchically coded moving image data; decoding means for decoding the moving image data acquired by the acquisition means; output means for outputting the decoded moving image data decoded by the decoding means to an external display device; information acquisition means for acquiring information related to the display frame rate of the external display device; and control means for controlling the decoding means not to decode the data in an upper hierarchy than a hierarchy corresponding to the frame rate of the external display device included in the moving image data.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  Conventionally, as an image processing device, a digital camera, a mobile phone, a computer, a hard disk recorder, and the like are known as devices that reproduce moving image data recorded on a recording medium and display the data on an external display device. Conventionally, when an image is output from these devices to a monitor or television which is an external display device, it is generally output via an interface conforming to a standard such as the HDMI (registered trademark) standard or the MHL standard. It was.

  On the other hand, in recent years, it has become possible to shoot and reproduce moving images at various frame rates including a frame rate that exceeds the display frame rate of a monitor or television.

  However, in standards such as the HDMI (registered trademark) standard and the MHL standard, switching of the frame rate is not permitted after the connection is made once. For this reason, when reproducing content with mixed frame rates, the frame rate must be constant.

  Regarding frame rate conversion, when playing back content that contains both 3D video and 2D video, the frame rate is doubled during the 2D video period to display the same frame rate as the 3D video content. A technique (Patent Document 1) is disclosed.

Japanese Patent Application No. 2010-531729

  However, in the technique of Patent Document 1, for example, when the frame rate of 3D video and 2D video is a predetermined frame rate and images of different frame rates are mixed, how is the frame rate of each image? It was unknown whether to grasp. Further, for example, there is no mention of how to decode the reproduced content itself when it is compressed.

  Therefore, the present invention provides an image processing device that can process compressed moving image data efficiently in accordance with the display frame rate of an external display device in an image processing device that processes compressed moving image data. For the purpose.

  In order to achieve such an object, the image processing apparatus of the present invention includes an acquisition unit that acquires hierarchically encoded moving image data, a decoding unit that decodes the moving image data obtained by the acquisition unit, and the decoding Output means for outputting the moving image data decoded by the means to an external display device, information acquisition means for acquiring information relating to the display frame rate of the external display device, and the external display device included in the moving image data And control means for controlling the decoding means so as not to decode data in a hierarchy higher than the hierarchy corresponding to the frame rate.

  According to the present invention, for example, in an image processing device that processes compressed moving image data, the compressed moving image data can be efficiently processed according to the display frame rate of the external display device.

1 is a diagram illustrating a configuration of an imaging apparatus according to an embodiment. The figure which shows the structure of TV2000 of a present Example. The figure which shows the system configuration | structure of a present Example. The figure which shows the example of the moving image data of a present Example. FIG. 3 is a flowchart showing moving image data reproduction processing according to the first embodiment. FIG. 3 is a flowchart showing moving image data reproduction processing according to the first embodiment. FIG. 3 is a flowchart showing moving image data reproduction processing according to the first embodiment. FIG. 9 is a flowchart showing a moving image data reproduction process according to the second embodiment. FIG. 9 is a flowchart showing a moving image data reproduction process according to the second embodiment. FIG. 9 is a flowchart showing a moving image data reproduction process according to the second embodiment.

  Hereinafter, examples of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiments. The following embodiments do not limit the invention according to the claims, and all combinations of features described in the embodiments are not necessarily essential to the solution means of the invention.

  Note that each functional block described in the present embodiment is not necessarily separate hardware. That is, for example, the functions of some functional blocks may be executed by one piece of hardware. In addition, the function of one functional block or the functions of a plurality of functional blocks may be executed by some hardware linked operations. Further, the function of each functional block may be executed by a computer program developed on the memory by the CPU.

  In this embodiment, an imaging apparatus will be described as an example of an image processing apparatus. However, any apparatus that can decode moving image data may be used. For example, a mobile phone, a smartphone, a tablet information terminal, a notebook information terminal, a computer, or the like may be used.

  Hereinafter, such an imaging apparatus will be described.

  First, the main configuration of the imaging apparatus 1000 according to the present embodiment will be described with reference to FIG. In the image pickup apparatus of the present embodiment, H.264 is used. It is possible to perform encoding, recording, reproduction, and decoding of moving image data in an encoding method according to the H.265 standard. In this embodiment, a part of the moving image data is H.264. Hierarchical coding according to the H.265 standard is performed. In FIG. 1, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are connected to an internal bus 1010. Further, an input processing unit 1004, an output processing unit 1006, a terminal control unit 1008, a recording medium control unit 1011, a camera signal processing unit 1015, and a code decoding processing unit 1016 are connected. Each unit connected to the internal bus 1010 can exchange data with each other via the internal bus 1010.

  The ROM 1002 stores various programs for the CPU 1001 to operate. A flash memory is also included. The RAM 1003 appropriately stores programs and variables necessary for the operation of the CPU 1001, temporary work data, and the like. The CPU 1001 controls each unit of the imaging apparatus 1000 using the RAM 1003 as a work memory according to a program stored in the ROM 1002 or the storage medium 1012.

  The optical system 1013 is a modularized photographing lens including a lens (not shown), a focus, a diaphragm, a zoom mechanism, and a processor for controlling them, and forms an optical image of a subject. The CPU 1001 can acquire and control the focus, aperture, and zoom states of the optical system 1013 via the control bus 1010. The imaging element 1014 includes a photoelectric conversion element such as a CCD or CMOS element, and an A / D converter. The image sensor 1014 converts an optical image into an analog electric signal, and then acquires an image signal converted into a digital signal. Based on the control of the CPU 1001, the camera signal processing unit 1015 performs resize processing such as predetermined pixel interpolation / reduction, color conversion, various correction processing, and the like on the image signal converted by the image sensor 1014.

  Based on the control of the CPU 1001, the encoding / decoding processing unit 1016 compresses and encodes the image signal processed by the camera signal processing unit 1015 at a predetermined bit rate and format format, and generates moving image data and still image data. At the time of reproduction, a process for decoding the compressed moving image data is performed.

  The input processing unit 1004 receives a user operation on the operation unit 1005, generates a control signal corresponding to the operation, and supplies the control signal to the CPU 1001. For example, the operation unit 1005 includes character information input devices such as a keyboard and pointing devices such as a mouse and a touch panel as input devices that receive user operations. In addition, remote control devices such as an infrared remote controller are also included. The touch panel is an input device that outputs coordinate information according to a position touched with respect to, for example, a planar input unit. Thereby, it is possible to cause the imaging apparatus 1000 to perform an operation according to the user operation. In this embodiment, the following description will be made assuming that the input device is a touch panel.

  The output processing unit 1006 causes the display unit 1007 to display display data such as a GUI (Graphical User Interface) generated by the CPU 1001 according to the program. In addition, an image decoded by the camera signal processing unit 1015 or the code decoding processing unit 1016 is displayed on the display unit 1007.

  Note that when a touch panel is used as the operation unit 1005, the operation unit 1005 and the display unit 1007 can be configured integrally. For example, the touch panel is configured so that light transmittance does not hinder display on the display portion 1007 and is attached to the upper layer of the display surface of the display portion 1007. Then, by associating the input coordinates on the touch panel with the display coordinates on the display unit 1007, a GUI is configured as if the user can directly operate the screen displayed on the display unit 1007. I get out. For example, when contact is detected on the coordinates corresponding to the button-like GUI component displayed on the display unit 1007, it is assumed that the button has been pressed, and the CPU 1001 executes the processing associated with the button to perform physical processing. It is possible to realize the same function as when the key is pressed. Hereinafter, in this embodiment, the button press of the physical key and the touch detection to the button-like GUI component (GUI button) are not particularly distinguished, and both are handled as button presses.

  Furthermore, the CPU 1001 can detect the following operations on the touch panel. Touching the touch panel with a finger or a pen (hereinafter referred to as touchdown). The touch panel is touched with a finger or a pen (hereinafter referred to as touch-on). The touch panel is moved while being touched with a finger or a pen (hereinafter referred to as a move). The finger or pen that was touching the touch panel is released (hereinafter referred to as touch-up). A state where nothing is touched on the touch panel (hereinafter referred to as touch-off). The CPU 1001 is notified of these operations and the position coordinates where the finger or pen touches the touch panel through the internal bus 1010, and the CPU 1001 determines what operation has been performed on the touch panel based on the notified information. . Regarding the move, the moving direction of the finger or pen moving on the touch panel can also be determined for each vertical component / horizontal component on the touch panel based on the change of the position coordinates. It is also assumed that a stroke is drawn when touch-up is performed on the touch panel through a certain move from touch-down. The operation of drawing a stroke quickly is called a flick. A flick is an operation of quickly moving a certain distance while touching a finger on the touch panel and then releasing it, in other words, an operation of quickly tracing a finger on the touch panel. If it is detected that the moving is performed at a predetermined speed or more over a predetermined distance, and a touch-up is detected as it is, it can be determined that a flick has been performed. In addition, when it is detected that the movement is performed at a predetermined distance or more and less than a predetermined speed, it is determined that the drag has been performed. The touch panel may be any of various types of touch panels such as a resistive film method, a capacitance method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, an image recognition method, and an optical sensor method. .

  The recording medium control unit 1011 is connected to a recording medium 1012 such as an HDD or a non-volatile semiconductor memory. Based on the control of the CPU 1001, data and video files are read from the connected storage medium 1012 and the recording medium 1012 is read. Write data and video files. The recording medium 1012 to which the recording medium control unit 1011 can be connected may be a detachable nonvolatile semiconductor memory such as a memory card, for example, via a socket (not shown). The recording medium 1012 can record information necessary for the control of the CPU 1001 in addition to the captured video data.

  The terminal control unit 1008 transmits / receives a connection notification signal, reads EDID (to be described later), outputs a video / audio signal, and transmits / receives a control command to / from the TV 2000 (described later) through the terminal 1009 based on the control of the CPU 1001. In this embodiment, it is assumed that the video / audio signal handled by the terminal control unit 1008 conforms to the HDMI (registered trademark) (High-Definition Multimedia Interface) standard, and the terminal 1009 is an HDMI (registered trademark) terminal. The terminal control unit 1008 includes an EDID (Extended Display Identification Data) reading unit (not shown). EDID is data describing device information of a sink device (hereinafter referred to as TV2000 in the embodiment) defined by the HDMI (registered trademark) standard including receivable video data information such as frame rate and resolution. When an external device is connected to the terminal 1009, the CPU 1001 analyzes the EDID of the TV 2000 received via the terminal control unit 1008. Then, after determining the type of signal that can be received by the TV 2000 and the capability information, a video / audio signal is transmitted to the TV 2000 according to the determination result. Also, the terminal 1009 in this embodiment has a signal line for transmitting a +5 V signal that is a connection notification signal from the source device (hereinafter, the imaging device 1000 in the embodiment) defined in the HDMI (registered trademark) standard to the TV 2000. . In addition, an HPD signal that is a connection notification signal from the TV 2000 to the imaging apparatus 1000, a TMDS signal that transmits a video / audio signal from the imaging apparatus 1000 to the TV 2000, and a signal line (not shown) that can transmit the EDID are provided.

  In addition, although description was not given about audio | voice, the audio | voice signal obtained with the microphone not shown may be encoded and recorded with moving image data. When reproducing, the compressed audio data is decoded and output from a speaker (not shown).

  FIG. 2 shows a configuration of a TV 2000 to which the embodiment of the present invention can be applied. Tuner 2001 demodulates a broadcast wave received by an antenna (not shown), and transmits the demodulated signal to display unit 2002. The TV 2000 can receive a video signal and an audio signal compliant with the HDMI (registered trademark) standard via the terminal 2003. The display unit 2002 displays a demodulated signal or a video signal received via the terminal 2003. The TV 2000 includes an EDID storage unit (not shown), and transmits an EDID to the imaging apparatus 1000 via a terminal 2003 when detecting a connection with the imaging apparatus 1000. The TV 2000 has a speaker (not shown) and can output received audio information. The EDID includes at least information on the number of pixels that can be displayed by the TV 2000 and the display frame rate.

  FIG. 3 is a diagram illustrating an example of a communication system according to an embodiment of the present invention. The imaging apparatus 1000 outputs a video / audio signal conforming to the HDMI (registered trademark) standard from the terminal 1009 to the TV 2000. The outputted video / audio signal is transmitted to the TV 2000 via the HDMI (registered trademark) cable 3100. The TV 2000 receives a video signal and an audio signal at a terminal 2003, displays an image corresponding to the received video signal on the display unit 2002, and outputs audio corresponding to the audio signal from a speaker.

  FIG. 4 is a diagram for explaining the data structure of moving image data handled in this embodiment. In this embodiment, H. It handles moving image data encoded by an encoding method according to the H.265 standard (HEVC standard). The moving image data shown in FIG. Hierarchical encoding is performed by an encoding method according to the H.265 standard (HEVC standard). H. In the H.265 standard (HEVC standard), data corresponding to each frame of a moving image is in units of NAL (Network Abstraction Layer) units. When reproducing and decoding the moving image data, the temporal_id information stored in the NAL unit header information (Nal_Unit_Header) is read to determine whether the encoded data corresponds to the layer. Note that when the compressed moving image data is decoded, inverse quantization processing and inverse discrete cosine transform processing (or inverse cosine transform processing) are performed on the moving image data. However, the header information (Nal_Unit_Header) of the NAL unit can be obtained without performing the inverse quantization process and the inverse discrete cosine transform process (or the inverse cosine transform process).

  In FIG. 4, reference numerals 4001, 4002, 4003, 4004, 4005, 4006, and 4007 each denote one frame of image data. The image data of each frame is stored in the NAL unit. Reference numerals 4020, 4021, and 4022 denote hierarchical structures based on the HEVC standard. Reference numeral 4020 denotes a layer 0 (temporal_id = 0), 4021 denotes a layer 1 (temporal_id = 1), and 4022 denotes a hierarchy corresponding to layer 2 (temporal_id = 2). “temporal_id” is a name of a value that designates each layer in the HEVC standard. As described above, since the hierarchy of the frame corresponding to Nal_Unit can be acquired by temporal_id, it can be said that temporal_id is information indicating the hierarchy. In this embodiment, hierarchical information acquisition is performed by the CPU 1001. Specifically, temporal_id is stored in the header of the NAL unit included in each frame image, and is referred to at the time of decoding. In the hierarchical encoding based on the HEVC standard, when a frame image to be decoded is reproduced, the encoding is performed so that only a layer in the same layer as the frame image or a layer in a lower layer is referred to.

  In FIG. 4, the image data 4001, 4003, and 4007 are temporal_id = 0. That is, the image data corresponding to 4020 belongs to layer 0. The image data 4002 and 4005 are temporal_id = 1. That is, the image data belonging to layer 1 corresponds to 4021. The image data 4004 and 4006 are temporal_id = 2. That is, the image data belonging to layer 2 corresponds to 4022. Image data 4001, 4003, and 4007 are encoded data compressed at the compression rate for layer 0 by the encoding / decoding processing unit 1016. The image data 4002 and 4005 are encoded data compressed by the encoding / decoding processing unit 1016 at the compression rate for layer 1. The image data 4004 and 4006 are encoded data compressed at the compression rate for layer 2 by the encoding / decoding processing unit 1016.

  The display time information of each image data from the image data 4001 to the image data 4007 is recorded for each image data according to the timing at which each image data is captured at the time of recording. At the time of reproduction of each image data from the image data 4001 to the image data 4007, display time information recorded for each image data is read, and reproduction and display are performed along the display time information.

  The moving image data shown in FIG. 4 has a hierarchical structure having three layers of layer 0 4020, layer 1 4021, and layer 2 222. At the time of reproduction, the frame rate at the time of reproduction can be designated by designating which layer or lower layer is the decoding target. Since the data structure shown in FIG. 4 has three layers, it is possible to specify a frame rate at the time of three-stage reproduction.

  In this embodiment, when the layer to be reproduced is layer 0, that is, image data 4001, 4003, and 4007 belonging to layer 0 are to be reproduced, and the reproduction frame rate during normal reproduction at that time is 30 frames / second. Suppose that If the layer to be played back is designated as layer 1 or lower, the image data 4001, the image data 4002, and the image data 4003, 4005, 4007 are played back. It is assumed that the playback frame rate during normal playback at that time is 60 frames / second. When the layer to be reproduced is designated as layer 2 or lower, all the image data from the image data 4001 to the image data 4007 are to be reproduced. However, if there is no frame for the specified layer, playback at the frame rate corresponding to that layer is not possible. Therefore, the image data 4001, 4002, and 4003 are reproduced at a frame rate equivalent to layer 1, 60 frames / second, and the image data from 4004 to 4007 are reproduced at a frame rate equivalent to layer 2, 120 frames / second.

  As described above, the moving image data handled in the present embodiment is moving image data whose playback frame rate can be switched by specifying a layer.

  Next, the operation of the imaging apparatus 1000 according to the present embodiment will be described. 5, 6, and 7 are flowcharts illustrating a procedure of processing performed by the CPU 1001 of the imaging apparatus 1000.

  FIG. 5 is a flowchart for explaining a procedure of processing performed before reproduction of moving image data. It should be noted that the pre-reproduction process may be started automatically after the power is turned on in a state where no user operation is input, may be started when the imaging apparatus 1000 is switched to the reproduction mode, or the TV 2000 (external It may be performed when connected to the display device. Further, in the playback mode, it may be performed when the moving image data to be played back is selected, or when a playlist that specifies the playback order of the plurality of moving image data is specified. It is assumed that the moving image data to be reproduced is all the moving image data recorded on the recording medium 1012 when the playback mode is switched to after the power is turned on or when an external display device is connected. Further, when moving image data is selected and a playlist is specified, the reproduced moving image data becomes the moving image data to be reproduced.

  In step S5000, the CPU 1001 starts pre-reproduction processing.

  In step S <b> 5001, the CPU 1001 controls the recording medium control unit 1011 to read data indicating the correspondence between temporal_id and frame rate of moving image data recorded on the recording medium 1012. The correspondence data indicating the correspondence between temporal_id and the frame rate is stored, for example, in a clip info file which is a metadata recording file defined by the AVCHD standard. In this embodiment, as described above, layer 2 4022 is 120 Hz, layer 1 4021 is 60 Hz, and layer 0 4020 is 30 Hz.

  In step S <b> 5002, the CPU 1001 acquires temporal_id (common temporal_id) corresponding to a layer that can be played back over the entire section of the moving image data to be played back. As described above, temporal_id is stored in Nal_Unit_Header corresponding to each frame of the moving image data. Therefore, in this embodiment, the CPU 1001 analyzes Nal_Unit_Header of moving image data to be reproduced. Then, the temporal_id corresponding to the layer in which the video data that can be played back over the entire section of the video data to be played back is determined. For example, if the temporal_id of the moving image data to be played back includes the id corresponding to layer 1 over the entire section, but does not include the id corresponding to layer 2 in the middle, the temporal_id corresponding to layer 1 becomes the common temporal_id. In the example of FIG. 4, temporal_id indicating the layer 0 4020 and the layer 1 4021 is the common temporal_id. In Layer 2 4022, there is no corresponding image data across all sections, so temporal_id indicating Layer 2 4022 is not determined to be a common temporal_id. Hereinafter, in this embodiment, the temporal_id indicating the layer 1 4021 is described as a common temporal_id. Note that temporal_id indicating the layer 0 4020 can be a common temporal_id, and therefore is called a candidate temporal_id. There may be a plurality of candidate temporal_ids.

  The common temporal_id is determined by analyzing all moving image data, but may be recorded in advance in a metadata recording area such as a clip info file at the time of recording. In this case, the common temporal_id is determined based on the setting at the time of recording, or is determined based on the layer information recorded after recording.

  In step S5003, the CPU 1001 determines a common frame rate (fcom) from the correspondence between the temporal_id and the frame rate read in step S5001 and the common temporal_id analyzed in step S5002. The common frame rate (fcom) is a frame rate that can be reproduced over the entire section. In this embodiment, since the layer 1 4021 is a common temporal_id, the common frame rate (fcom) is 60 Hz, which is the frame rate corresponding to the layer 1 4021. Further, the frame rate (candidate frame rate (falt)) corresponding to the candidate temporal_id is also determined. Here, 30 Hz, which is the frame rate corresponding to layer 0 4020, is determined as the candidate frame rate. Both the determined common frame rate (fcom) and candidate frame rate (falt) are recorded in the RAM 1003. In step S5004, the pre-reproduction process ends.

  FIG. 6 is a flowchart illustrating a processing procedure when the imaging apparatus 1000 and the TV 2000 are connected via HDMI (registered trademark).

  In step S5100, the CPU 1001 starts an HDMI (registered trademark) connection process. In step S5101, the CPU 1001 communicates with the TV 2000, receives the EDID information of the TV 2000, and writes it in the RAM 1003. Next, in step S5102, the CPU 1001 analyzes the EDID information written in the RAM 1003 in step S5101, and acquires display frame rate information that can be displayed by the TV 2000.

  Next, in S5103, the CPU 1001 compares the display frame rate of the TV 2000 analyzed in S5102 with the common frame rate (fcom). Then, it is determined whether the display frame rate of the TV 2000 includes a common frame rate. If the common frame rate is included, the process proceeds to S5104. Otherwise, the process proceeds to S5105.

  In step S5104, the CPU 1001 notifies the TV 2000 to output a video signal at the common frame rate (fcom) determined in step S5103, and connects via HDMI (registered trademark).

  On the other hand, in S5105, if there is a candidate frame rate (falt) that is not determined to be included in the EDID among candidate frame rates that are not the common frame rate, the processing of S5103 is performed again with falt as fcom. Run. If all the candidate frame rates have been compared with EDID, the process proceeds to S5106.

  In step S5106, the CPU 1001 determines that there is no frame rate connectable to the TV, and disconnects the connection with the TV 2000 via HDMI (registered trademark). When this step is executed, the processing for outputting HDMI (registered trademark) corresponding to the moving image data is not executed in S5200 to S5208 described later.

  Finally, in step S5107, the CPU 1001 ends the HDMI (registered trademark) connection process.

  In the present embodiment, description will be made assuming that the display frame rate of the TV 2000 is 60 frames / second. That is, fcom is 60 frames / second.

  FIG. 7 is a flowchart for explaining a procedure for reproducing moving image data. The flowchart in FIG. 7 shows processing during reproduction of moving image data.

  In step S5200, the CPU 1001 starts moving image data reproduction processing. The CPU 1001 controls the recording medium control unit 1011 so as to read out the header information of the moving image data to be reproduced from the recording medium 1012.

  Next, in step S5201, the CPU 1001 controls the recording medium control unit 1011 to read information including at least the Nal_Unit_Header of the frame of the moving image data to be reproduced from the recording medium 1012. Next, in S5202, the CPU 1001 acquires temporal_id stored in Nal_Unit_Header of the frame of the moving image data read in S5201. In step S5203, the CPU 1001 determines the frame rate corresponding to the temporal_id read in step S5202 using the correspondence data between the temporal_id and the frame rate read in step S5001 in FIG.

  Next, in S5024, the CPU 1001 compares the common frame rate (fcom) determined in S5103 of FIG. 6 with the frame rate (f) determined in S5203. Then, it is determined whether or not the frame of the moving image data read in S5201 is a decoding target frame in connection with the TV 2000. As described above, since the common frame rate (fcom) is 60 frames / second, in this embodiment, any frame having temporal_id corresponding to layer 1 and layer 0 is a frame to be decoded. If the frame rate (f) determined in S5203 is equal to or less than the common frame rate (fcom), the process proceeds to S5205. If the frame rate (f) is greater than the common frame rate (fcom), the process proceeds to S5206.

  In step S5205, the CPU 1001 sets Nal_Unit data corresponding to the Nal_Unit_Header read in step S5201 as a decoding target. The CPU 1001 controls the encoding / decoding processing unit 1016 to decode this Nal_Unit data, transmits the decoded image data to the terminal control unit 1008, and transmits the decoded image data to the TV 2000 via the terminal 1009. The decoded image data may be transmitted to the output processing unit 1006 and displayed on the display unit 1007. H. In H.265, since inter-frame predictive coding may be performed, the decoding order does not always match the display and output order.

  On the other hand, in S5206, the CPU 1001 controls the code decoding processing unit 1016 so as not to decode the Nal_Unit data of the frame corresponding to the Nal_Unit_Header read in S5201. Note that if the Nal_Unit data is not read from the recording medium 1012, the recording medium control unit 1011 may be controlled so that the Nal_Unit data of the frame corresponding to the Nal_Unit_Header read in S5202 is not read.

  When the processes of S5205 and S5206 are completed, the process proceeds to S5207. In step S5027, if the Nal_Unit corresponding to the Nal_Unit_Header read in SS5201 is the last Nal_Unit, the CPU 1001 determines that the reproduction process has ended and proceeds to step S5208. On the other hand, if it is not the last Nal_Unit, the process moves to S5201, and the next Nal_Unit is processed.

  Finally, in S5208, the CPU 1001 ends the reproduction process.

  As described above, the imaging apparatus according to the present embodiment acquires moving image data that is hierarchically encoded from a recording medium, decodes the acquired moving image data, and outputs corresponding image data to an external display device. Device. Then, the imaging apparatus according to the present embodiment acquires hierarchical information corresponding to each frame of the moving image data, and does not decode data corresponding to a frame higher than the hierarchy corresponding to the display frame rate of the external display device. Like that. Therefore, the imaging apparatus of the present embodiment can efficiently process the compressed moving image data in accordance with the display frame rate of the external display device. It is also possible not to switch the output frame rate for an external display device.

  In the first embodiment, the operation related to the playback processing of one moving image file has been described. However, the playback target is not limited to one file. It may be a playlist in which a plurality of moving image files are grouped, or all reproducible video data, for example, all moving images displayed on an index screen displaying a list of moving images. In this case, the search range of the common temporal_id is set to the moving image data of all files included in the playlist or all video data included in all reproducible files.

  Next, Example 2 will be described.

  In the second embodiment, an imaging apparatus will be described. The second embodiment will be described as having the same block configuration, system configuration, and data configuration as the first embodiment. That is, also in Example 2, since FIGS. 1-4 is common, description is abbreviate | omitted.

  Next, the operation of the imaging apparatus 1000 according to the second embodiment will be described. 8, 9, and 10 are flowcharts for explaining a procedure of processes performed by the CPU 1001 of the imaging apparatus 1000.

  FIG. 8 is a flowchart for explaining a procedure of processing performed before reproduction of moving image data. It should be noted that the pre-reproduction process may be started automatically after the power is turned on in a state where no user operation is input, may be started when the imaging apparatus 1000 is switched to the reproduction mode, or the TV 2000 (external It may be performed when connected to the display device. Further, in the playback mode, it may be performed when the moving image data to be played back is selected, or when a playlist that specifies the playback order of the plurality of moving image data is specified. It is assumed that the moving image data to be reproduced is all the moving image data recorded on the recording medium 1012 when the playback mode is switched to after the power is turned on or when an external display device is connected. Further, when moving image data is selected and a playlist is specified, the reproduced moving image data becomes the moving image data to be reproduced.

  In step S6000, the CPU 1001 starts pre-reproduction processing.

  In step S <b> 6001, the CPU 1001 controls the recording medium control unit 1011 to read data indicating the correspondence between temporal_id and frame rate of moving image data recorded on the recording medium 1012. The correspondence data indicating the correspondence between temporal_id and the frame rate is stored, for example, in a clip info file which is a metadata recording file defined by the AVCHD standard. In this embodiment, as described above, layer 2 4022 is 120 Hz, layer 1 4021 is 60 Hz, and layer 0 4020 is 30 Hz.

  In step S <b> 6002, the CPU 1001 acquires temporal_id (maximum temporal_id) corresponding to the deepest layer (displayed in the case of playback with a high frame rate) of all sections of the moving image data to be played. As described above, temporal_id is stored in Nal_Unit_Header corresponding to each frame of the moving image data. Therefore, in this embodiment, the CPU 1001 analyzes Nal_Unit_Header of moving image data to be reproduced. Then, the temporal_id (maximum temporal_id) corresponding to the deepest layer (displayed in the case of reproduction with a high frame rate) of all sections of the moving image data to be reproduced is determined. For example, in the example of FIG. 4, the maximum temporal_id of the moving image data to be reproduced is temporal_id corresponding to layer 2.

  The maximum temporal_id is determined by analyzing all moving image data, but may be recorded in advance in a metadata recording area such as a clip info file at the time of recording. In this case, the maximum temporal_id is determined by the setting at the time of recording, or is determined based on the layer information recorded after recording. Also in the second embodiment, a temporal_id of a layer lower than the maximum temporal_id is set as a candidate temporal_id.

  In step S6003, the CPU 1001 determines the maximum frame rate (fmax) from the correspondence between the temporal_id and the frame rate read in step S6001 and the maximum temporal_id analyzed in step S6002. In this embodiment, since layer 2 4022 is set to the maximum temporal_id, the maximum frame rate (fmax) is 120 Hz, which is the frame rate corresponding to layer 2 4022. Further, the frame rate (candidate frame rate (falt)) corresponding to the candidate temporal_id is also determined. Here, 60 Hz, which is the frame rate corresponding to layer 1 4021, is determined as the candidate frame rate. Further, 30 Hz which is a frame rate corresponding to layer 0 4020 is also a candidate frame rate. Both the maximum frame rate (fmax) and the candidate frame rate (falt) are recorded in the RAM 1003. In step S6004, the pre-reproduction process is terminated.

  FIG. 9 is a flowchart illustrating a processing procedure when the imaging apparatus 1000 and the TV 2000 are connected via HDMI (registered trademark).

  In step S6100, the CPU 1001 starts an HDMI (registered trademark) connection process. In step S6101, the CPU 1001 communicates with the TV 2000, receives the EDID information of the TV 2000, and writes it in the RAM 1003. Next, in step S6102, the CPU 1001 analyzes the EDID information written in the RAM 1003 in step S6101, and acquires display frame rate information that can be displayed by the TV 2000.

  In step S6103, the CPU 1001 compares the display frame rate of the TV 2000 analyzed in step S6102 with the maximum frame rate (fmax). Then, it is determined whether or not the maximum frame rate is included in the display frame rate of the TV 2000. If the maximum frame rate is included, the process proceeds to S6104. If not, the process proceeds to S5105.

  In step S6104, the CPU 1001 notifies the TV 2000 to output a video signal at the maximum frame rate (fmax) determined in step S6103, and connects via HDMI (registered trademark).

  On the other hand, in S6105, if there is a candidate frame rate (falt) that is not determined to be included in the EDID among candidate frame rates that are not the maximum frame rate (fmax), the fault is set to fmax and S6103 is again performed. Execute the process. If all candidate frame rates have been compared with EDID, the process advances to step S6106.

  In step S6106, the CPU 1001 determines that there is no frame rate that can be connected to the TV, and disconnects the connection with the TV 2000 via HDMI (registered trademark). When this step is executed, the processing for outputting HDMI (registered trademark) corresponding to the moving image data is not executed in S6200 to S6208 described later.

  Finally, in step S6107, the CPU 1001 ends the HDMI (registered trademark) connection process.

  In the present embodiment, description will be made assuming that the display frame rate of the TV 2000 is 120 frames / second. That is, fmax is 120 frames / second.

  FIG. 10 is a flowchart for explaining the procedure for reproducing moving image data. The flowchart in FIG. 10 shows processing during reproduction of moving image data.

  In step S6200, the CPU 1001 starts reproduction processing of moving image data. The CPU 1001 controls the recording medium control unit 1011 so as to read out the header information of the moving image data to be reproduced from the recording medium 1012.

  In step S <b> 6201, the CPU 1001 first controls the recording medium control unit 1011 to read Nal_Unit corresponding to the first frame from the recording medium 1012. Then, the encoding / decoding processing unit 1016 is controlled so as to decode the read Nal_Unit data. The CPU 1001 stores the decoded image data in the RAM 1003, transmits it to the terminal control unit 1008, and transmits it to the TV 2000 via the terminal 1009. At this time, the CPU 1001 reads temporal_id stored in Nal_Unit_Header, and determines the corresponding frame rate using the correspondence data between temporal_id and the frame rate read in S6001 of FIG. This is stored in the RAM 1003 as the frame rate (fprev) of the immediately preceding frame.

  Next, in step S6202, the recording medium control unit 1011 is controlled to read Nal_Unit corresponding to the next frame from the recording medium 1012. Then, the encoding / decoding processing unit 1016 is controlled so as to decode the read Nal_Unit data of the next frame, and the CPU 1001 stores the decoded image data in the RAM 1003.

  Next, in S6203, temporal_id stored in Nal_UnitHeader of Nal_Unit of the frame read in S6202 is read. In step S6204, the CPU 1001 determines the frame rate (f) corresponding to the temporal_id using the correspondence between the temporal_id read in step S6001 and the data corresponding to the frame rate. That is, at this time, there is a frame rate (fprev) of the immediately preceding frame and a frame rate (f) of the current frame.

  In step S6205, the CPU 1001 first compares the frame rate (f) of the current frame with the maximum frame rate (fmax). If the current frame rate (f) is smaller than the maximum frame rate (fmax), the process proceeds to S6206. If the current frame rate (f) is not smaller than the maximum frame rate (fmax), the process is performed. Move to S6209.

  In step S6206, the CPU 1001 compares the frame rate (fprev) of the previous frame with the frame rate (f) of the current frame, and calculates the number of dummy video frames to be output. In the case of the second embodiment, for example, the frame rate (f) of the image data 4002 is 60 frames / second, and the frame rate (fprev) of the immediately preceding image data 4001 is 30 frames / second. In this case, it is determined from the hierarchical structure of the video data shown in FIG. 4 that one image data corresponding to layer 2 4022 is insufficient in order to output a video signal at a frame rate equivalent to 120 frames / second. it can. In this case, the number of dummy repetitions is 1.

  In step S6207, the CPU 1001 transmits the image data of the immediately previous frame stored in the RAM 1003 as dummy image data to the terminal control unit 1008 via the bus 1001. The video data transmitted to the terminal control unit 1008 is transmitted to the TV 2000 via the terminal 1009.

  Next, in S6208, the CPU 1001 subtracts 1 from the number of repetitions, and if the subtraction result is 0, the CPU 1001 proceeds to S6210. If 1 or more, the process advances to step S6207.

  On the other hand, in S6209, the CPU 1001 compares the maximum frame rate (fmax) determined in S6103 with the frame rate (f) read in S6204. If f is greater than fmax, the process proceeds to S6211. The process proceeds to S6210. That is, image data of a frame corresponding to a frame rate higher than the maximum frame rate (fmax) is not output to the TV 2000. Note that Nal_Unit data corresponding to this frame may not be decoded or may not be read from the recording medium 1012.

  In step S 6210, the CPU 1001 transmits the current image data via the bus 1001 to the terminal control unit 1008. Although transmitted to the terminal control unit 1008, the diagram data is transmitted to the TV 2000 via the terminal 1009.

  In step S6211, the CPU 1001 determines whether the Nal_Unit read in step 6202 is the last Nal_Unit of the playback target Nal_Units. If it is the last Nal_Unit, it is determined that the reproduction process has been completed, and the process proceeds to S6212. If it is not the last Nal_Unit, the process returns to S6202, and the next Nal_Unit is further processed. In step S6211, the CPU 1001 ends the reproduction process.

  Note that the number of dummy frames obtained in S6206 may be calculated from the display interval between the current image data and the previous image data. For example, in the second embodiment, since the playback frame rate of layer 1 4021 is 60 frames / second, if the image data 4001 is the previous image data and the image data 4002 is the current image data, the playback interval between the two image data is This is about 16.6 ms (t1). On the other hand, the maximum frame rate is 120 frames / second, and the display interval of the image data in this case is about 8.3 ms (t2). Therefore, the number of dummy frames is 1 using the formula: (t1 / t2) -1. Can be requested.

  As described above, according to the present embodiment, when there is no image data corresponding to the designated layer, the immediately preceding image data is repeatedly transmitted to the TV 2000. In this way, the apparent frame rate for the TV 2000 can be matched with the frame rate corresponding to the designated layer. As a result, it is possible to suppress a change in the frame rate of a signal output from HDMI (registered trademark), and as a result, it is possible to avoid an unexpected interruption of HDMI (registered trademark) connection during reproduction of a moving image.

  The imaging apparatus according to the present exemplary embodiment is an imaging apparatus that acquires hierarchically encoded moving image data from a recording medium, decodes the acquired moving image data, and outputs corresponding image data to an external display device. . Then, the imaging apparatus according to the present embodiment acquires hierarchical information corresponding to each frame of the moving image data, and does not decode data corresponding to a frame higher than the hierarchy corresponding to the display frame rate of the external display device. Like that. This corresponds to the processing of S6209, for example. Therefore, the imaging apparatus of the present embodiment can efficiently process the compressed moving image data in accordance with the display frame rate of the external display device. It is also possible not to switch the output frame rate for an external display device.

  In the second embodiment, the operation related to the reproduction processing of one moving image file has been described. However, the reproduction target is not limited to one file. It may be a playlist in which a plurality of moving image files are grouped, or all reproducible video data, for example, all moving images displayed on an index screen displaying a list of moving images. In this case, the search process range of the maximum temporal_id is set to the moving image data of all files included in the playlist or all video data included in all reproducible files.

  In addition, you may enable it to implement the process of Example 2 and the process of Example 1 both.

(Other embodiments)
The above-described embodiment can also be realized in software by a computer of a system or apparatus (or CPU, MPU, etc.). Therefore, the computer program itself supplied to the computer in order to implement the above-described embodiment by the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiments is also one aspect of the present invention.

  The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, but is not limited thereto. A computer program for realizing the above-described embodiment is supplied to a computer via a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a magnetic storage medium such as a flexible disk, a hard disk, and a magnetic tape, an optical / magneto-optical storage medium such as an MO, CD, and DVD, and a nonvolatile semiconductor memory.

  As a computer program supply method using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable format or a source code. Then, the program file is supplied by downloading to a client computer that has accessed the server. In this case, the program file can be divided into a plurality of segment files, and the segment files can be distributed and arranged on different servers. That is, a server apparatus that provides a client computer with a program file for realizing the above-described embodiment is also one aspect of the present invention.

  In addition, a storage medium in which the computer program for realizing the above-described embodiment is encrypted and distributed is distributed, and key information for decrypting is supplied to a user who satisfies a predetermined condition, and the user's computer Installation may be allowed. The key information can be supplied by being downloaded from a homepage via the Internet, for example. Further, the computer program for realizing the above-described embodiment may use an OS function already running on the computer. Further, a part of the computer program for realizing the above-described embodiment may be configured by firmware such as an expansion board attached to the computer, or may be executed by a CPU provided in the expansion board. Good.

Claims (12)

Acquisition means for acquiring video data that has been hierarchically encoded;
Decoding means for decoding moving image data obtained by the acquisition means;
Output means for outputting the moving image data decoded by the decoding means to an external display device;
Information acquisition means for acquiring information on a display frame rate of the external display device;
An image processing apparatus comprising: control means for controlling the decoding means so as not to decode data in a hierarchy higher than a hierarchy corresponding to a frame rate of the external display device included in the moving image data.   The image processing apparatus according to claim 1, further comprising: a hierarchy information acquisition unit that acquires information indicating a hierarchy of each frame of the moving image data acquired by the acquisition unit.   The image processing apparatus according to claim 2, wherein the information indicating the hierarchy is information that is included in the moving image data and can be acquired without performing inverse quantization processing of the moving image data.   The image processing apparatus according to claim 2, wherein the information indicating the hierarchy is information that is included in the moving image data and can be acquired without performing an inverse cosine transform process on the moving image data. .   The image processing apparatus according to claim 2, wherein the information indicating the hierarchy is temporal_id corresponding to each frame.   The image processing apparatus according to claim 5, wherein the temporal_id is stored in a Nal_Unit_Header corresponding to each frame.   The moving image data is H.264. 7. The image processing apparatus according to claim 1, wherein the image processing apparatus is moving image data that is hierarchically encoded by an encoding method according to the H.265 standard. The acquisition means acquires moving image data from a recording medium,
The control means does not decode data corresponding to frames in a layer higher than the layer corresponding to the display frame rate of the external display device for all the moving image data recorded in the recording medium. The image processing apparatus according to claim 1, wherein the decoding unit is controlled. The acquisition means acquires video data corresponding to a playlist indicating the playback order of the video data,
The hierarchy information acquisition means acquires information indicating a hierarchy of video data corresponding to the playlist,
The control means controls the decoding means so as not to decode data corresponding to a frame higher than a hierarchy corresponding to a display frame rate of the external display device for moving picture data corresponding to the playlist. The image processing apparatus according to claim 1, wherein:   The control means controls the decoding means so as not to decode data corresponding to a frame in a layer higher than a layer corresponding to a display frame rate of the external display device when the external display device is connected. The image processing apparatus according to claim 1, wherein the image processing apparatus is controlled. An acquisition step of acquiring hierarchically encoded video data;
A decoding step of decoding the moving image data obtained in the acquisition step;
An output step of outputting the video data decoded in the decoding step to an external display device;
An information acquisition step of acquiring information regarding a display frame rate of the external display device;
The image processing method according to claim 1, wherein in the decoding step, control is performed so that data in a layer higher than a layer corresponding to a display frame rate of the external display device included in the moving image data is not decoded. The computer program for operating a computer as each means of any one of Claim 1 to 10.

Images