JP5045213B2 - Imaging device, playback device, and recording file creation method - Google Patents

Description

  The present invention relates to, for example, an imaging apparatus having a position information acquisition function and a communication function, and a recording file creation method and playback apparatus used therefor, and particularly, generates and records a recording file in which at least moving image data and map image data are associated with each other. The present invention relates to a possible imaging device, a method of creating the recording file, and a reproducing device capable of reproducing based on the recording file.

  Conventionally, in an imaging apparatus such as a digital camera or a digital video camera, it is possible to present various information to a user in addition to a captured image, but map data is provided by a service provider or the like via the Internet. Various proposals have been made for an imaging apparatus that displays a map based on the map data.

  For example, Patent Document 1 discloses an information processing apparatus that displays a map even if there is no map data therein, and displays arbitrary data associated with position information on the map for effective use. That is, in this technology, image data, its position information, and additional information having link information to the map are stored in a memory stick in association with each other, and map information is acquired based on the link information when displayed. Is displayed on the LCD.

JP 2002-196667 A

  However, in the prior art, since position information at the time of shooting is recorded in content units (for example, units of moving image data shot from when shooting is started until shooting is stopped), the location at the time of moving image shooting is changed. In the case of a large movement, it may not be possible to display a map that reflects the real-time shooting location during moving image shooting during moving image playback.

  In other words, when displaying a corresponding map based on position information at the time of shooting, which is held as additional information of the moving image data, when playing a movie, on the map where the actual movie shooting location is displayed. In some cases, there was a discrepancy between the positions. Therefore, the user may not be able to accurately grasp the position at the time of moving image shooting during playback.

  It is an object of the present invention to display a map image reflecting a real-time shooting location during moving image shooting when playing back a shot moving image.

The image pickup apparatus according to the first aspect of the present invention can acquire map data from a map data providing server, the image pickup means picks up a subject image and outputs an image signal, and the image signal processing means processes the image signal. Then, moving image data is generated and output, the moving image compression means compresses and encodes the moving image data, the position / orientation information acquisition means acquires position / orientation information at the time of shooting by the imaging means, and the communication means The position / orientation information is transmitted to the map data providing server via an external network, the map image data corresponding to the position / orientation information is obtained, and the map image obtaining unit performs the process of obtaining the map image data. A map moving image file storing the map image data is generated while controlling, and the recording means records the map moving image file and the moving image file on the recording medium. , So that the control means generates the moving image file storing moving image data after the compression coding, the moving image file in correspondence with the map video files recorded on the recording medium by the recording means together with the map video file To control.

  Accordingly, a moving image file storing moving image data and a map moving image file storing acquired map image data are generated and recorded on a recording medium.

In the playback apparatus according to the second aspect of the present invention, a map moving image file including at least a moving image file and map image data corresponding to the position / orientation at the time of shooting of the moving image file is recorded on the recording medium, and the reproducing means is the recording medium. Then, the moving image file and the map moving image file are read out, the moving image decoding unit decodes the moving image data of the moving image file, the map image decoding unit decodes the map image data of the map moving image file, and the combining unit decodes the moving image data. The moving image data and the map image data are combined, the holding unit temporarily stores the data related to the combined image, and the display unit reads out and displays the data related to the stored image, and stores the data on the recording medium. Is a content management file that holds data related to the correspondence between video files and map video files. The control means, when reproduction by said reproducing means to identify the map video file corresponding to the moving image file based on the content management file.

  Accordingly, a corresponding map image is displayed during moving image reproduction.

A recording file creation / recording method according to a third aspect of the present invention is a moving image data obtained by photographing and map image data obtained from a map data providing server , wherein the moving image file has a position / orientation at the time of photographing. Corresponding map image data is recorded on a recording medium in a predetermined file format, and the moving image data obtained by shooting is compressed and encoded and stored in a moving image file, and the map image data is stored in a map moving image file. And storing the data related to the correspondence between the moving image file and the map moving image file in the content management file, and associating the moving image file, the map moving image file, and the content management file with the recording file As a recording medium.

  Accordingly, the correspondence between the moving image file storing the moving image data and the map moving image file storing the map image data is managed by the content management file.

  According to the present invention, it is possible to generate and record a recording file in which moving image data and map image data are associated with each other during moving image shooting, and display a map image reflecting a real-time shooting location during moving image shooting during moving image reproduction. An image pickup apparatus that performs recording, a method for creating the recording file, and a reproducing apparatus that can perform reproduction based on the recording file can be provided.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, a best mode for carrying out the invention (hereinafter simply referred to as an embodiment) will be described in detail with reference to the drawings.

  First, FIG. 1 shows and describes the configuration of an imaging apparatus according to the first embodiment of the present invention.

  As shown in FIG. 1, the imaging apparatus includes a microcomputer 13 that controls the entire system. The microcomputer 13 includes, for example, a so-called “built-in” microcomputer that incorporates a ROM (Read Only Memory), a RAM (Random Access Memory), a serial interface, or a parallel interface.

  The microcomputer 13 can communicate with each unit such as a buffer memory 5, a moving image compression unit 6, a pixel number conversion unit 7, a still image compression unit 12, an audio compression unit 8, a buffer memory 10, and a drive 11 via a control bus. It is connected. The outputs of the mode dial 14 and the recording start / stop button 15 are connected to the input of the microcomputer 13.

  The microcomputer 13 is also connected to a map image acquisition unit 18 so as to be communicable. The map image acquisition unit 18 is connected to the position / orientation information acquisition unit 16, the external network communication unit 17, and the drive 11 so as to be capable of communication. ing. In the imaging apparatus according to this embodiment, the map image acquisition unit 18 is also configured by a microcomputer. However, the function of the map image acquisition unit 18 can be assigned to the microcomputer 13.

  The imaging unit 1 includes, for example, an optical system such as a lens and a diaphragm, and an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The output of the imaging device is transmitted via a moving image input interface 2. Are connected to the buffer memory 5. The audio conversion unit 36 is composed of, for example, a microphone and the output thereof is connected to the buffer memory 5 via the audio input interface 4.

  The buffer memory 5 is connected to a moving image compression unit 6, a pixel number conversion unit 7, and an audio compression unit 8, and outputs of the moving image compression unit 6 and the audio compression unit 8 are connected to a multiplexer 9. The output of the pixel number conversion unit 7 is connected to the still image compression unit 12, and the output of the still image compression unit 12 is connected to the buffer memory 10. The multiplexer 9 is connected to the buffer memory 10, and the buffer memory 10 is connected to the drive 11 so as to be able to communicate. The buffer memories 5 and 10 are made of, for example, a semiconductor memory.

  In such a configuration, the microcomputer 13 executes the control program stored in the built-in ROM to control the entire recording apparatus. This microcomputer 13 functions as a control means, for example.

  In the photographing unit 1, a subject image incident through an optical system such as a lens and a diaphragm is formed on an image sensor, and the image sensor supplies an image signal related to a moving image to the moving image input interface 2. The imaging device of the photographing unit 1 functions as an imaging unit, for example.

  The moving image input interface 2 performs analog / digital conversion on the image signal supplied from the image sensor of the photographing unit 1, performs serial / parallel conversion, converts the image signal into moving image data of a predetermined method, and moves the moving image Image data is supplied to the buffer memory 5. The moving image input interface 2 functions as, for example, an image signal processing unit.

  On the other hand, the voice conversion unit 3 acquires the voice from the subject or the surrounding voice and supplies a voice signal corresponding to the acquired voice to the voice input interface 4. This audio signal is synchronized with the image signal output from the photographing unit 1. The audio input interface 4 converts the audio signal from analog to digital, serial / parallel conversion, converts the audio signal into audio data of a predetermined system, and supplies the audio data to the buffer memory 5.

  The buffer memory 5 temporarily stores the moving image data supplied from the moving image input interface 2 and the audio data supplied from the audio input interface 4.

  The buffer memory 5 supplies the stored moving image data to the moving image compression unit 6 and the pixel number conversion unit 7, and supplies the stored audio data to the audio compression unit 8. The moving image compression unit 6 compresses and encodes the moving image data supplied from the buffer memory 5 under a control of the microcomputer 13 by a predetermined method such as MPEG (Moving Pictures Experts Group) 2 method. The encoded moving image data is supplied to the multiplexer 9. The moving image compression unit 6 functions as, for example, a moving image compression unit.

  The pixel number conversion unit 7 extracts a predetermined picture (frame or field) from the moving image data under the control of the microcomputer 13 and converts the number of pixels of the picture.

  For example, the pixel number conversion unit 7 converts the number of pixels in the picture by thinning out pixels from the extracted picture. For example, as illustrated in FIG. 4, the pixel number conversion unit 7 extracts one frame from two GOPs in the moving image data compressed and encoded by the moving image compression unit 6 using the MPEG2 method. More specifically, for example, when the moving image compression unit 6 compresses and encodes a moving image of 30 frames per second in units of 15 frames of GOPs, the pixel number conversion unit 7 has 2 One frame is extracted from 30 frames constituting one GOP, and the number of pixels is converted by thinning out pixels from the frame. The image number conversion unit 7 supplies the image data obtained by converting the number of pixels to the still image compression unit 12.

  The still image compression unit 12 encodes the image data supplied from the pixel number conversion unit 7 under the control of the microcomputer 13 by a compression encoding method that compresses a still image such as a JPEG (Joint Photographic Experts Group) method. Turn into. The still image compression unit 12 supplies the encoded image data to the buffer memory 10 as thumbnail image data.

  On the other hand, the audio compression unit 8 compresses audio data supplied from the buffer memory 5 under a control of the microcomputer 13 by a predetermined method such as an AC3 (Audio Code Number 3 (Dolby Digital (registered trademark)) method. The encoded and compressed audio data is supplied to the multiplexer 9. The audio data output from the audio compression unit 8 is synchronized with the image data output from the moving image compression unit 6. The audio compression unit 8 supplies the compression-encoded audio data to the multiplexer 9.

  The multiplexer 9 multiplexes the moving image data supplied from the moving image compression unit 6 and the audio data supplied from the audio compression unit 8 as a moving image stream such as an MPEG2 system stream. (Consisting of moving image data and audio data) is supplied to the buffer memory 10. The buffer memory 10 temporarily stores the moving image stream (consisting of moving image data and audio data) supplied from the multiplexer 9 and the thumbnail image data supplied from the still image compression unit 12.

  The microcomputer 13 executes a predetermined control program and arranges a file system for various data stored in the buffer 10 to obtain a predetermined file system to be described later.

  The drive 11 reads the multiplexed video stream (moving image data and audio data) and thumbnail image data in a predetermined file format from the buffer memory 10 and records them on the recording medium 19. As the recording medium 19, various media such as a magnetic disk, an optical disk, or a magneto-optical disk can be adopted. The drive 11 functions as a recording unit, for example.

  At the time of moving image shooting, the position / orientation information acquisition unit 16 acquires the position / orientation information of the imaging device 100 under the control of the map image acquisition unit 18. More specifically, the position / orientation information acquisition unit 16 acquires the position information of the imaging apparatus 100 at the time of shooting using, for example, a GPS (Global Positioning System) function. That is, the position (latitude / longitude) on which the imaging apparatus 100 is located on the earth is detected based on the distance obtained using the time until the radio wave arrives from the artificial satellite 103. As a result, information such as latitude, longitude, and direction is acquired as the position / orientation information. The position / orientation information acquisition unit 16 functions as, for example, a position / orientation information acquisition unit.

  The position / orientation information acquisition unit 16 supplies the position / orientation information to the map image acquisition unit 18. The map image acquisition unit 18 transmits the position / orientation information to the map data providing server 101 via the external network communication unit 17 and the external network 102 such as the Internet, and the map data corresponding to the position / orientation information. Get reference destination information. This map data reference destination information is described by URL (uniform resource locator) etc., for example.

  When the map image acquisition unit 18 can acquire the map data reference destination information, the map image acquisition unit 18 accesses the reference destination and acquires the map image data. The map image data is supplied to the drive 11 and is recorded on the recording medium 19 by the drive 11.

  That is, the map image acquisition unit 18 converts the map image data into a predetermined file format as will be described in detail later. Then, the map image acquisition unit 18 drives the driver 11 while taking a correspondence relationship with the moving image file in the buffer memory 10, and records it on the recording medium 19 in association with the one folder generated by the microcomputer 13. Control. This map image acquisition part 18 functions as a map image acquisition means, for example.

  In the course of the above processing, when the microcomputer 13 receives a signal from the recording start / stop button 15 or the mode dial 14 according to the user's operation, the microcomputer 13 instructs each unit to operate based on the signal. For example, when there is a normal mode and a special mode, the microcomputer 13 can detect from the signal whether any mode is desired by operating the mode dial 14, and map image data to be acquired according to the mode. It is also possible to switch the type. That is, when the special mode is selected, for example, map image data for 3D display (3D) or map image data of various scales can be selected and acquired as this type. It is not limited. Note that when the normal mode is selected, the normal image capturing operation is performed without acquiring the map image data.

  Here, FIG. 2 shows an example of a file structure of moving image content recorded on the recording medium 19 by moving image shooting.

  As shown in FIG. 2, the moving image content recorded on the recording medium 19 by shooting a moving image stores a content management file for managing a file group, for example, a moving image stream (consisting of moving image data and audio data) according to the MPEG2 system. Video files (for example, GAZO001.MPG, GAZO002.MPEG ...), map video files (for example, TIZU001.MAP ...) that store map video streams (consisting of map image data) corresponding to the video files, and these video streams and maps It consists of a map video management file (for example, TIZU001.MNR) that holds the relationship with the video stream. These moving image file, map moving image file, and content management file correspond to, for example, recording files.

  The correspondence relationship between the moving image file, the map moving image file, and the map moving image management file is recorded by the content management file in a file system as shown in FIG.

  The content management file includes a header and a body part, and the body part includes image file management information for managing each file group. This image file management information includes a moving image file name, a map moving image file name, and a map moving image management file name. Therefore, the microcomputer 13 can easily and quickly specify the map moving image file corresponding to the moving image file based on the image file management information.

  The relationship between the moving image stream and the map moving image stream is as shown in FIG.

  In FIG. 4, the first frame constituting each GOP is shown in black.

  As shown in the figure, map image data of the map moving image stream is associated with a predetermined frame of the moving image stream. For example, in this embodiment, map image data P1 is assigned to the first frame F1 of the moving image stream.

  Next, when the threshold value for acquiring the map image data is 2 GOP, for example, when the first map image data has already been acquired when recording the video stream after 2 GOP, the map image corresponding to the first frame F2 of 3 GOP Data P2 is assigned. Thereafter, when one map image data can be acquired every 2 GOPs, for example, map image data P3 is allocated to the frame F3, the map image data is recorded in association with each 2 GOPs.

  Next, FIG. 5 shows the file structure of the map moving image file, and FIG. 6 shows the file structure of the map moving image management file, which will be described in detail. As shown in FIG. 5, the map moving image file stores a map moving image stream including a plurality of map image data.

  As shown in FIG. 6, the map moving image management file includes a header portion and a body portion, and map image management information is stored in the body portion. This map image management information includes moving image stream frame information indicating which frame in the moving image stream corresponds to the map image data, position / orientation information acquired from the position / orientation information acquisition unit 16, and from the map data providing server 101. It consists of map data reference destination information for acquiring a map moving image stream made up of map image data, the top address of map image data stored in a map moving image file, and its data size. Therefore, by referring to the map moving image management file, it is possible to easily and quickly detect which frame of the moving image stream corresponds to the map image data, and also the storage location of the map image data.

  Hereinafter, with reference to the flowcharts of FIGS. 7 to 10, processing by the imaging apparatus according to the first embodiment of the present invention will be described in detail. Part or all of the following processing corresponds to a recording file creation method.

  First, with reference to the flowchart of FIG. 7, the map moving image recording process by the imaging device according to the first embodiment of the present invention will be described in detail.

  When the map moving image recording process is started, the moving image compression unit 6 compresses and encodes the moving image data supplied from the buffer memory 5 under the control of the microcomputer 13, and the compressed and encoded moving image data is sent to the multiplexer 9. Send out (step S1).

  Subsequently, the audio compression unit 8 compresses and encodes the audio data supplied from the buffer memory 5, and sends the compressed and encoded audio data to the multiplexer 9 (step S2).

  The multiplexer 9 multiplexes the moving image data and audio data, and sends them to the buffer memory 10 as a moving image stream. The drive 11 reads the moving picture stream from the buffer memory 10 and records it on the recording medium 19 (step S3). Details of the recording process of the moving image stream in step S3 will be described later with reference to FIG.

  Subsequently, the microcomputer 13 determines whether the previous acquisition of the map image data has been completed (step S4). This determination is made until the external map communication unit 17 completes the process of acquiring the map image data by communicating with the map data providing server 101 via the external network 102 such as the Internet. This is because it cannot be transferred to the acquisition process. If the previous acquisition of map image data has not been completed in step S4 (step S4 branches to No), the process returns to step S1 to perform compression encoding of moving image data and compression encoding of audio data. repeat.

  On the other hand, if it is determined that the previous acquisition of the map image data has been completed (step S4 branches to Yes), it is determined whether or not the number of recorded GOPs in the video stream has exceeded the threshold (step S5). In step S5, when the number of recording GOPs of the moving image stream does not exceed the threshold value (step S5 branches to No), the process returns to step S1 and the above processing is repeated. On the other hand, when the number of recorded GOPs of the moving image stream exceeds the threshold (step S5 branches to Yes), the map image acquisition unit 18 performs the next map image data acquisition process (step S6), and the above steps The processes after S1 are repeated. Details of the map image data acquisition process in step S6 will be described later with reference to FIG.

  As described above, in this map moving image recording process, each process is executed by the moving image compression unit 6 and the audio compression unit 8 and the acquisition of the previously processed map image is completed, and the previous map image acquisition is performed. When the number of recorded GOPs of the moving image stream exceeds the threshold after the execution of the process, the map image acquisition process by the map image acquisition unit 18 is executed.

  Here, with reference to the flowchart of FIG. 8, the moving image stream recording process in step S3 of FIG. 7 will be described in more detail.

  The microcomputer 13 acquires the data amount of the moving image stream stored from the buffer memory 10, and the data amount of the moving image stream stored in the buffer memory 10 is equal to or greater than a predetermined moving image stream recording start threshold. It is determined whether or not (step S11). Here, the determination process is repeated until the data amount of the moving image stream becomes equal to or greater than the system stream recording start threshold (step S11 branches to No).

  On the other hand, when it is determined in step S11 that the data amount of the moving image stream stored in the buffer memory 10 is equal to or greater than the moving image stream recording start threshold (step S11 branches to Yes), the microcomputer 13 Then, the moving image stream stored in the buffer memory 10 is recorded on the recording medium 19 by one cluster (step S12). Subsequently, the microcomputer 13 determines whether or not the data amount of the moving image stream stored in the buffer memory 10 has become less than one cluster (step S13). Here, when it is determined that the data amount of the moving image stream stored in the buffer memory 13 is not less than one cluster, the moving image stream can be further recorded in the stream unit, and the process returns to step S12. Then, the process of recording the moving image stream to the cluster is repeated (step S13 branches to No).

  Through the loop of steps S12 and S13, the moving image stream is recorded up to the end of the stream unit. On the other hand, if it is determined in step S13 that the data amount of the moving image stream stored in the buffer memory 10 has become less than one cluster (step S13 branches to Yes), the moving image stream is further recorded in the stream unit. Therefore, in order to wait until the moving image stream is accumulated in the buffer memory 10, the process returns to step S11 and the above-described processing is repeated.

  With the above processing, when the data amount of the moving image stream on the buffer memory 10 exceeds the threshold, the moving image stream can be recorded on the recording medium 19.

  Next, the map image data acquisition process executed in step S6 of FIG. 7 will be described in more detail with reference to the flowchart of FIG.

  When the map image data acquisition process is started, the position / orientation information acquisition unit 16 acquires position / orientation information (step S21). Subsequently, the map image acquisition unit 18 communicates with the map data providing server 101 via the external network 102 such as the Internet by the external network communication unit 17 to acquire map image data (step S22).

  When acquiring the map image data, the map image acquiring unit 18 drives and controls the drive 11, and records the map image data in the map moving image file on the recording medium 19 (step S23). Then, the map image acquisition unit 18 records the map image management information as described above in detail in the map moving image management file (step S24), and ends the process.

  With the above processing, map image data can be recorded in the map moving image file on the recording medium 19, and further map image management information can be recorded in the map moving image management file.

  The data provided from the map data providing server 101 is, for example, an image file in the JPEG file format. In addition, an image in which instruction information in which a shooting position and a shooting direction on a map are recorded is combined is provided. When an image to which instruction information is not combined is provided, the instruction information may be combined by an image combining unit during reproduction as described later. The map image data provided from the map data providing server 101 may be obtained by combining character information such as place name information and building information, or landmark information serving as a mark representing a station or the like, with a map image.

  Next, map image data reacquisition processing by the imaging apparatus according to the first embodiment of the present invention will be described with reference to the flowchart of FIG.

  When the map image data reacquisition process is started, the microcomputer 13 confirms the position information presence flag state of the map image management information in the map moving image management file of the content management file (step S310), and the position information presence flag is set. If “false” (step S31 branches to No), the process returns. On the other hand, if the position information presence flag is “true”, the microcomputer 13 checks whether map data reference destination information is set (step S32). Here, when the map data reference destination information is not set, the microcomputer 13 acquires the map data reference destination information (step S33).

  Subsequently, it is determined whether or not map data reference destination information has been acquired (step S34). Here, when it is determined that the information could not be acquired (step S34 is branched to No), this process is ended and the process returns. On the other hand, if it is determined that the map data reference destination information has been acquired (step S34 is branched to Yes), the process proceeds to step S35. Moreover, also when map data reference destination information is set by step S32 (step S32 is branched to Yes), it transfers to step S35. In this way, in step S35, map image data is acquired based on the map data reference destination information, and this process ends and returns.

  Next, FIG. 11 shows and describes the configuration of the reproduction system of the imaging apparatus according to the first embodiment of the present invention. This playback system configuration also functions as a playback device, for example.

  The imaging apparatus has a microcomputer 67 that controls the entire system.

  The microcomputer 67 is composed of, for example, a so-called “built-in” microcomputer that incorporates a ROM, a RAM, a serial interface, a parallel interface, or the like. The microcomputer 67 includes a drive 52, a buffer memory 53, a moving image expansion unit 55, a map image expansion unit 56, a file format conversion unit 57, an audio expansion unit 58, a buffer memory 59, a resolution conversion unit 60, an image composition unit 61, and a buffer. Controllably connected to the memory 62 via a control bus. The outputs of the mode dial 69 and the playback start / stop button 68 are connected to the input of the microcomputer 67.

  In such a configuration, the microcomputer 67 executes a predetermined control program stored in the ROM, and instructs each unit to operate based on a signal from the reproduction start / stop button 68 according to a user operation. The microcomputer 67 functions as a control unit, for example. Under the control of the microcomputer 67, the drive 52 reads a moving image stream or the like from the attached recording medium 51 and stores it in the buffer memory 53. The drive 51 functions as a reproducing unit, for example.

  The demultiplexer 54 separates the moving image data and the audio data multiplexed in the moving image stream stored in the buffer memory 53, and supplies the separated moving image data to the moving image decompression unit 55 for separation. The audio data is supplied to the audio expansion unit 58. The demultiplexer 54 functions as a separating unit, for example.

  The moving image decompression unit 55 decodes the compressed and encoded moving image data supplied from the demultiplexer 54 under the control of the microcomputer 67, decompresses it, and sends it to the resolution conversion unit 61. The moving image decompression unit 55 corresponds to, for example, a moving image decoding unit.

  The resolution conversion unit 60 converts the resolution of the decoded moving image data and sends it to the image composition unit 61. The map image decompression unit 56 decodes the map image data of the map moving image stream, decompresses it, and supplies it to the buffer memory 59. This map image expansion unit 56 functions as, for example, a map image decoding unit. The map image data in the buffer memory 59 is sent to the image composition unit 61. Then, the moving image data and the map image data are combined and sent to the buffer memory 62. The image composition unit 61 functions as a composition unit, for example.

  The still image expansion unit 70 acquires thumbnail image data from the buffer memory 53 storing thumbnail image data under the control of the microcomputer 67, and decodes the thumbnail image data compressed and encoded by the JPEG method or the like. Thus, the decompressed thumbnail data is supplied to the buffer memory 62.

  The audio decompression unit 58 decompresses the audio data supplied from the demultiplexer 54 under the control of the microcomputer 67 and is compressed and encoded by, for example, the AC3 method, and buffers the decoded audio data. This is supplied to the memory 62.

  The image output interface 63 converts the image data supplied to the buffer memory 62 into an image signal of a predetermined method that can be used by the display unit 64, such as parallel-serial conversion, and displays the image signal. Supplied to the unit 64.

  The display unit 64 includes, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display device, and displays a moving image and a still image based on an image signal supplied via the image output interface 63. become. The display unit 64 functions as a display unit, for example.

  The audio output interface 65 converts the audio data supplied from the buffer memory 62 into an audio signal of a predetermined method that the audio output unit 66 can use, such as parallel-serial conversion or digital-analog conversion. Then, the audio signal is supplied to the audio output unit 66. The audio output unit 66 includes an audio amplifier or a loudspeaker, and outputs audio based on an audio signal supplied via the audio output interface 65.

  Hereinafter, with reference to the flowchart of FIG. 12, the map image and moving image reproduction process by the imaging apparatus according to the first embodiment of the present invention will be described in detail.

  When the process is started, the microcomputer 67 determines whether the display target frame is moving image data (frame) associated with the map image data and the map image management information (step S51). Here, when it is not the moving image data (frame) matched with map image data (step S51 is branched to No), it transfers to step S55.

  On the other hand, when it is the moving image data (frame) associated with the map image data in step S51 (step S51 branches to Yes), the microcomputer 67 determines from the map image management information in the map moving image management file. The video stream frame information associated with the map image data is acquired (step S52), the corresponding map image data is acquired from the map video file (step S53), and the map image data is expanded by the map image expansion unit 56. Then, it is recorded in the buffer memory 59 (step S54).

  Subsequently, the microcomputer 67 expands the moving image data separately supplied from the demultiplexer 54 by the moving image expansion unit 55 (step S55), and changes the resolution of the moving image data by the resolution changing unit 60 (step S56). ) The map image data and the moving image data are combined by the image combining unit 61 (step S57). The audio data separated and supplied from the demultiplexer 54 is expanded by the audio expansion unit 58 (step S58), and the combined image is reproduced by the display unit 64 (step S59). Is reproduced (step S60). Thereafter, the process returns to step S51, and the above process is repeated until the process shifts to an interrupt process based on the operation of the reproduction start / stop button 68.

  Through the above process, the moving image stream of the moving image file recorded on the recording medium 51 and the map moving image stream of the map moving image file are reproduced in synchronization.

  Here, a display example of the display unit 64 when the thumbnail images of the moving image content are displayed as a list is as shown in FIG. 13, for example. In the playback mode, the user can select the normal playback mode and the playback mode with map by appropriately selecting the icons 200c and 200d by operating the mode dial 69. From the moving image list display on the screen 200, the moving image content desired to be reproduced is selected by referring to the moving image thumbnail displayed in the display area 200a and moving and selecting the cursor 200b. When the moving image content to be reproduced is selected in the state where the map-added reproduction mode is selected, the reproduction processing of the map moving image stream is executed on the selected moving image content in synchronization with the moving image stream.

  A display example when a moving image and a map moving image are reproduced by selecting the mode dial 69 is as shown in FIG. 14, for example. In this case, the moving image stream and the map moving image stream are combined and reproduced on the screen 201 of the display unit 64. Specifically, the moving image stream is reproduced in the moving image reproduction area 201a, and the map moving image stream is reproduced in the map image display area 201b. As described above, the map image is updated for each frame specified by the threshold value of the moving image stream, and the moving image stream and the map moving image stream are reproduced in synchronization. At this time, character information such as place name information and building information, landmark information serving as a mark representing a station, etc., and an index 201c based on position / orientation information are combined with a map image.

  As described above, according to the imaging apparatus according to the first embodiment of the present invention, the user refers to a map reflecting accurate position / direction information at the time of shooting with respect to the moving image, and adds the additional information of the moving image. Since it becomes possible to know more accurately, convenience can be improved.

  More specifically, for example, for each frame of several GOPs in the video stream, map image data corresponding to the shooting position of the frame and position / orientation information at the time of shooting are recorded and stored in association with the frame. Thus, when reproducing the moving image stream, the position / orientation information at the time of moving image shooting can be displayed in real time when the map image data corresponding to the additional information of the moving image stream is reproduced in synchronization with the moving image stream. Therefore, the user can grasp more accurate position / orientation information at the time of shooting.

  In addition, since the location name information and building information on the map image reflecting the real-time shooting position can be referenced during video playback, the location name information and building information corresponding to the image being played back at that time are more accurate. It becomes possible to grasp. Furthermore, when the means for acquiring the map image from the shooting position information uses the external communication function, the map image data synchronized with the video stream is recorded on the recording medium at the time of video shooting. Sometimes, it is possible to reproduce a map image even in a situation where the external communication function cannot acquire map image data due to poor communication. Further, regarding the means for acquiring map image data from the position / orientation information at the time of shooting, even if the map image data providing function is to sequentially update the map information included in the generated map image. In this embodiment, since map image data at the time of shooting is recorded, map image data reproduced in synchronization with the moving image stream can be displayed as accurate information.

  Next, a second embodiment of the present invention will be described with reference to FIGS.

  In the first embodiment described above, the external network communication unit 17 acquires the corresponding map image data from the map data providing server 101 based on the position / orientation information acquired by the position / orientation information acquisition unit 16. However, the type of map image data to be acquired was not particularly mentioned. In contrast, the imaging apparatus according to the second embodiment acquires a plurality of map image data according to a plurality of scales and map image data according to a three-dimensional (3D) display, and a plurality of scales and viewpoint maps. It is possible to display an image.

  Here, the viewpoint of the map includes a two-dimensional image when the map is viewed from above and a 3D image when the map is viewed in the shooting direction.

  Note that the hardware configuration of the recording system and display system of the imaging apparatus according to the second embodiment is the same as that of the first embodiment (FIGS. 1 and 11). Explanation will be made using the same reference numerals. As a premise, the map data providing server 101 can provide map image data of a specified scale / viewpoint.

  Here, FIG. 15 shows an example of a file structure of moving image content recorded on the recording medium 19 by moving image shooting. The moving image content recorded on the recording medium 19 by shooting a moving image corresponds to a content management file that manages a group of files to be generated, for example, a moving image file (for example, GAZO001.MPG) that stores a moving image stream according to the MPEG2 system. Map video files (for example, TIZU001.MAP, TIZU002.MAP, TIZU003.MAP) that store the map video stream, and a map video management file (for example, TIZU001.MNR) that holds the relationship between the video stream and the map video stream. . In particular, as the map moving image file, files relating to a plurality of scales / viewpoints are provided.

  As a second embodiment in particular, as shown in FIG. 16, the image file management information includes a plurality of scales / viewpoints in addition to the movie file name, the map movie file name, and the map movie management file. The type information is stored as “map moving image type information”. That is, during the recording process, it is possible to record the specified scale / viewpoint type together with the image file management information. In FIG. 16, although the illustration is simplified, this map moving image type information represents a scale / viewpoint type for each map moving image file, and therefore is provided individually for each map moving image file. Become. At the time of reproduction processing, the image composition unit 61 makes it possible to compose and display the scale / viewpoint information.

  This display example is as shown in FIG.

  In this case, the moving image stream and the map moving image stream are combined and reproduced on the screen 202 of the display unit 64. Specifically, the moving image stream is reproduced in the moving image reproduction area 202a, and the map moving image stream is reproduced in the map image display area 202b. As described above, the map image is updated for each frame specified by the threshold value of the moving image stream, and the moving image stream and the map moving image stream are reproduced in synchronization. At this time, character information such as place name information and building information, landmark information serving as a mark representing a station, etc., and an index 202c based on direction information are combined with a map image. Then, the scale / viewpoint of the displayed map can be changed as appropriate by pressing the “3D”, “enlarge”, and “reduced” icons 202d to 202f on the right end of the screen.

  As described above, according to the second embodiment of the present invention, it is possible to improve user convenience by enabling recording of additional information of a video stream preferred by the user.

  The embodiment of the present invention has been described above, but the present invention is not limited to this, and it is needless to say that various improvements and changes can be made without departing from the spirit of the present invention.

  For example, in the above-described embodiment, an example of application to an imaging apparatus such as a digital camera or a digital video camera has been shown. Of course, it is applicable.

The block diagram of the imaging device which concerns on the 1st Embodiment of this invention. FIG. 3 is a diagram showing a file structure of moving image content recorded on a recording medium by moving image shooting by the imaging apparatus according to the first embodiment of the present invention. The figure which shows an example of the file structure of a content management file. The figure which shows the relationship between a video stream and a map video stream. The figure which shows the file structure of a map moving image file. The figure which shows the file structure of a map moving image management file. The flowchart explaining in detail the map moving image recording process by the imaging device which concerns on the 1st Embodiment of this invention. FIG. 8 is a flowchart for explaining in more detail the processing of recording a moving image stream in step S3 of FIG. 8 is a flowchart for explaining in more detail the map image data acquisition process executed in step S6 of FIG. The flowchart explaining the re-acquisition process of the map image data by the imaging device which concerns on the 1st Embodiment of this invention. 1 is a configuration diagram of a reproduction system of an imaging apparatus according to a first embodiment of the present invention. 5 is a flowchart for explaining in detail map image and moving image reproduction processing by the imaging apparatus according to the first embodiment of the present invention. The figure which shows the example of a display when the thumbnail image of a moving image content is displayed as a list. The figure which shows the example of a display at the time of reproducing | regenerating a moving image and a map moving image. The figure which shows an example of the file structure of the moving image content recorded on a recording medium by imaging | photography of the moving image by the imaging device which concerns on the 2nd Embodiment of this invention. The figure which shows the content of image file management information. The figure which shows the example of a display at the time of reproducing | regenerating a moving image and a map moving image.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image pick-up part, 2 ... Moving image input interface, 3 ... Audio | voice conversion part, 4 ... Audio | voice input interface, 5 ... Buffer memory, 6 ... Movie compression part, 7 ... Pixel number conversion part, 8 ... Audio | voice compression part, 9 ... Multiplexer, 10 ... Buffer memory, 11 ... Drive, 12 ... Still image compression unit, 13 ... Microcomputer, 14 ... Mode dial, 15 ... Recording start / stop button, 16 ... Position / orientation information acquisition unit, 17 ... External network communication unit 18 ... Map image acquisition unit, 19 ... Recording medium, 51 ... Recording medium, 52 ... Drive, 53 ... Buffer memory, 54 ... Multiplexer, 55 ... Moving image decompression unit, 56 ... Map image decompression unit, 57 ... File format conversion 58: Audio decompression unit, 59 ... Buffer memory, 60 ... Resolution conversion unit, 61 ... Image composition unit, 62 ... Buffer memo 63 ... Image output interface, 64 ... Display unit, 65 ... Audio output interface, 66 ... Audio output unit, 67 ... Microcomputer, 68 ... Playback start / stop button, 69 ... Mode dial, 70 ... Still image expansion unit, 100 ... Imaging device (recording system), 101 ... Map data providing server, 102 ... External network, 103 ... Artificial satellite, 104 ... Imaging device (reproduction system)

Claims (7)

In an imaging device capable of acquiring map image data from a map data providing server,
Imaging means for capturing a subject image and outputting an image signal;
Image signal processing means for processing the image signal to generate and output moving image data;
Moving image compression means for compressing and encoding the moving image data;
Position / orientation information acquisition means for acquiring position / orientation information at the time of shooting by the imaging means;
Communication means for transmitting the position / orientation information to the map data providing server via an external network and acquiring map image data corresponding to the position / orientation information;
Map image acquisition means for generating a map moving image file storing the map image data while controlling the processing of acquiring the map image data;
Recording means for recording the map moving image file and moving image file on the recording medium;
A moving image file storing the moving image data after the compression encoding is generated, and the moving image file is associated with the map moving image file and controlled to be recorded on the recording medium by the recording means together with the map moving image file. An image pickup apparatus comprising: a control unit. Display means for reading out the moving image file and the map moving image file recorded on the recording medium, and displaying at least the moving image data stored in the moving image file and the image related to the map image data stored in the map moving image file Have
The thumbnail image of the moving image data is displayed at the time of display by the display means, and an icon for instructing display of map image data corresponding to the thumbnail image is displayed together with the thumbnail image. The imaging apparatus according to 1. The video recording file and the map video file are stored on the recording medium, and a content management file holding data relating to the correspondence between the video file and the map video file is also stored. The imaging device described. In case of failure to obtain the map image data corresponding to the position / orientation information by the map image acquisition unit during the shooting, the map image acquisition means, the position / orientation information again via the external network The imaging apparatus according to claim 2, further comprising: transmitting to the map data providing server and acquiring map image data corresponding to the position / orientation information. Display means for reading out the moving image file and the map moving image file recorded on the recording medium, and displaying at least the moving image data stored in the moving image file and the image related to the map image data stored in the map moving image file Have
The map image acquisition means acquires map image data relating to a plurality of scales and viewpoints from the map data providing server, and when displaying by the display means, the map image relating to the map image data relating to the plurality of scales and viewpoints The image pickup apparatus according to claim 1, wherein the image pickup device can be selectively displayed. A recording medium on which a map video file including at least a video file and map image data corresponding to the position / orientation at the time of shooting the video file is recorded;
Reproducing means for reading out the moving image file and the map moving image file from the recording medium,
Moving image decoding means for decoding moving image data of the moving image file;
Map image decoding means for decoding the map image data of the map moving image file;
Combining means for combining the decoded moving image data and map image data;
Holding means for temporarily holding data relating to the synthesized image;
Display means for reading out and displaying data relating to the held image;
The recording medium also stores a content management file that holds data relating to the correspondence between the moving image file and the map moving image file, and corresponds to the moving image file based on the content management file when played by the playing means. And a control means for specifying a map moving image file. The moving image data obtained by shooting and the map image data obtained from the map data providing server, and the map image data corresponding to the position / orientation at the time of shooting of the moving image file are recorded on the recording medium in a predetermined file format. A way to
Video data obtained by shooting is compressed and encoded and stored in a video file.
Store the map image data in a map video file,
Data related to the correspondence between the video file and the map video file is generated and stored in a content management file, and the video file, the map video file, and the content management file are associated with each other and recorded as a recording file on a recording medium. A recording file creation method characterized by:

Images