JP2022176202A - Picture processing apparatus and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging system and an imaging method, for recording and reproducing a picture indicating positional information and a picture imaged by a user, as a single moving picture, without performing complex processing.

SOLUTION: A video camera comprises: means for inputting a normal imaged picture; means for inputting a picture from another video input source; and a picture input arbitration unit that arbitrates and captures video data from the two input means and stores the video data in a memory. Thereby, by taking, as the other video input source, a video output terminal of car navigation, a screen (map information) for car navigation and a picture imaged by a user are capable of being captured in the memory. In the case that the video camera is one for monitoring, map image data can be similarly captured in the memory by taking, as the other video input source, a medium storing the map image data in advance, such as a USB flash memory or an SD card, and reading means for reading the map image data from the medium. By performing compression/encoding processing on data in the memory as a moving picture, map information and a picture imaged by a user are capable of being processed as a single moving picture.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to a system and method for recording and reproducing an image indicating position information and a captured image of a user as one moving image.

In recent years, the digitalization of still cameras and video cameras has progressed, making it possible for users to shoot and record very easily, and to record a large amount of video information. For this reason, for example, when a person goes on a long trip or the like, when he or she looks at the captured images after the trip, it is conceivable that the user may lose track of which images were captured where. At this time, if the map information and the captured image are included in one piece of image data, it is very convenient for later viewing and sorting. In addition, it is very convenient if the image indicating the location information and the captured image are included in one piece of image data, even for the data captured by surveillance video cameras installed in multiple locations.

As a technology similar to this, there is a technology of a driving recorder, which has started to be installed in taxis and the like in recent years. The driving recorder automatically determines the timing of photographing and records situation data including position information and photographing data. Techniques have been invented to prevent the recording of a large amount of unnecessary data due to the automatic determination of the photographing timing. For example, in Patent Document 1, a learning function is used to select recording data, and the user learns whether or not photography is necessary by inputting necessary/unnecessary information after photographing.

Japanese Unexamined Patent Application Publication No. 2007-11907

However, since shooting is performed automatically, there is a possibility that it will not be completely matched to the needs of the user, and it is not suitable for the user to freely shoot a desired image together with the map information.

SUMMARY OF THE INVENTION The present invention has been made in view of the problems described above, and provides a system and method for recording and reproducing an image indicating position information and an image captured by the user as a single moving image without performing complicated processing. offer.

The video camera is provided with a means for inputting an image from another video input source in addition to a means for inputting an ordinary photographed image, and the video data from these two input means are arbitrated and fetched and stored in a memory. An image input arbitration unit is provided. As a result, by using the video output terminal of the car navigation system as the other video input source, the screen of the car navigation system (map information) and the image captured by the user can be captured in the memory together. In a video camera for surveillance use, a medium in which map image data is stored in advance, such as a USB flash memory or an SD card, and a reading means for reading out the map image data from the medium can be used as separate video input sources. , can also be captured in memory. By compressing and encoding the data in this memory as a moving image, the image indicating the position information and the captured image of the user can be processed as one moving image.

According to the present invention, it is possible to record/reproduce an image indicating position information and a photographed image of the user as one moving image without performing complicated processing.

Schematic diagram of the image recording/reproducing system of the first embodiment Flow chart of the image recording/playback system of the first embodiment Schematic diagram of stream data generated by the image recording/playback system of Example 1 Schematic diagram of image recording/playback system of embodiment 2 Flow chart of image recording/playback system of embodiment 2 Schematic diagram of stream data generated by the image recording/playback system of Example 2 Configuration diagram of the image recording/reproducing system of the third embodiment Schematic diagram of a system for transferring stream data over hotspots Thumbnail display example

The best mode for carrying out the present invention will be described.

Hereinafter, embodiments of the image recording/reproducing system and method of the present invention will be described with reference to the drawings.

Example 1 will be described. In this embodiment, the operation from when the user issues a photographing instruction to when photographing is performed will be described with reference to FIGS. 1 to 3. FIG. FIG. 1 shows an image recording/reproducing system according to a first embodiment of the present invention. A video camera 1 allows a user to directly input various instructions including a photographing instruction, such as a button or touch panel. a control unit 12 for controlling each unit related to photographing; a photographed image input unit 13 for inputting a target image to be photographed by a user, digitizing it and outputting photographed image data; and a navigation system to be described later. Image input arbitration for arbitrating the order of the external video input unit 14 that inputs the video from 2, the captured image data input from the captured image input unit 13, and the navigation image data input from the external image input unit 14, and outputting. section 15, an image memory section 16 for temporarily holding data output from the image input arbitration section 15, image data is read out from the image memory section 16, and MPEG2 or H.264 is processed. An image encoding unit 17 that performs image compression/encoding by a method such as H.264 and outputs encoded image data, and an audio input unit that inputs audio of an object to be photographed by the user, digitizes it, and outputs audio data. 18. Audio data output from the audio input unit 18 is audio-compressed by a method such as AC3 or Dolby Digital, and an audio encoding unit 19 for outputting encoded audio data; A data generation unit 1A that multiplexes and outputs stream data, a data recording unit 1B that records the stream data on a recording medium described later, and a recording medium 1C such as a DVD, Blu-ray (registered trademark), hard disk, semiconductor memory, etc. have. A video camera 1 is connected to a navigation system 2 via an AV cable 3. The navigation unit 2 has a navigation main unit 21 and a navigation display unit 22, and the navigation main unit 21 has a navigation image output unit 211 for outputting navigation images. The navigation 211 outputs a signal to the AV cable 3 connected to an external video terminal (not shown) or the like in the same way as it always outputs the video signal to the navigation display section 22 .

FIG. 2 shows the operation flow of each part in the first embodiment.

A photographing procedure in the image recording/reproducing system of the first embodiment will be described. A user issues a photographing instruction to the operation input unit 11 using buttons, switches, or the like of the video camera 1 . The operation input unit 11 issues an imaging start instruction to the control unit 12 . The control unit 12 issues an activation instruction to the captured image input unit 13 , the voice input unit 18 and the external image input unit 14 . The external image input unit 14 takes in a video signal constantly output from an external video terminal or the like of the navigation 2 via the AV cable 3 in response to the activation instruction, generates navigation image data, and outputs it to the image input arbitration unit 15. do. The photographed image input unit 13 receives an image of a subject to be photographed by the user, digitizes it to generate photographed image data, and outputs the photographed image data to the image input arbitration unit 15 . The audio input unit 18 receives the audio of the object to be photographed by the user, digitizes it to generate audio data, and outputs the audio data to the audio encoding unit 19 . The navigation image data and the photographed image data input to the image input arbitration unit 15 are temporarily held in an area of the image memory 16 determined to be encoded in this order. That is, the navigation image data is stored in an area of the image memory unit 16 that is read out when the subsequent image encoding unit 17 processes the first image data, and the image memory is read out when processing the second and subsequent image data. The photographed image data is temporarily held in another area of the unit 16 . The navigation image data and the photographic image data temporarily held in the image memory section 16 are read as image data by the image encoding section 17 in this order. The image encoding unit 17 performs image compression and encoding of the image data. Here, the image encoding unit 17 compresses and encodes the navigation image data, which is the first image, using only the data within one screen. It compresses and encodes as an I picture of H.264. The photographed image data following the navigation image data is similarly compressed and encoded using the image data for the first screen of the photographed image data as an I-picture. Subsequent pictures are, for example, P pictures and B pictures that are predicted from forward and backward pictures and are compressed and coded, or for example, I pictures that are compressed and coded only with data within one screen, or The mixed data is compressed and encoded to generate encoded image data, which is output to the data generation unit 1A. Further, the image encoding unit 17 performs processing with rough quantization in order to reduce the code amount and lighten the processing, or temporarily increases the clock frequency, in order to bring the compression/encoding processing closer to real-time operation. to change the processing speed. On the other hand, the audio data input to the audio encoding section 19 is compressed and encoded by the audio encoding section 19 to generate encoded audio data, which is output to the data generating section 1A. The data generator 1A multiplexes the coded image data and the coded audio data to generate moving image stream data. FIG. 3 shows a schematic diagram (one example) of stream data. In FIG. 3, stream data are shown in the order of images subjected to decoding and decompression processing from the top. The first image shows the navigation screen, which is compressed and coded as an I-picture, multiplexed so that there is no audio data, and the second, third, and fourth images show the captured image data. The second picture is an I picture, the third picture is a B picture, and the fourth picture is compressed and coded as a P picture, and audio data is multiplexed using the coded audio data. The stream data generated by the data generating section 1A is output to the data recording section 1B, and the data recording section 1B records the stream data on the recording medium 1C.

In order to reproduce the stream data recorded in this way on the video camera 1, a picture analysis unit (not shown) for analyzing the picture type of the stream data is provided. For example, after the first I-picture is output to the display unit of the video camera 1 (not shown), the same image is displayed for a preset time, and then the next I-picture is output to the display unit. If the following pictures are not I-pictures, they are output to the display section one after another like normal video data reproduction. If two or more I-pictures are stream data, the first I-picture is output to the display unit of the video camera 1 (not shown), and then the same image is displayed for a preset time. However, the next image may be displayed starting from a user operation, such as displaying the same image until the user performs an operation such as pressing a play button.

For example, in a moving image compression/encoding method that embeds data for stopping stream playback, such as the still command of the DVD-Video standard, in a video image, the picture of the navigation screen and the captured images of the second and subsequent screens are used. For example, a VOBU (Video Object Unit) may be separated from a picture and a VOBU Still command may be inserted by the data generation unit 1A to support playback on a device other than the video camera 1. FIG.

As described above, in addition to the means for inputting normal shot images into the video camera, means for inputting images from another video input source is provided, and video data from these two input means are arbitrated and captured. By providing an image input arbitration unit for storing in memory and using the video output terminal of the car navigation system as the other video input source, the screen of the car navigation system (map information) and the image taken by the user are combined into the memory. can be taken inside. By compressing and encoding the data in this memory as a moving image, it is possible to process the map information and the image captured by the user as one moving image.

Also, in this embodiment, the navigation 2 is connected to the outside of the video camera 1, but the same effect can be obtained even if the video camera 1 and the navigation 2 are one device.

Also, the stream data recorded on the recording medium 1C may be automatically transferred to a server on the network via a wireless LAN environment (hot spot) installed in the city. FIG. 8 shows a schematic diagram of a system for transferring the stream data to a server under hotspot environment. The video camera 1 has a data reading section 1E for reading the stream data from the recording medium 1C and a wireless communication section 1F for wirelessly transmitting the stream data to the network 4. FIG. The stream data output from the wireless communication unit 1F is transferred to the server 5 connected to the network 4. FIG.

In this embodiment, the data generation unit 1A simply generates stream data. It may be output to the recording section 1B and recorded on the recording medium 1C. At this time, when the thumbnail screen of the stream data recorded on the recording medium 1C is generated, the current position information may be used as a clue and combined with the map screen as shown in FIG. 9 to configure the thumbnail display screen. In the example of FIG. 9, the position on the map corresponding to the position information associated with the stream data is indicated by a circle, and the first image of the shot images of the stream data is displayed in association with the circle. is doing. When the video camera 1 has a plurality of recording media, for example, a hard disk and a DVD, the video camera 1 normally uses the hard disk for recording, and the user selects and stores the stream data. When editing and recording on a DVD, a title screen such as the one shown in FIG. 9 may be automatically created and recorded on the DVD.

In this embodiment, the user operates the buttons and switches of the video camera 1 to issue a photographing instruction from the operation input unit 11. However, the navigation 2 and the video camera 1 may be connected to communication means such as USB, Bluetooth, wireless LAN, or the like. is provided, and the user sets a point to be photographed in advance for the navigation 2, and when the user moves to the set point, the navigation 2 outputs a photographing instruction to the operation input unit 11 of the video camera 1, and the operation input The photographing may be started by the unit 11 outputting a photographing start instruction to the control unit 12 .

Also, instead of the navigation 2 and the AV cable 3, a medium in which the map image data is stored in advance, such as a USB flash memory or an SD card, and a reading device for reading out the map image data from the medium may be substituted. Transferring data to a server on a network under a hotspot environment as shown, reading compressed image data recorded on a recording medium, and packetizing the compressed image data at a predetermined timing set by a user A similar effect can be obtained in a video camera for monitoring use installed in a plurality of places by having a transmission means for transmitting to a network. At this time, the wireless communication unit 1F may be replaced with a function of communicating with a wired LAN.

Example 2 will be described. FIG. 4 illustrates an image recording/reproducing system according to a second embodiment of the present invention. This embodiment differs from the first embodiment in that, in addition to the configuration shown in FIG. A voice input arbitration unit 182 that arbitrates the order of data and navigation voice data input from the external voice input unit 181 and outputs them, a voice memory unit 183 that temporarily holds the data output from the voice input arbitration unit 182, In addition to the configuration shown in FIG. A navigation voice activation unit 213 that receives a voice activation instruction from the voice activation unit 1D via infrared communication or the like, issues a activation instruction to output a navigation voice, and a navigation voice output unit 212 that outputs a navigation voice according to the activation instruction. It is a point that has

FIG. 5 shows the operation flow of each part in the second embodiment.

A photographing procedure in the image recording/reproducing system of the second embodiment will be described. A user issues a photographing instruction to the operation input unit 11 of the video camera 1 . This photographing instruction may be an operation for activating a voice indicating position information from the navigation 2, such as an operation of pressing the "current location" button of the navigation. The operation input unit 11 issues an imaging start instruction to the control unit 12 . Further, the operation input unit 11 issues a voice activation instruction to the voice activation unit 1D. The control unit 12 issues an activation instruction to the captured image input unit 13 , the audio input unit 18 , the external image input unit 14 and the external audio input unit 181 . The external image input unit 14 takes in a video signal constantly output from an external video terminal or the like from the navigation device 2 according to the activation instruction, generates navigation image data, and outputs the generated navigation image data to the image input mediation unit 15 . The navigation image data in this case is data of one or more pictures for a certain period of time set in advance by a user or the like. The photographed image input unit 13 receives an image of a subject to be photographed by the user, digitizes it to generate photographed image data, and outputs the photographed image data to the image input arbitration unit 15 . On the other hand, in accordance with the voice activation instruction issued to the voice activation section 1D by the operation input section 11, the voice activation section 1D issues the voice activation instruction to the navigation voice activation section 213 of the navigation 2 by infrared communication or the like. This voice activation instruction is a signal that is output to the navigation device 2 from a remote control that is generally included as a function of navigation and that is output to the navigation device 2 when a "current location" button or the like on a remote control (not shown) attached to the navigation device is pressed. , may be output from the voice activation unit. Upon receiving the voice activation instruction, the navigation voice activation unit 213 issues an activation instruction to the navigation voice output unit 212 . In response to the activation instruction, the navigation voice output unit outputs a voice such as, for example, "My current location is near Yoshida-cho, Totsuka-ku, Yokohama." The external voice input unit 181 takes in the navigation voice output from the navigation device 2 through an external voice terminal or the like in response to the activation instruction from the control unit 12 , generates navigation voice data, and outputs the navigation voice data to the voice input arbitration unit 15 . The navigation voice data in this case is data for a certain period of time set in advance by the user or the like. This set time may be the same as the time when the external image input unit 14 captures the image. Also, the audio input unit 18 inputs the audio of the object to be photographed by the user, digitizes it to generate audio data, and outputs it to the audio encoding unit 19 . The navigation image data and the photographed image data input to the image input arbitration unit 15 are temporarily held in an area of the image memory 16 determined to be encoded in this order. That is, the navigation image data is stored in an area of the image memory unit 16 that is read out when the subsequent image encoding unit 17 processes the first image data, and the image memory is read out when processing the second and subsequent image data. The photographed image data is temporarily held in another area of the unit 16 . The navigation image data and the photographic image data temporarily held in the image memory section 16 are read as image data by the image encoding section 17 in this order. The image encoding unit 17 performs image compression and encoding of the image data. Here, the image encoding unit 17 compresses and encodes the navigation image data, which is the first image, using only the data within one screen. It compresses and encodes as an I picture of H.264. The photographed image data following the navigation image data is similarly compressed and encoded using the image data for the first screen of the photographed image data as an I-picture. Subsequent pictures are, for example, P pictures and B pictures that are predicted from forward and backward pictures and are compressed and coded, or for example, I pictures that are compressed and coded only with data within one screen, or The mixed data is compressed and encoded to generate encoded image data, which is output to the data generation unit 1A. On the other hand, the navigation voice data and the voice data input to the voice input arbitration unit 182 are temporarily held in an area of the voice memory 183 determined to be encoded in this order. The navigation voice data and the voice data temporarily held in the voice memory unit 183 are read out to the voice encoding unit 19 in this order as mixed voice data. The mixed audio data input to the audio encoding unit 19 is compressed and encoded by the audio encoding unit 19 to generate encoded audio data, which is output to the data generating unit 1A. The data generator 1A multiplexes the coded image data and the coded audio data to generate moving image stream data. FIG. 6 shows a schematic diagram (one example) of stream data. In FIG. 6, stream data are shown in the order of images subjected to decoding and decompression processing from the top. The upper three images show the navigation screen, which are compressed and encoded as I, B, and P pictures in order from the top, and audio data is multiplexed with data corresponding to external audio data of the encoded audio data. . The lower three images show captured image data, which are compressed and encoded as I, B, and P pictures in order from the fourth image from the top, and the audio data corresponds to the audio data of the encoded audio data. Multiplexing with data. Although the top three images in FIG. 6 are the same image, they may be different navigation images. The stream data generated by the data generating section 1A is output to the data recording section 1B, and the data recording section 1B records the stream data on the recording medium 1C.

By the above processing, the same effects as those of the first embodiment can be obtained.

In this embodiment, the navigation image data is data for a certain period of time set in advance by the user or the like. When the voice ends, the navigation voice end notification may be output from the navigation device 2 to the external voice input section 181 of the video camera 1, and the external voice input section 181 may determine the end of the data based on the end notice.

Example 3 will be described. The difference from embodiment 1 is that in embodiment 1 the video camera 1 and the navigation 2 are connected by the AV cable 3, but the video camera 1 has the C side wireless communication means 32 and the navigation 2 has the N side wireless communication means. 31 is connected. The C-side wireless communication means 32 and the N-side wireless communication means 31 communicate with each other and exchange data using wireless communication protocols such as infrared communication, wireless LAN, and Bluetooth (registered trademark).

In the imaging procedure in the image recording/reproducing system of the third embodiment, the navigation image is exchanged from the navigation image output unit 211 to the external image input unit 14 via the AV cable in the first embodiment, but the N side wireless communication means 31 and It is the same as the first embodiment except that the navigation image is exchanged by the C-side wireless communication means 32 .

By the above processing, the same effects as those of the first embodiment can be obtained.

1 video camera 11 operation input unit 12 control unit 13 captured image input unit 14 external image input unit 15 image input arbitration unit 16 image memory unit 18 audio input unit 181 external audio input unit 182 audio input arbitration unit 183 audio memory unit 1D audio activation Section 2 Navigation 21 Navigation body 211 Navigation image output section 212 Navigation sound output section 213 Navigation sound activation section 22 Navigation display section 3 AV cable

Claims (2)

An image processing device,
a first interface for receiving input of a plurality of images;
a second interface, different from the first interface, for receiving an input of a map image associated with the capture location from which the plurality of images were captured;
a display unit for displaying a moving image generated using the map image and the plurality of images,
The display unit, while displaying the moving image, displays one thumbnail image corresponding to one image selected from the plurality of images constituting the moving image being displayed. An image processing apparatus, characterized in that the image is displayed together with the map image in association with a position on the map image corresponding to the acquired position. An image processing method comprising:
receiving input of a plurality of images via a first interface;
receiving an input of a map image associated with an acquisition location from which the plurality of images were acquired via a second interface different from the first interface;
displaying a moving image generated using the map image and the plurality of images on a display unit;
acquiring one thumbnail image corresponding to one image selected from the plurality of images constituting the moving image being displayed while the moving image is being displayed on the display unit; An image processing method, wherein the image is displayed on the display unit together with the map image in association with the position on the map image corresponding to the position.

Images