JP6399745B2 - Video processing apparatus and program - Google Patents

Description

  The present invention relates to a video processing technique.

A technique is disclosed in which a moving image such as a bus or a train displays a live-action image or guidance information acquired in advance on a vehicle window when the vehicle window scenery cannot be seen due to a soundproof wall or the like (for example, Patent Document 1). ).
In this technique, a moving body is provided with a device for specifying a position such as GPS (Global Positioning System) and a device for recording video, and the positional information and video information are stored in association with each other when the moving body travels.
Then, this data (position information and video information) is stored in a storage medium of another mobile body, and the closest video is acquired and displayed according to the position of another mobile body. .

JP 2004-20223 A

In Patent Document 1, since a captured image is used when an image is displayed during traveling, there is a problem that the displayed image is uncomfortable when there is a difference between the traveling speed at the time of photographing and the current traveling speed. was there.
For example, in the case of a bus, the scenery can be seen from the front window (driver's seat window) even if the scenery is not visible from the side window.
For this reason, when there is a difference between the traveling speed at the time of shooting (the traveling speed in the image) and the current traveling speed, the scenery of the front window and the scenery image displayed on the side window are inconsistent, and passengers Will feel uncomfortable.

  The main object of the present invention is to solve the above-described problems, and to provide a smooth and high-definition landscape image even if there is a speed difference between the traveling speed in the image and the current traveling speed of the moving object. The main purpose.

The video processing apparatus according to the present invention is
A video processing device mounted on a moving body that travels along a travel route,
A video data storage unit that stores video data of a video representing a situation in which a landscape around the travel route is moved by traveling of the travel route;
A decoding speed designating unit for analyzing the video, calculating a traveling speed in the video as a video traveling speed, and designating a decoding speed of the video data based on the calculated video traveling speed and a current traveling speed of the moving body When,
A decoding processing unit that decodes the video data at a decoding speed specified by the decoding speed specifying unit.

In the present invention, the video traveling speed, which is the traveling speed in the video, is calculated, the decoding speed of the video data is designated based on the calculated video traveling speed and the current traveling speed of the moving object, and the video is reproduced at the designated decoding speed. Decode data.
Therefore, a smooth and high-definition landscape image can be provided even if there is a speed difference between the traveling speed in the image and the current traveling speed of the moving object.

1 is a diagram illustrating a configuration example of a video processing apparatus according to Embodiment 1. FIG. FIG. 6 is a diagram illustrating a configuration example of a video processing apparatus according to a third embodiment. FIG. 10 is a diagram illustrating an example of an output image according to the third embodiment. FIG. 3 is a flowchart showing an operation example of the video processing apparatus according to the first embodiment. FIG. 3 is a diagram illustrating a hardware configuration example of a video processing device according to the first to third embodiments.

Embodiment 1 FIG.
In the present embodiment, a video processing apparatus that provides a smooth and high-definition landscape image even when there is a speed difference from the time of shooting when the field of view is blocked by the moving body will be described.

FIG. 1 shows a configuration example of a video processing apparatus 100 according to the present embodiment.
The video processing apparatus 100 is mounted on a moving body.
The moving body on which the video processing apparatus 100 is mounted is, for example, a vehicle (bus, taxi, hire, train, etc.) that travels along a specified travel route.
The moving body may be a ship or an aircraft that navigates a predetermined route.
In this specification, “running” includes navigation by a ship or an aircraft.

In FIG. 1, a video data storage unit 101 accumulates video corresponding to positions.
The video data storage unit 101 stores, for example, video data of a landscape video that can be seen from a car window on the side of the bus, together with position information.

  The position information acquisition unit 102 acquires position information obtained from a sensor such as a GPS installed on the moving body.

  The speed information acquisition unit 103 acquires speed information obtained from a speed sensor or the like installed on the moving body.

The video decoder 104 changes the processing speed of the video data stored in the video data storage unit 101 using the position information and speed information obtained from the position information acquisition unit 102 and the speed information acquisition unit 103 to change the video data. Decode.
More specifically, the video decoder 104 analyzes the landscape video and calculates the travel speed in the landscape video as the video travel speed. Based on the calculated video travel speed and the current travel speed of the moving object, the video decoder 104 Specify the decoding speed.
Then, the video decoder 104 decodes the video data at the designated decoding speed.
The video decoder 104 corresponds to an example of a decoding speed specifying unit and a decoding processing unit.

  The image buffer 105 temporarily stores a decoding result image output from the video decoder 104 for a predetermined number of frames.

  The image selection unit 106 determines an image to be displayed from the images stored in the image buffer 105 using the position / speed information obtained from the position information acquisition unit 102 and the speed information acquisition unit 103.

  The display unit 107 displays the image selected by the image selection unit 106.

  Next, the operation of the video processing apparatus 100 according to the present embodiment will be described.

The video data storage unit 101 stores video corresponding to the position acquired in advance.
Here, the video data that can be stored in the video data storage unit 101 is, for example, video data of a live-action video obtained by photographing a landscape around the travel route while actually traveling on the travel route.
Instead of the live-action video, it may also be video data of a landscape video created by using computer graphics technology together with the live-action video.
In addition, the video data storage unit 101 simultaneously stores the position information corresponding to each frame on the video data along with the video data.
At this time, geographical coordinates indicated by latitude and longitude can be used as the position information, but information indicated by the distance from the starting base point of the moving body may be used.
The video data storage unit 101 may store a continuous bird's-eye view image created from an aerial photograph or a satellite photograph together with the video data of a landscape video.

  When the mobile object travels, the video decoder 104 performs the operation shown in FIG.

Specifically, the video decoder 104 inputs position information from the position information acquisition unit 102 (S401).
Then, the video decoder 104 cues the video data using the position information obtained from the position information acquisition unit 102 as an initial condition (S402).
In addition, the video decoder 104 inputs speed information from the speed information acquisition unit 103 (S403).
Then, the video decoder 104 determines the number of decoding frames (decoding speed) per unit time using the speed obtained from the speed information acquisition unit 103.
Hereinafter, a case where the speed is simply used will be described.

The video decoder 104 determines the moving speed at two consecutive points from the position information recorded in association with the video data for the video data targeted for cueing in S402, so that the traveling at the time of shooting is performed. The speed is calculated (S404).
That is, the video data represents a situation in which the scenery of the travel route moves according to the travel of the travel route, but the video decoder 104 calculates the travel speed between two consecutive points so that the travel in the video is performed. The speed (video running speed) can be calculated.

Next, the video decoder 104 calculates the number of decoding frames (decoding speed) per unit time from the traveling speed at the time of shooting calculated in S404 and the current traveling speed of the moving object indicated in the speed information input in S403. decide.
For example, if the current travel speed is slower than the travel speed at the time of shooting, a decoding speed lower than the frame rate at the time of video shooting is determined.
Conversely, when the current traveling speed is faster than the traveling speed at the time of shooting, a decoding speed that is faster than the frame rate at the time of shooting the video is determined.
If the traveling speed at the time of shooting is the same as the current traveling speed, a decoding speed that matches the frame rate at the time of shooting the video is determined.
More specifically, the video decoder 104 designates a decoding speed obtained by multiplying the frame rate of the video data by a division value obtained by dividing the current traveling speed of the moving object by the traveling speed at the time of shooting. .
For example, when the traveling speed at the time of shooting is 60 km / h and the current traveling speed is 50 km / h, 50/60 = approximately 0.83, so that the frame rate at the time of video recording is 0.83 times the frame rate. To decode at the speed of.
Further, for example, when the speed at the time of shooting is 60 km / h and the current speed is 70 km / h, 70/60 = 1.17, so that the frame rate at the time of video recording is 1. Decode at 17 times the speed.

When driving at a speed higher than the shooting speed and decoding at the frame rate at the time of video recording, video that is delayed from the actual landscape will be displayed, and provide appropriate video I can't.
However, in the present embodiment, as described above, the decoding speed is adjusted in accordance with the current traveling speed. Therefore, even when there is a speed difference between the traveling speed at the time of shooting and the current traveling speed, an appropriate landscape image is displayed. The presentation can be continued.

It is also possible to predict the future decoding speed by using acceleration instead of the speed difference between the traveling speed at the time of shooting and the current traveling speed.
Furthermore, when the moving body is a railway vehicle, it is possible to improve the prediction accuracy by providing the video data storage unit 101 with information indicating a basic operation speed called a run curve (travel curve) for the travel route in advance. is there.
It is also possible to set a physical limit speed and set the video decoder processing speed in advance with reference to the limit speed.
However, in this case, it is conceivable that the image buffer 105 is insufficient because the speed difference becomes too large when there is a traffic jam or operation failure.
For this reason, the present method based on the physical limit speed can be applied to a moving body that travels in a short section or in the future when the image buffer 105 is large enough to hold all the decoded data of the traveling section. This is effective when a large-capacity memory device that can be developed is used.

The video decoder 104 decodes the video data at the decoding speed determined in S406 (S406), and stores the decoded data in the image buffer 105 (S407).
Thereafter, the video decoder 104 repeats the operations after S402.

The image buffer 105 temporarily stores the video data decoded by the video decoder 104.
At this time, not only video data but also positional information of each image is held in the buffer.
The amount of memory required by the image buffer 105 was calculated from the conditions of the traveling characteristics and speed characteristics of the assumed moving body to determine how many times the traveling speed at the time of shooting is likely, and the frame rate at the time of shooting was obtained. The amount can be an amount multiplied by a multiple.
In this way, processing can be performed without running out of buffers.

The image selection unit 106 receives position information and speed information from the position information acquisition unit 102 and speed information acquisition unit 103, selects an image to be displayed from the image buffer 105, and displays it on the display unit 107.
At this time, the following method is used to select an image to be displayed.
First, the image selection unit 106 grasps the position information of the currently displayed image, and grasps the current traveling speed obtained from the speed information acquisition unit 103.
Furthermore, the image selection unit 106 stores the position of the image displayed immediately before and the traveling speed at that time.
The acceleration is calculated using the current travel speed obtained from the speed information acquisition unit 103 and the immediately preceding travel speed.
The next required image position is predicted from the acceleration and the current traveling speed, and an image closest to the corresponding position is acquired.
This indicates that the frame to be displayed can be thinned out or interpolated according to the traveling state of the moving body.
As described above, if there is run curve information, the frame to be thinned out can be predicted in advance.
If the current traveling speed is slower than the time of shooting, the closest frame position may be shifted.
In this case, a common object is detected from the image closest to the current location and the image closest to the second location, and a corresponding point between them is obtained, and an image corresponding to the viewpoint of the current location is generated. You can also.
Since this can be calculated by using the current three-dimensional computer analysis technology, details of the processing are omitted in this specification.

  Although the video processing apparatus 100 is shown in the present embodiment, it can be realized by a computer program in which each processing unit shown in FIG. 1 is installed as software.

  In addition, as described above, not only live-action images but also virtual landscapes created by computer graphics are modeled together with terrain information, and the data of the part where the field of view is obstructed on the travel route is transparent. It is also possible to create and use images that appear as if they are visible, or to visualize and use a bird's-eye view of the aerial and satellite images from above.

  According to the present embodiment, by configuring and operating as described above, it is possible to smoothly provide a landscape video even if there is a speed difference from that during video shooting.

Embodiment 2. FIG.
In the first embodiment, it is assumed that there is not much difference in speed between actual running and shooting, but in this embodiment, a pseudo-low-speed running image is generated using a high-speed camera during shooting. An example of doing this is shown below.
The configuration of the video processing apparatus 100 is the same as that of the first embodiment.

In the video processing apparatus 100 according to the first embodiment, a good video can be displayed when the vehicle is traveling at a speed similar to that at the time of shooting. In many cases, it becomes impossible to provide detailed information.
Therefore, in the first embodiment, the example in which the interpolation image is generated and displayed has been described.
However, in order to display a higher-definition image, it is possible to capture the original image at a low speed and use this to make the image finer, but actually shooting at low speed is difficult due to traffic conditions. There are many cases.

Therefore, a high speed camera is used when shooting video data input to the video data storage unit 101.
The high-speed camera here is assumed to be capable of shooting about 900 images per second, for example.
For example, when traveling at a speed of 210 km / h, using an ordinary video camera, about 30 images can be acquired per second.
Assuming that the vehicle is traveling at a constant speed in the shooting section, the position interval between two consecutive images is about 1.94 m.
That is, the deviation is about 97 cm.
This is not a big problem when only the distant view is targeted, but the sense of discomfort increases when there are many things that look close-up.
By using the above high speed camera, it is synonymous with traveling at a speed of 1/30.
In this example, the distance between frames is about 6 cm because it is the same as when shooting from a vehicle traveling at a speed of 7 km / h.
In this case, video by computer graphics can be used here, but the same effect can be obtained by outputting and using video simulating low-speed driving in consideration of the contents in this embodiment at the time of output. can get.

  Thus, as described in this embodiment, by using a high-speed camera at the time of shooting, it is possible to simulate low-speed driving, so the distance difference between images is shortened, so even when driving at low speed, there is a sense of incongruity. Can provide high-definition video.

Embodiment 3 FIG.
In the first and second embodiments, the image displayed on the display unit 107 is handled as a single image, but the required storage capacity increases particularly when the method of the second embodiment is used.
If you want to show the scenery image of the train window, the distant view is not very moving, and the close view is very moving.
When the high-speed camera is not used as in the first embodiment, there is a problem that the foreground image becomes extremely rough if the current traveling speed is significantly slower than the traveling speed at the time of shooting.
Therefore, in the third embodiment, the video data input to the video data storage unit is divided into a far view and a near view.

  FIG. 2 shows a configuration example of the video processing apparatus 200 according to the third embodiment.

The video data storage unit 201 has the same function as the video data storage unit 101 in FIG. 1 (function to store video data according to position).
Here, the video data storage unit 201 has video data of a distant view video, video data of a foreground video, and position information corresponding to each position.
The distant view image data is image data representing a situation in which the distant view of the travel route moves as the travel route travels.
The video data of the foreground video is video data representing a situation in which the foreground of the travel route moves as the travel route travels.
Since the video data of the distant view video does not move much, it is generated at a low frame rate.
The video data of the foreground video is generated at a high frame rate because of its intense movement.
Further, in order to reduce the amount of stored data, it is determined here which region of the final output video each video corresponds to.
For example, as shown in FIG. 3, when an image of 1920 × 1080 pixels is output, an image is considered in a coordinate system with the upper left as the origin, and an area of (0, 0) − (1920, 540) is a distant image area, (0 , 540)-(1920, 1080) is used as the foreground image area, and the video data stored in the video data storage unit 201 is prepared as video data of a size corresponding thereto.
Further, since the video data of the distant view video does not feel uncomfortable even with a low frame rate video, for example, a video of about 10 fps (Frames Per Second) is prepared.
As the video data of the foreground video, a high frame rate video as shown in the second embodiment is prepared.
By doing so, the data size can be reduced to half or less compared to the case of the second embodiment.

The video decoder 204 also has the same function as the video decoder 104 in FIG.
However, the video decoder 204 extends so as to process both the video data of the distant view video and the video data of the foreground video in parallel.

That is, the video decoder 204 analyzes the foreground video and calculates the travel speed in the foreground video as the foreground video travel speed, and based on the calculated foreground video travel speed and the current travel speed of the moving object, the video data of the foreground video Specify the decoding speed.
The video decoder 204 analyzes the distant view image and calculates the travel speed in the distant view image as the distant view image travel speed. Based on the calculated distant view image travel speed and the current travel speed of the moving body, the image data of the distant view image is calculated. Specify the decoding speed.
The video decoder 204 decodes the video data of the foreground video at the decoding speed specified for the video data of the foreground video, and decodes the distant video at the decoding speed specified for the video data of the far background video. Decode video data.
Note that the decoding speed determination method itself is the same as in the first embodiment.
That is, if the current traveling speed of the moving body is faster than the foreground video traveling speed, the video decoder 204 designates a decoding speed higher than the frame rate of the foreground video data for the foreground video data.
Also, if the current traveling speed of the moving body is slower than the foreground video traveling speed, the video decoder 204 designates a decoding speed that is slower than the frame rate of the foreground video data for the foreground video data.
Also, if the current traveling speed of the moving body is the same as the foreground video traveling speed, the video decoder 204 designates a decoding speed that matches the frame rate of the foreground video data for the foreground video data.
Also, if the current traveling speed of the moving body is faster than the far-field video traveling speed, the video decoder 204 designates a decoding speed faster than the frame rate of the far-view video data for the far-view video data.
In addition, if the current traveling speed of the moving body is slower than the far-field video traveling speed, the video decoder 204 designates a decoding speed slower than the frame rate of the far-view video data for the far-view video data.
In addition, if the current traveling speed of the moving object is the same as the far-field video traveling speed, the video decoder 204 designates a decoding speed that matches the frame rate of the far-view video data for the far-view video data. .
For example, as in the first embodiment, the video decoder 204 divides the division value obtained by dividing the current traveling speed of the moving object by the near-field video traveling speed or the far-field video traveling speed, as video data of the foreground video or the far-field video. Specifies the decoding speed obtained by multiplying the frame rate.

The function of the image buffer 205 is the same as that of the image buffer 105 in FIG. 1, but it has a distant view image buffer and a close view image buffer.
The position information acquisition unit 202 has the same function as the position information acquisition unit 102 of FIG.
The speed information acquisition unit 203 has the same function as the speed information acquisition unit 103 of FIG.

The image selection unit 206 performs the same processing as the image selection unit 106 described in Embodiments 1 and 2 in FIG.
The image selection unit 206 performs processing of video data of the distant view video in parallel and creates an interpolation image.
Note that if the difference between the images is small, there is a case where the interpolation image need not be created.

The video synthesizing unit 208 performs a process of synthesizing each of the distant view video and the near view video selected by the image selection unit 206 into the final output image format.
In this embodiment, since the distant view video and the foreground image are divided into the upper and lower halves of the image, the final output image can be obtained simply by pasting the corresponding image in each region.
Distant view and foreground images are not only divided in the upper and lower halves, but each is set to overlap, and the overlapping part is processed by combining the distant view on the previous layer from the distant view, It is also possible to express the wrap region so as not to feel strange by α blending.

  The display unit 207 has the same function as the display unit 107 in FIG. 1 and displays the final output image synthesized by the video synthesis unit 208.

  In this embodiment, an example of synthesizing live-action images has been shown. However, it is also possible to use an image created by computer graphics, or to provide an intermediate layer in addition to a distant view or a foreground. is there.

  With the configuration described above, it is possible to provide a high-definition image that does not feel uncomfortable even when traveling at a low speed in a low storage area.

Finally, a hardware configuration example of the video processing apparatuses 100 and 200 shown in the first to third embodiments will be described with reference to FIG.
The video processing devices 100 and 200 are computers, and each element of the video processing devices 100 and 200 can be realized by a program.
As a hardware configuration of the video processing devices 100 and 200, an arithmetic device 901, an external storage device 902, a main storage device 903, and an input / output device 904 are connected to a bus.

The arithmetic device 901 is a CPU (Central Processing Unit) that executes a program.
The external storage device 902 is, for example, a ROM (Read Only Memory), a flash memory, or a hard disk device.
The video data storage units 101 and 201 are realized by an external storage device 902, for example.
The main storage device 903 is a RAM (Random Access Memory).
The image buffers 105 and 205 are realized by the main storage device 903, for example.
The input / output device 904 is, for example, a mouse, a keyboard, a display device, or the like.

The program is normally stored in the external storage device 902, and is loaded into the main storage device 903 and sequentially read into the arithmetic device 901 and executed.
The program is a program that realizes a function described as “˜unit” (excluding the video data storage units 101 and 201, the same applies below) shown in FIGS.
Further, an operating system (OS) is also stored in the external storage device 902. At least a part of the OS is loaded into the main storage device 903, and the arithmetic device 901 executes the OS, as shown in FIG. 1 and FIG. Executes a program that realizes the function of "~ part".
In the description of the first to third embodiments, “to be specified”, “determined to”, “calculation of”, “analysis of”, “prediction of”, “acquisition of”, “to” Information, data, signal values, and variable values indicating the results of the processing described as “Detection of”, “Selection of”, “Generation of”, “Input of”, “Output of”, etc. It is stored as a file in the device 903.

  Note that the configuration in FIG. 5 is merely an example of the hardware configuration of the video processing devices 100 and 200, and the hardware configuration of the video processing devices 100 and 200 is not limited to the configuration illustrated in FIG. It may be a configuration.

  DESCRIPTION OF SYMBOLS 100 Video processing apparatus, 101 Video data storage part, 102 Position information acquisition part, 103 Speed information acquisition part, 104 Video decoder, 105 Image buffer, 106 Image selection part, 107 Display part, 200 Video processing apparatus, 201 Video data storage part 202 Position information acquisition unit 203 Speed information acquisition unit 204 Video decoder 205 Image buffer 206 Image selection unit 207 Display unit 208 Video composition unit

Claims (7)

A video processing device mounted on a moving body that travels along a travel route,
Video data of a foreground image representing a situation in which the foreground of the travel route is moved by the travel of the travel route, and a distant view of a situation in which the background of the travel route is moved by the travel of the travel route A video data storage unit that stores video data of the video, position information of a foreground image corresponding to a position on the foreground image, and position information of a distant view video corresponding to a position on the far view image;
The position information of the foreground image is analyzed to calculate the travel speed in the foreground image as the foreground image travel speed. Based on the calculated foreground image travel speed and the current travel speed of the moving body, the image data of the foreground image The decoding speed is specified, the position information of the distant view image is analyzed, and the traveling speed in the distant view image is calculated as the distant view image traveling speed. Based on the calculated distant view image traveling speed and the current traveling speed of the mobile object A decoding speed designating unit for designating a decoding speed of the video data of the distant view video,
The decoding speed designating unit decodes the foreground video data at the decoding speed designated for the foreground video data, and the decoding speed designating unit designates the foreground video data. And a decoding processing unit for decoding video data of the distant view video at a decoding speed. The decoding speed designation unit
If the current traveling speed of the mobile body is faster than the foreground video traveling speed, a decoding speed higher than the frame rate of the foreground video data is specified for the foreground video data,
If the current traveling speed of the moving body is slower than the foreground video traveling speed, a decoding speed slower than the frame rate of the foreground video data is designated for the foreground video data,
If the current traveling speed of the mobile body is the same as the foreground video traveling speed, a decoding speed that matches the frame rate of the foreground video data is designated for the foreground video data,
If the current traveling speed of the moving body is faster than the far-field video traveling speed, a decoding speed faster than the frame rate of the far-field video data is specified for the far-view video data,
If the current traveling speed of the mobile body is slower than the far-field video traveling speed, a decoding speed slower than the frame rate of the far-field video data is specified for the far-field video data,
If the current traveling speed of the mobile body is the same as the far-field video traveling speed, a decoding speed that matches a frame rate of the far-field video data is specified for the far-field video data. The video processing apparatus according to claim 1. The decoding speed designation unit
Analyzing the position information of the foreground image corresponding to the current position of the mobile body to calculate the near-field image traveling speed,
The video processing apparatus according to claim 1, wherein the distant view image traveling speed is calculated by analyzing position information of the distant view image corresponding to the current position of the moving body. The video data storage unit
2. The video processing apparatus according to claim 1, wherein video data of a foreground video having a frame rate higher than that of the video data of the far-view video is stored. The video data storage unit
The video processing apparatus according to claim 1 , wherein video data of a foreground video photographed by a high speed camera is stored. The video processing device further includes:
A video synthesizing unit that synthesizes the foreground image and the distant view video that have been decoded by the decoding processing unit, and generates a video representing a situation in which a landscape around the travel route moves;
The video processing apparatus according to claim 1, further comprising: a display unit that displays the video generated by the video synthesis unit. It is mounted on a moving body that travels along the travel route,
Video data of a foreground image representing a situation in which the foreground of the travel route is moved by the travel of the travel route, and a distant view of a situation in which the background of the travel route is moved by the travel of the travel route A computer storing video data of a video, position information of a foreground image corresponding to a position on the foreground image and position information of a distant view image corresponding to a position on the distant view image,
The position information of the foreground image is analyzed to calculate the travel speed in the foreground image as the foreground image travel speed. Based on the calculated foreground image travel speed and the current travel speed of the moving body, the image data of the foreground image The decoding speed is specified, the position information of the distant view image is analyzed, and the traveling speed in the distant view image is calculated as the distant view image traveling speed. Based on the calculated distant view image traveling speed and the current traveling speed of the mobile object Decoding speed designation processing for designating the decoding speed of the video data of the distant view video;
The video data of the foreground video is decoded at the decoding speed designated for the video data of the foreground video by the decoding speed designation process, and the video data of the far view video is designated by the decoding speed designation process. And a decoding process for decoding video data of the distant view video at a decoding speed.

Images