JP7274917B2 - Video playback device and video playback method - Google Patents

Description

The present invention relates to a video reproducing device and a video reproducing method.

2. Description of the Related Art In recent years, for example, there is a drive recorder that records video data obtained by separately capturing the surroundings of a vehicle such as a front camera and a rear camera of the vehicle in a storage medium (see, for example, Patent Document 1).

JP 2010-244473 A

However, in the conventional technology, when a plurality of pieces of video data are reproduced at the same time, a time lag may occur between the reproduction times of the respective pieces of video data. Specifically, there have been cases in which video data captured at different times are displayed even though they are being played back at the same time.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a video reproducing apparatus and a video reproducing method capable of eliminating the time lag in reproduction time that occurs during reproduction.

In order to solve the above-described problems and achieve the object, a video playback device according to an embodiment includes an acquisition unit, a playback unit, and a correction unit. The acquisition unit acquires a plurality of files in which image data captured by a plurality of imaging devices provided in each of the moving bodies is divided and saved in different formats for each imaging device. The reproducing unit reads out the image data from the plurality of files acquired by the acquiring unit, and simultaneously reproduces the image data for each imaging device. The correcting section corrects a reproduction time lag that occurs when the image data that are simultaneously reproduced by the reproducing section are simultaneously reproduced.

According to the present invention, it is possible to eliminate the time lag in playback time that occurs during playback.

FIG. 1A is a diagram showing an overview of a video playback system. FIG. 1B is a diagram showing an overview of the video playback method. FIG. 2 is a block diagram of an in-vehicle device. FIG. 3 is a block diagram of a video playback device. FIG. 4 is a diagram showing an example layout of a playback screen. FIG. 5 is a diagram illustrating an example of processing by a detection unit; FIG. 6 is a diagram showing a specific example of correction content by the correction unit. FIG. 7 is a flow chart showing a processing procedure executed by the video playback device.

A video reproducing apparatus and a video reproducing method according to embodiments will be described below with reference to the accompanying drawings. In addition, this invention is not limited by embodiment shown below. In the following description, the mobile object is a vehicle, but the mobile object is not limited to a vehicle, and may be a mobile object other than a vehicle, such as a motorcycle, a ship, or an aircraft.

First, an outline of a video playback device and a video playback method according to an embodiment will be described with reference to FIGS. 1A and 1B. FIG. 1A is a diagram showing a configuration example of a video playback system. FIG. 1B is a diagram showing an overview of the video playback method.

As shown in FIG. 1A, an image processing system S according to the embodiment includes a video playback device 1 and an in-vehicle device 50 . The in-vehicle device 50 is, for example, a storage device that stores image data captured on the vehicle. In the example of FIG. 1A, the in-vehicle device 50 is connected to a front camera 10f and a rear camera 10b that captures an image of the rear of the vehicle. Remember.

The front camera 10f is a camera for imaging the front of the vehicle, and the rear camera 10b is a camera for imaging the rear of the vehicle. In this embodiment, the front camera 10f and the rear camera 10b generate image data at different frame rates. Note that the front camera 10f and the rear camera 10b each correspond to an example of an imaging device. Further, the imaging device is not limited to the front camera 10f and the rear camera 10b, and may include a side camera for imaging the side of the vehicle and an in-vehicle camera for imaging the interior of the vehicle. Further, hereinafter, the front camera 10f and the rear camera 10b may be referred to as the camera 10 when there is no need to distinguish between them.

A video reproduction device 1 shown in FIG. 1A is a computer having so-called viewer software for reproducing image data acquired from an in-vehicle device 50 . Note that the video playback device 1 is an information processing device such as a mobile phone including a smart phone, a tablet terminal, a desktop PC, a notebook PC, or a PDA (Personal Digital Assistant). The video playback device 1 also includes a wearable device, which is an information processing terminal in the form of spectacles or a watch.

For example, as shown in FIG. 1A, the video playback device 1 can acquire image data from an in-vehicle device 50 via a portable memory M or a network N. FIG. Here, the in-vehicle device 50 divides the image data of the front camera 10f and the image data of the rear camera 10b into files each having a predetermined capacity (for example, 36 MB) and records them. Note that hereinafter, image data captured by the front camera 10f may be referred to as a front image, and image data captured by the rear camera 10b may be referred to as a rear image.

Here, if the front camera 10f and the rear camera 10b record in different formats, the imaging period of the image data that can be stored in one file differs between the front image and the rear image. The different format specifically means that the bit rate at the time of saving is different. For example, the rear image may be saved at a lower bit rate than the front image. In this case, as the bit rate decreases, the image quality also decreases, so the frame rate and image resolution may be lowered to maintain the image quality of the recorded image.

In general, when reproducing image data, switching processing is performed at the timing when files are switched, so a time lag occurs. For this reason, as described above, if the front image and the rear image are stored in one file and have different imaging periods, if they are played back at the same time, the front image and the rear image will be played back as the playback progresses. The time gradually slips away. This is because the recording period of one file is different, and therefore the frequency of occurrence of switching processing differs between the front image and the rear image.

Specifically, it will be described with reference to FIG. 1B. FIG. 1B is an example in which the recording period per file for the front image is about ⅔ of the recording period per file for the rear image. In this case, one switching timing T occurs in the rear image while two switching timings T for switching files occur in the front image. Therefore, the front image is reproduced with a delay from the rear image by the switching timing T of one time.

Specifically, it is assumed that the front image is recorded for 20 seconds and the rear image is recorded for 30 seconds per file. In this case, immediately after the front image 1-1 is reproduced, the reproduction time of the front image is 20 seconds, and the reproduction time of the rear image is also 20 seconds. After that, the front image is subjected to switching processing, and it is assumed that the processing takes one second. Then, after the switching timing T, the front image is switched to the front image 1-2 and playback is started, so the playback time remains at 20 seconds. On the other hand, since the rear image 2-1 continues to be reproduced even during the switching process, the reproduction time is 21 seconds. After the switching timing T, a difference of 1 second occurs in the reproduction time between the front image and the rear image.

Since such a lag in playback time between videos increases as playback progresses, it is expected to give the user a sense of discomfort.

Therefore, in the video reproduction method according to the embodiment, when the front image and the rear image are reproduced simultaneously, the difference in reproduction time between the front image and the rear image is corrected. For example, in the video playback method, it is possible to correct the lag in playback time by adjusting the playback speed.

Specifically, in the video playback method, by increasing the playback speed of a delayed image whose playback time is delayed and slowing down the playback speed of a preceding image whose playback time is earlier, Correct the misalignment.

As a result, in the video reproduction method according to the embodiment, it is possible to simultaneously reproduce the front image and the rear image in a state in which the times at which the images were actually captured are synchronized. That is, according to the video reproduction method according to the embodiment, it is possible to eliminate the time lag of the reproduction time.

In addition, it is assumed that changing the playback speed may give the user a sense of discomfort. Therefore, in the video reproduction method according to the embodiment, it is also possible to perform the correction at the timing at which the discomfort given to the user is reduced. Details of this point will be described later with reference to FIG.

Next, a configuration example of the in-vehicle device 50 will be described with reference to FIG. FIG. 2 is a block diagram of the in-vehicle device 50. As shown in FIG. As shown in FIG. 2, the in-vehicle device 50 is connected to the front camera 10f, the rear camera 10b, and the sensor group 15, respectively.

As described above, the front camera 10f is a camera for imaging the front of the vehicle, and the rear camera 10b is a camera for imaging the rear of the vehicle. For example, the front camera 10 f and the rear camera 10 b generate image data at different bit rates and output the image data to the in-vehicle device 50 . In this embodiment, it is assumed that the bit rate of the front camera 10f is higher than that of the rear camera 10b. Note that the rear camera 10b captures images at a lower frame rate than the front camera 10f in order to maintain image quality even at a low bit rate.

The sensor group 15 is various sensors that detect the running state of the vehicle, and outputs detection results to the in-vehicle device 50 . The sensor group 15 includes, for example, a speed sensor that detects the traveling speed of the vehicle, an acceleration sensor that detects the acceleration generated in the vehicle, a steering angle sensor that detects the steering angle of the vehicle, a position sensor that detects the current position of the vehicle, and the like. be

As shown in FIG. 2 , the in-vehicle device 50 includes a control section 6 and a storage section 7 . The control unit 6 is a controller that controls various functions of the in-vehicle device 50 . The control unit 6 may be a processor such as CPU or MPU, or may be an integrated circuit such as FPGA or ASIC.

The control unit 6 adds the photographing times as time stamps to the front image and the rear image input from the front camera 10f and the rear camera 10b, respectively, and stores the front image in the storage unit 7 as front image information 71. , the rear image is stored as the rear image information 72 . Similarly, the control unit 6 gives the acquisition time as a time stamp to various detection results input from the sensor group 15 and stores them in the storage unit 7 as the running state information 73 .

Further, the control unit 6 transfers the front image information 71, the rear image information 72, and the running state information 73 to a portable storage medium or transmits them to the video reproducing apparatus 1 via the network N by a predetermined user operation. or

Next, a configuration example of the video playback device 1 according to the embodiment will be described with reference to FIG. FIG. 3 is a block diagram of the video playback device 1. As shown in FIG. Note that FIG. 3 also shows the display device 100 . The display device 100 is a display device that displays a video signal input from the video reproduction device 1 . In addition, the video playback device 1 can be connected to an operation unit (not shown) to accept user operations.

As shown in FIG. 3 , the video playback device 1 according to the embodiment includes a control section 2 and a storage section 3 . The control unit 2 includes an acquisition unit 21 , a reproduction unit 22 , a detection unit 23 and a correction unit 24 . The storage unit 3 stores front image information 31 , rear image information 32 , running state information 33 and map information 34 .

Here, the control unit 2 includes, for example, a computer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), input/output ports, and various circuits. include.

The CPU of the computer functions as an acquisition unit 21, a reproduction unit 22, a detection unit 23, and a correction unit 24 of the control unit 2 by reading and executing programs stored in the HDD, for example.

Also, the storage unit 3 corresponds to, for example, a RAM or an HDD. The RAM and HDD can store front image information 31, rear image information 32, running state information 33, map information 34, various program information, and the like. Note that the video reproducing apparatus 1 may acquire the above-described programs and various information via another computer or a portable recording medium connected via a wired or wireless network.

The front image information 31 and the rear image information 32 are information about image data captured by the front camera 10f and the rear camera 10b, respectively. In this embodiment, the front image information 31 and the rear image information 32 store the front image and the rear image divided into a plurality of files, respectively. The file format is, for example, MPEG4 (Moving Picture Experts Group 4), but may be changed arbitrarily.

The running state information 33 is information about the running state of the vehicle detected by the sensor group 15 shown in FIG. The map information 34 is information about a map, and is information about an image of a bird's-eye view or a bird's-eye view.

The control unit 2 simultaneously reproduces the front image and the rear image, and corrects the difference in reproduction time between the front image and the rear image that occurs during reproduction.

The acquisition unit 21 acquires image data captured by a plurality of cameras 10 each provided in a vehicle (an example of a mobile object). Specifically, the acquisition unit 21 acquires a plurality of files divided and saved in different formats for each camera 10 . For example, the acquisition unit 21 acquires a file in which the front image information 31 and the rear image information 32 are stored via a portable recording medium.

The acquisition unit 21 also performs processing for storing the detection result of the sensor group 15 as the running state information 33 in the storage unit 3 .

The reproduction unit 22 stores predetermined parts of the plurality of files acquired by the acquisition unit 21 in the storage unit 3 as front image information 31 and rear image information 32 . The predetermined portion is necessary for reproduction processing such as decoding and caching of encoded data by MPEG4 or the like. The reproduction unit 22 reads image data from the stored front image information 31 and rear image information 32 and reproduces them simultaneously for each camera 10 . At this time, the reproducing unit 22 performs switching processing for switching files when the files of the image data are switched. Note that the reproducing unit 22 may store all of one file in the storage unit 3 .

FIG. 4 is a diagram showing an example layout of a playback screen. As shown in FIG. 4, the playback screen D has a front image area Rf, a rear image area Rb, a map image area Rm, and a sensor image area Rs. The front image area Rf is an area in which a front image is reproduced, and the rear image area Rb is an area in which a rear image is reproduced.

Further, the map image area Rm is an area where a map image indicating the imaging positions of the front image and the rear image being reproduced in the front image area Rf and the rear image area Rb is displayed. For example, the reproducing unit 22 reads information about the imaging position from the driving state information 33 and reads a map image of a predetermined range based on the imaging position from the map information 34, thereby generating a map to be reproduced in the map image region Rm. do. An icon is displayed at the position of the vehicle in the map image displayed in the map image area Rm.

The sensor image area Rs is an area for displaying a sensor image showing the detection result of the sensor group 15 at the time when the front image and the rear image reproduced in the front image area Rf and the rear image area Rb are captured.

For example, the reproducing unit 22 can display, as the sensor image, a speed image indicating the traveling speed, an acceleration image indicating the acceleration generated in the vehicle, or the like in the sensor image area Rs. Note that the layout of the playback screen D shown in FIG. 4 is an example, and may be changed arbitrarily.

Returning to the description of FIG. 3, the detection unit 23 will be described. The detection unit 23 detects a difference in reproduction time of image data reproduced by the reproduction unit 22 . For example, the detection unit 23 detects the difference in reproduction time based on the time stamps (imaging time) given to the front image and the rear image that are currently being reproduced by the reproduction unit 22, thereby accurately detecting the reproduction time difference. It is possible to detect

That is, the detection unit 23 detects the difference between the time stamps of the front image and the rear image being reproduced as the difference in reproduction time. For example, the detection unit 23 checks both time stamps at a predetermined cycle, and if the difference in reproduction time exceeds a predetermined value, causes the later-described correction unit 24 to correct the difference in reproduction time.

At this time, the detection unit 23 also determines a correction timing for correcting the reproduction time lag based on the running state information 33 in the storage unit 3 . That is, the detection unit 23 determines the correction timing based on the running information indicating the running state of the vehicle.

FIG. 5 is a diagram showing a specific example of processing by the detection unit 23. As shown in FIG. Note that in FIG. 5, the horizontal axis indicates the playback time. Further, A of FIG. 5 shows switching timing of the front image, and B of FIG. 5 shows switching timing of the rear image. Further, the amount of deviation shown in C of FIG. 5 indicates the difference between the playback times of the front image and the rear image, and the acceleration shown in D of FIG. show.

Here, the acceleration is the lateral acceleration acting in the width direction of the vehicle, but it is not limited to this, and may be the acceleration acting in the vertical direction of the vehicle, or the acceleration acting in the traveling direction of the vehicle. may be Also, the acceleration may be a combination of these.

As shown in FIGS. 5A and 5B, when the cycle of the switching timing of the front image is shorter than the cycle of the switching timing of the rear image, the reproduced image of the front image is delayed compared to the reproduced image of the rear image. I will continue.

Therefore, as shown in FIG. 5C, the difference in reproduction time gradually increases as the reproduction time advances. For example, the detection unit 23 detects a difference in reproduction time at time t11 when the amount of difference exceeds a predetermined value Th.

After detecting the reproduction time lag, the detection unit 23 subsequently determines the correction timing. In this embodiment, the detection unit 23 can determine the correction timing based on the transition of acceleration shown in D of FIG.

Specifically, when the detecting unit 23 detects the reproduction time lag, it checks the transition of the acceleration and determines whether or not the acceleration changes in a predetermined manner within a predetermined period. Specifically, it is whether or not a change in vertical acceleration exceeding a predetermined threshold occurs. In the example shown in D of FIG. 5, it is assumed that the change in acceleration during the period from time t12 to time t13 corresponds.

In such a case, the detection unit 23 determines the period from time t12 to time t13 as the correction timing. This means that when the vehicle is accelerating, the camera 10 is also accelerating. .

In other words, the rear image and the front image themselves are disturbed during the period in which the acceleration is occurring, so that even if the reproduction time lag is corrected during this period, the user will hardly notice it. In other words, this is because it is possible to correct the difference in playback time without giving the user a sense of discomfort.

On the other hand, for example, when the acceleration does not change within a predetermined period of time after detection of the reproduction time shift, the detection unit 23 determines, for example, a predetermined period based on the next switching timing T as the correction timing. The predetermined period referred to here is two seconds before and after the switching timing T, for example.

As described above, the switching timing T (see FIG. 1B) occurs at different predetermined cycles in the front image and the rear image. Therefore, the detection unit 23 may estimate the reproduction time lag from the cycle of the switching timing T. FIG.

At this time, the detection unit 23 may determine the correction timing without detecting the reproduction time lag. In this case, the correction timing may be determined so that the reproduction time shift is corrected before the reproduction time shift amount exceeds a predetermined value Th.

In the example of FIG. 5, the detection unit 23 determines the correction timing based on the acceleration of the vehicle as the travel information, but the present invention is not limited to this. For example, the correction timing may be determined based on changes in the steering angle or the imaging position (driving position).

Specifically, a period in which the steering angle changes significantly (such as when driving on a sharp curve) is determined as the correction timing. That is, by determining a period in which the image itself is largely disturbed as the correction timing, it is possible to reduce the sense of discomfort given to the user.

On the other hand, when the correction timing is determined based on the imaging position, the correction timing is determined for the period during which the image data at the point where the image changes per frame is small, such as when the vehicle is traveling on a straight road. do. Even in such a case, it is difficult for the user to notice that the correction has been made, and the user's sense of discomfort can be reduced. In this case, the detection unit 23 may determine the correction timing by using image analysis such as the background subtraction method instead of the imaging position.

Returning to the description of FIG. 3, the correction unit 24 will be described. The correction unit 24 corrects the reproduction time lag of the image data reproduced by the reproduction unit 22 . In the present embodiment, the correction unit 24 corrects the reproduction time lag at the correction timing determined by the detection unit 23 .

For example, the correction unit 24 can correct the reproduction time lag by adjusting the reproduction speed of the reproduction unit 22 . FIG. 6 is a diagram showing a specific example of correction contents by the correction unit 24. As shown in FIG.

As shown in FIG. 6, when correcting the reproduction speed of a delayed image whose reproduction time is delayed, the correction unit 24 performs double-speed reproduction as correction processing. Double-speed playback refers to playback at a higher playback speed. On the other hand, when correcting the reproduction speed of the preceding image whose reproduction time is earlier, the correction unit 24 performs slow reproduction as correction processing.

That is, the playback time of the delayed image is advanced by double-speed playback, and the playback time of the preceding image is delayed by slow playback. As a result, it is possible to correct the discrepancy in playback time.

Note that the correction unit 24 may simultaneously perform the double-speed reproduction of the delayed image and the slow reproduction of the preceding image, or may perform only one of them. Further, the correcting unit 24 can appropriately change the magnification of double-speed reproduction and slow-speed reproduction according to the amount of difference in reproduction time and the period of correction timing.

Moreover, the correction process is not limited to the above example. That is, for example, without changing the reproduction speed, the image data of the delayed image may be thinned out and reproduced.

Next, a processing procedure executed by the video playback device 1 according to the embodiment will be described with reference to FIG. FIG. 7 is a flow chart showing a processing procedure executed by the video playback device 1. As shown in FIG.

As shown in FIG. 7, the video reproducing apparatus 1 first acquires image data including a front image and a rear image (step S101), and starts reproducing the front image and the rear image based on a predetermined operation (step S102). ).

Subsequently, the video reproducing apparatus 1 determines whether or not the amount of time lag in reproduction exceeds a predetermined value Th (step S103). It is determined whether or not there is correction timing based on information (for example, acceleration) (step S104).

Subsequently, when there is a correction timing based on the travel information within a predetermined period (step S104, Yes), the video reproduction device 1 corrects the reproduction time lag at the correction timing (step S105). In addition, in the determination process of step S104, if there is no correction timing based on the travel information within a predetermined period (step S104, No), the video reproduction device 1 corrects the difference in reproduction time at the timing of file switching ( step S107).

Subsequently, the video reproducing apparatus 1 determines whether or not the reproduction of the front image and the rear image has ended (step S106), and if the reproduction has ended (step S106, Yes), the process ends. Further, when the reproduction of the front image and the rear image is being continued (step S106, No), the video reproducing apparatus 1 proceeds to the process of step S103.

Further, in the determination process of step S103, if the deviation amount is equal to or less than the predetermined value Th (step S103, No), the video reproducing apparatus 1 proceeds to the process of step S106.

As described above, the video playback device 1 according to the embodiment includes the acquisition unit 21 , the playback unit 22 and the correction unit 24 . The acquisition unit 21 acquires a plurality of files in which image data captured by a plurality of imaging devices provided in a vehicle (an example of a moving body) is divided and saved in a different format for each imaging device. . The reproduction unit 22 reads out image data from the plurality of files acquired by the acquisition unit 21 and simultaneously reproduces them for each imaging device.

The correction unit 24 corrects a reproduction time lag that occurs during simultaneous reproduction of the image data that is simultaneously reproduced by the reproduction unit 22 . Therefore, according to the video playback device 1 according to the embodiment, it is possible to eliminate the time lag of the playback time that occurs during playback.

By the way, in the above-described embodiment, the case where the imaging devices are mounted on the same vehicle has been described, but the present invention is not limited to this. That is, the imaging devices may be mounted on separate vehicles, or may be provided outside the vehicle. In other words, any image data may be used as long as it is imaged at the same playback time.

Also, in the above-described embodiment, the case where the bit rates are different between the imaging devices has been described, but the present invention is not limited to this. That is, the present invention can be applied even when the number of pieces of image data per file is different, or when the bit rate is the same between imaging devices and the data capacity per file is different. In other words, it is possible to apply the present invention to all devices with different file switching cycles during simultaneous playback.

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments shown and described above. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

Reference Signs List 1 video playback device 10 camera 10f front camera 10b rear camera 15 sensor group 21 acquisition unit 22 playback unit 23 detection unit 24 correction unit

Claims (5)

Acquiring a plurality of files in which image data captured by a plurality of imaging devices respectively provided in a moving body is divided and saved in a different format for each imaging device,
reading out the image data from the plurality of acquired files and simultaneously reproducing them for each imaging device;
detecting, based on time stamps, a difference in playback time that occurs when the image data that are played back simultaneously are reproduced;
A control unit that corrects a reproduction time lag that occurs during the simultaneous reproduction when the reproduction time lag is detected.
A video playback device. The control unit
Acquiring running information related to the running state of the moving object at the time the image data was captured;
performing the correction at a correction timing determined based on the travel information ;
The video playback device according to claim 1 . The control unit
Acquiring acceleration information of the moving body as the running state,
determining the correction timing based on a change in acceleration indicated by the acceleration information ;
3. The video playback device according to claim 2 . The control unit
correcting the reproduction time lag by changing the reproduction speed of the image data for each imaging device ;
The video playback device according to any one of claims 1 to 3 . an acquisition step of acquiring a plurality of files in which image data captured by a plurality of imaging devices respectively provided in a moving body is divided and saved in a different format for each imaging device;
a reproducing step of reading out the image data from the plurality of files obtained in the obtaining step and reproducing them simultaneously for each imaging device;
detecting, based on time stamps, a difference in reproduction time that occurs during simultaneous reproduction of the image data that is simultaneously reproduced in the reproduction step; and a correction step of correcting the video playback method.

Images