JP2021129216A - Image processing apparatus and image processing method - Google Patents

Abstract

To provide an image processing apparatus for efficiently retrieving a desired scene from a landscape video captured from a moving body, and an image processing method.SOLUTION: In an image processing apparatus 10, a video control unit includes: an acquisition unit for acquiring a landscape video captured from a moving body; a video generation unit which generates an information-added video formed by adding importance information based on information on an imaging status in each point of time in an imaging period of the landscape video, to the landscape video; and a thumbnail group image generation unit which extracts a plurality of thumbnail images based on the importance information from a plurality of frame images constituting the landscape video, and generates a thumbnail group image by arranging the thumbnail images in time series. The thumbnail group image generation unit changes the number of thumbnail images to be extracted per unit imaging time in accordance with the importance information.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing apparatus for processing an image and a processing method thereof.

In recent years, a playback device has become widespread in which a landscape moving image can be taken with a shooting device provided on a moving body such as a vehicle, and a scene shot from the vehicle can be viewed later.

For example, in Patent Document 1, a time bar corresponding to the shooting time from the start to the end of shooting of a landscape video shot by an in-vehicle camera mounted on a moving body and a position on the time bar can be instructed by a user. An instruction display image including the slider shown and the landmark icon indicating the landmark existing along the traveling route when the landscape video was shot at the corresponding position on the time bar is displayed on the display, and the landscape video is further displayed. Among them, a display system for a vehicle that displays a landscape moving image associated with a shooting time corresponding to the position of a slider on the time bar on a display is disclosed.

Further, for example, in Patent Document 2, a still image is extracted from moving image data to generate a thumbnail image, and the thumbnail image is displayed side by side at regular time intervals on a timeline to record a desired scene in the moving image. A moving image search / viewing device that makes it possible to easily find a moving image is disclosed.

Japanese Unexamined Patent Publication No. 2013-32949 Japanese Unexamined Patent Publication No. 2002-281432

However, in the display system of Patent Document 1, when trying to find a part in which a desired landscape is recorded in a moving object in which an external landscape is photographed, the slider position is changed or a landmark icon is displayed. Since it is necessary to search for a desired scene while visually recognizing the image displayed on the display while playing back from the vicinity, it may take time to reach the desired scene if the shooting time of the moving image is long.

Further, in the moving image search / viewing device of Patent Document 2, for example, when the moving speed of the moving body is slow and the change of the landscape per predetermined time is small, the thumbnail image at regular time intervals has the same landscape or image. Since a plurality of images are lined up, it may take some time to find a thumbnail image showing a desired landscape.

The present invention has been made in view of the above points, and provides an image processing device capable of efficiently searching for a desired scene from a landscape moving image taken from a moving body, and an image processing method thereof. The purpose is.

The invention according to claim 1 provides an acquisition unit that acquires a landscape moving image shot from a moving body, and importance information based on information on a shooting situation at each time point during the shooting period of the landscape moving object. A plurality of thumbnail images are extracted based on the importance information from the moving image generation unit that generates the information-holding moving image assigned to the above and the plurality of frame images constituting the landscape moving image, and the thumbnail images are arranged in chronological order. It has a thumbnail group image generation unit that generates thumbnail group images, and the thumbnail group image generation unit is characterized in that the number of thumbnail images extracted per unit shooting time is changed according to the importance information. There is.

The invention according to claim 10 is an image processing method in which an image processing apparatus extracts a thumbnail image from an information-holding moving image to which importance information is added, and is based on the importance information and the number of thumbnail images generated. An interval determination step of determining an extraction interval for extracting thumbnail images, and a frame image determination step of determining a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting the information-holding moving image based on the extraction interval. It is characterized by including an extraction step of extracting a plurality of frame images to be extracted as the thumbnail image from the information-holding moving image to generate a thumbnail group image.

The invention according to claim 11 is an image processing program executed by an image processing apparatus provided in a computer, and an extraction interval for extracting thumbnail images to the computer based on importance information and the number of thumbnail images generated. A frame image determination step of determining a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting the information-holding moving image based on the extraction interval, and the thumbnail from the information-holding moving image. It is characterized by executing an extraction step of extracting a plurality of frame images to be extracted as an image and generating a thumbnail group image.

It is a perspective view of the automobile M equipped with the image processing apparatus 10 which concerns on embodiment of this application. It is a figure which looked at the rear door of the vehicle M equipped with the image processing device 10 according to the embodiment of the present application from the rear seat of the vehicle M. It is a figure which shows an example of the operation acceptance display displayed on the touch panel 17 of the image processing apparatus 10 which concerns on embodiment of this application. It is a functional block diagram of the image processing apparatus 10 which concerns on embodiment of this application. FIG. 1 is a flowchart FC1 for generating an information-holding moving image by adding a virtual time code to a landscape moving image taken by an in-vehicle camera CM in the image processing device 10 according to the embodiment of the present application. This is an example of the elapsed time from the start of shooting of the landscape moving image of the image processing device 10 according to the embodiment of the present application, the frame numbers of a plurality of frames at predetermined times, the basic time codes of the plurality of frames, and the virtual time code. .. It is a figure which shows the relationship between the frame number of a plurality of frames constituting the information-holding video in the example of the information-holding video shown in FIG. 6, the virtual time code given to each frame number, and the relation of the extracted thumbnail image. .. These are the time code, frame number, and their importance associated with the thumbnail images SM1 to SM10 in the 10 thumbnail images SM1 to SM10 extracted from the landscape moving image of this embodiment. FIG. 2 is a flowchart FC2 for generating a thumbnail image of the image processing device 10 according to the embodiment of the present application. It is a figure which showed the display example of the touch panel 17 at the time of playing | reproduction of the information holding moving material shown in FIGS. 6, 7 and 8. It is a figure which showed the display example of the touch panel 17 at the time of playing | reproduction of the information holding moving material shown in FIGS. 6, 7 and 8.

Examples of the present invention will be described in detail below.

FIG. 1 is a perspective view of an automobile M equipped with the image processing device 10 according to the embodiment of the present application.

The image processing device 10 is an image processing device that captures a landscape moving image around the automobile M by an in-vehicle camera mounted on the automobile M and reproduces the captured landscape moving image. In the following description, a configuration capable of displaying a landscape moving image in the window of the automobile M will be described.

The image processing device 10 is mounted in, for example, the dashboard of the automobile M. The image processing device 10 is configured to be capable of reproducing and outputting a moving image acquired from the outside or held by itself.

The in-vehicle camera CM as the acquisition unit is mounted on, for example, the roof portion or the door panel portion of the automobile M, and is configured to be capable of photographing the scenery in front, side, or rear of the automobile M. The in-vehicle camera CM is communicably connected to the image processing device 10, and is configured to be able to transmit a captured still image or moving image to the image processing device.

The display 15 as a moving image display unit is, for example, a display composed of a transparent panel such as a transparent liquid crystal display or an organic EL display, attached to the window WD of the automobile M, and capable of displaying an image visible from the inside of the automobile. be. The display 15 is communicably connected to the image processing device 10, and is configured to be able to display a still image or a moving image output from the image processing device 10.

For example, the display 15 is configured so that landscape moving images of the front, side, or rear of the automobile M taken by the in-vehicle camera CM can be displayed in the front window, the door window, and the rear window, respectively. The display 15 may be, for example, a display of a car navigation system installed inside the automobile M.

FIG. 2 is a view of the rear door of the automobile M equipped with the image processing device 10 according to the embodiment of the present application as viewed from the rear seat of the automobile M.

As described above, the display 15 is composed of a transparent panel such as a transparent liquid crystal display or an organic EL display, and is attached to the window WD of the automobile M. Therefore, for example, when the moving image is not displayed on the display 15, it is possible to see the outside scenery from the inside of the automobile M through the display 15.

The touch panel 17 as an operation receiving unit is a touch panel display attached to the inside of the door panel. For example, the touch panel 17 is a touch panel display including a display 17a such as a liquid crystal panel and a touch pad 17b attached to the display surface of the display and accepting an operation with a human finger or the like.

The touch panel 17 is communicably connected to the image processing device 10 and can display an image output from the image processing device 10. Further, the touch panel 17 can accept the operation of playing, fast-forwarding, or fast-rewinding the moving image displayed on the display 15, and can transmit an instruction based on the acceptance of the operation to the image processing device 10.

Specifically, for example, on the touch panel 17, thumbnail images of the moving image being reproduced on the display 15 at predetermined time intervals are displayed side by side in the order of reproduction time.

FIG. 3 is a diagram showing an example of an operation reception display displayed on the touch panel 17 of the image processing device 10 according to the embodiment of the present application.

The operation reception display is, for example, a thumbnail image string SR in which thumbnail images SM1 to SM6 of a landscape video being played on the display 15 at predetermined time intervals are arranged in the order of playback time, and a button for instructing to switch playback or pause. It includes a PL, a button FW for instructing a fast-forward operation, a button RW for instructing a fast-rewind operation, a bar BA indicating the current playback time position of a landscape moving image, and a time display unit TM. Further, the thumbnail images included in the thumbnail image string SR are displayed while scrolling from right to left according to the passage of the playback time of the landscape moving image.

FIG. 4 is a functional block diagram of the image processing device 10 according to the embodiment of the present application. The image processing device 10 is, for example, a device in which the input unit 20, the output unit 30, the operation input unit 40, the storage unit 50, and the moving image control unit 60 cooperate with each other via the system bus 13.

The input unit 20 is an interface for acquiring data from an external device of the image processing device 10. The input unit 20 is connected to an in-vehicle camera CM provided in the automobile M. For example, the image processing device 10 can capture a landscape image from the in-vehicle camera CM via the input unit 20.

Further, the input unit 20 is a speed sensor SS that acquires the traveling speed of the automobile and a global positioning system (GPS) that is one of the global navigation satellite systems (GNSS: Global Navigation Satellite System) that acquires the position information of the automobile. : Global Positioning System) is connected to the GPS receiver GP that can receive information. That is, the image processing device 10 is configured so that the speed of the automobile M can be acquired from the speed sensor SS and GPS information can be acquired from the GPS receiver GP.

The output unit 30 is an interface that outputs data to an external device of the image processing device 10. The output unit 30 is connected to the display 17a of the display 15 and the touch panel 17 described above, and can output an image to these. Specifically, for example, the image processing device 10 outputs a moving image to the display 15 via an output unit, and outputs an operation image to be an operation reception display for a reproduction operation to the display 17a.

The operation input unit 40 is an interface for acquiring an operation input signal from an external device. The operation input unit 40 is connected to the touch pad 17b, and can acquire a signal indicating an operation received by the touch pad 17b with respect to the operation image.

The storage unit 50 is composed of, for example, a hard disk device, an SSD (solid state drive), a flash memory, and the like, and stores various programs related to the generation of thumbnail images, the reproduction of moving images, and the like in the image processing device 10. The various programs may be acquired from another server device or the like via a network (not shown), or may be recorded on a recording medium and read via various drive devices. May be good.

Various programs stored in the storage unit 50 can be transmitted via a network, and can be recorded on a computer-readable recording medium and transferred. Further, in the storage unit 50, data captured from the outside of the image processing device 10, for example, landscape moving image data taken from the camera CM, the vehicle at the time of taking the scenery moving image taken from the speed sensor SS and the GPS receiver GP. The speed and position information of M can be stored. Further, the storage unit 50 can store the data generated inside the image processing device 10.

The moving image control unit 60 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and functions as a computer. In the moving image control unit 60, the CPU realizes various functions by reading and executing various programs stored in the ROM and the storage unit 50.

The moving image control unit 60 has a moving image generation unit 63 that adds information to the landscape moving image acquired via the input unit 20 to generate an information-holding moving image for playback. The moving image generation unit 63 reads out the landscape moving image recorded in the storage unit 50 and the speed and position information data of the automobile M at each time when the scenery moving image is taken. Information based on the speed or position information of the automobile M at each time when each of the plurality of frames is photographed is added to each of the plurality of frame images constituting the read landscape moving image. For example, the moving image generation unit 63 determines the importance of each frame image according to the traveling speed of the automobile M, and adds importance information, which is information based on the importance, to the landscape moving image.

The importance is taken by, for example, the traveling speed of the automobile M, the shooting position, the appearance of the landscape associated with the position on the map information corresponding to the shooting position, or the in-vehicle camera CM based on the traveling direction of the automobile M. Judgment is made based on the shooting direction.

For example, when the automobile M is traveling at a low speed or is stopped, the surrounding landscape seen from the automobile hardly changes. Therefore, when determining the importance with respect to the traveling speed of the automobile, the moving image generation unit 63 determines that the importance of the landscape while the automobile M is traveling at a low speed or is stopped is low. On the contrary, when the automobile is traveling at high speed, the landscape moving image changes significantly in a short time, so that the importance is judged to be high.

Further, the moving image generation unit 63 determines that the importance is high when, for example, the position information of the automobile by GPS and the map database recorded in advance are linked and the automobile is in a position having a high highlight rank on the map database. On the contrary, in the landscape moving image in which the automobile M is at a position having a low highlight rank on the map database, the moving image generation unit 63 determines that the importance is low.

Further, the importance based on the location information may be determined without interlocking with the map database. For example, in a mountainous area with a high altitude, the scenery and the like may be highly spectacular. Therefore, it may be determined that the importance is high when the positioning in the range of a predetermined position is detected from the positioning information such as GPS.

Further, in the landscape moving image in which the automobile M is traveling, the landscape moving image in which the side in the traveling direction is photographed changes greatly in a short time with respect to the landscape moving image in which the front or the rear of the traveling direction of the automobile M is photographed. Therefore, the moving image generation unit 63 determines that the landscape moving image on the side in the traveling direction is more important than the scenery moving image in front of or behind the traveling direction.

Further, if the GPS receiver GP cannot acquire GPS information for a predetermined time or longer, it is highly possible that the automobile M is traveling in a structure such as a tunnel. In such a case, the landscape video is displayed. The monotonous content will continue. Therefore, the moving image generation unit 63 determines that the importance of the scenery moving image when the GPS information is not received is low with respect to the scenery moving image when the GPS information is received.

The above-mentioned importance determination conditions can be set arbitrarily. Further, the determination condition of the importance is the traveling speed of the automobile M, the shooting position, the appearance of the landscape associated with the position on the map information corresponding to the shooting position, or the shooting taken by the photographing unit with respect to the traveling direction of the automobile M. The importance is determined not only in one direction but also by a plurality of determination conditions.

In other words, the moving image generation unit 63 uses the importance as the moving speed of the automobile M, the shooting position, the appearance of the landscape associated with the position on the map information corresponding to the shooting position, and the moving direction of the automobile M. Judgment is made based on the shooting direction or the radio wave reception status of the satellite system.

The importance is not limited to the speed and position when shooting a moving image. For example, each of the plurality of frame images constituting the moving image may be compared, and the frame having a large change in the image between the frame images may be determined as having a high importance.

Further, for example, the moving image generation unit 63 assigns a virtual time code indicating a time passage different from a time code (hereinafter, referred to as a basic time code) indicating the actual time passage at the time of shooting the landscape moving image based on the importance. Generate the information-holding video.

The virtual time code is calculated from the average value of the importance for each predetermined time, and is the importance information in the form of reflecting the importance in the virtual passage of time.

Specifically, the moving image generation unit 63 has a time elapsed different from the basic time code for each of the plurality of frame images at each predetermined time of the landscape moving image according to the average value of the importance calculated for each predetermined time. Give a virtual time code including. That is, the moving image generation unit 63 functions as an importance information acquisition unit that generates an importance and a virtual time code from the data of the landscape moving image and the speed or position information of the automobile M when the scenery moving image is taken. In addition, it functions as a moving image generation unit that generates an information-holding moving image by adding a virtual time code as the generated importance information to the landscape moving image.

The virtual time code is set so that the higher the importance, the closer to the time lapse of the basic time code, and the lower the importance, the earlier the time lapse of the basic time code.

Further, the moving image generation unit 63 can arbitrarily set a predetermined time for calculating the average value of the importance.

The information-holding moving image generated by the moving image generation unit 63 is stored in the storage unit 50. The importance information given to the moving image is based on the moving speed of the automobile M, the shooting position, the appearance of the landscape associated with the position on the map information corresponding to the shooting position, and the moving direction of the automobile M. It may include one or more of directions, importance based on satellite system radio reception, and virtual timecode.

The thumbnail generation unit 65 as the thumbnail group image generation unit is framed at predetermined time intervals based on the virtual time code and the generation quantity of the thumbnail images from a plurality of frame images constituting the information-holding moving image recorded in the storage unit 50. An image is extracted and a thumbnail image to be displayed in the thumbnail display area SR of the touch panel 17 is generated. The thumbnail generation unit 65 supplies the generated thumbnail image to the display 17a via the output unit 30 and displays it on the thumbnail image string SR.

The number of thumbnail images generated can be set arbitrarily. For example, a predetermined quantity may be set for one landscape image, or may be automatically set from the recording time of the captured landscape moving image.

The reproduction control unit 67 reads out the information-retaining moving image recorded in the storage unit 50, and controls the reproduction of the information-retaining moving image in response to an instruction to play, pause, fast-forward, or fast-rewind. The reproduction control unit 67 supplies the reproduced information-holding moving image to the display 15 via the output unit 30. Further, the reproduction control unit 67 supplies the operation image corresponding to the reproduced information-holding moving image to the display 17a via the output unit 30. The reproduction control unit 67 acquires the operation signal received by the touch pad 17b and controls the reproduction of the moving image based on the operation signal.

In other words, an information-holding video in which an in-vehicle camera CM that acquires a landscape video shot from a moving object and a virtual time code based on information on the shooting status at each time during the shooting period of the landscape video are added to the landscape video. A thumbnail that generates a thumbnail group image in which a plurality of thumbnail images are extracted based on a virtual time code from a plurality of frame images constituting a landscape video and the thumbnail images are arranged in chronological order. It has a generation unit 65, and the thumbnail generation unit 65 changes the number of thumbnail images extracted per unit shooting time according to a virtual time code.

Further, the image processing device 10 includes a moving image generation unit 63 that acquires information on the shooting status at each time point during the shooting period of the landscape moving image, a storage unit 50 that records the information-holding moving image generated by the moving image generation unit 63, and the storage unit 50. The playback control unit 67 that reads the information-holding video from the storage unit 50 and controls the playback, fast-forwarding, or fast-rewinding of the landscape video to output the video, and the thumbnail images generated by the thumbnail generation unit 65 are arranged in order of playback time. The playback control unit 67 further includes a touch panel 17 that displays the thumbnail image string SR and accepts an instruction operation including a selection operation of a thumbnail image included in the thumbnail image string SR or a scroll operation for the thumbnail image string SR. Control is performed based on the operation.

In addition, the moving image generation unit 63 sets a series of virtual images based on information on the shooting status when each of the plurality of frame images is shot for each of the plurality of frame images constituting the shot landscape moving image. A virtual time code, which is a different time code, is added as importance information to generate an information-holding moving image, and the thumbnail generation unit 65 extracts a thumbnail image based on the virtual time code.

The thumbnail image may be the extracted frame image itself or a reduced version of the extracted frame image. Hereinafter, both the process of extracting a frame image from a landscape video and using it as it is as a thumbnail image and the process of reducing the extracted frame image to generate a thumbnail image will be described as simply extracting a thumbnail image. ..

Further, the moving image generation unit 63 determines the importance of each of the plurality of frame images based on the information regarding the shooting situation, and assigns a virtual time code corresponding to the importance to the plurality of frame images.

Further, the moving image generation unit 63 assigns the virtual time code so that the lower the importance, the faster the passage of time indicated by the virtual time code in the group of frame images.

FIG. 5 shows a flowchart FC1 for generating an information-holding moving image by adding a virtual time code to a landscape moving image taken by an in-vehicle camera CM in the image processing device 10 according to the embodiment of the present application. Here, an example of setting the importance of the landscape image according to the traveling speed of the automobile M will be described.

First, the moving image generation unit 63 determines in step S101 whether or not the shooting of the landscape moving image is completed (step S101). This determination may be made, for example, by accessing the camera CM and confirming whether or not the moving image shooting is completed. Further, this determination may be made by, for example, whether or not the reception of the moving image data from the camera CM is completed.

When it is determined that the shooting of the landscape moving image is completed (step S101: Yes), the moving image generation unit 63 travels the landscape moving image recorded in the storage unit 50 and the vehicle M acquired by the speed sensor SS during the shooting period. Read the speed (step S102).

Next, the moving image generation unit 63 assigns the traveling speed of the automobile M at the time when the frame is taken to each of the plurality of frame images constituting the read landscape moving image (step S103).

The moving image generation unit 63 determines the importance according to the traveling speed of the automobile M in each of the plurality of frames constituting the landscape moving image to which the traveling speed of the automobile M is given, and further assigns the importance to the landscape moving image. (Step S104).

Next, the moving image generation unit 63 divides the landscape moving image in which the importance is given to each of the frames at predetermined time intervals, and calculates the average value of the importance for each predetermined time (step S105).

After that, the moving image generation unit 63 assigns a virtual time code including a time passage different from the basic time code according to the average value of the importance for each predetermined time. (Step S106).

After the end of step S106, the moving image generation unit 63 records the landscape moving image to which the virtual time code is attached in the storage unit 50 as an information holding moving image (step S107).

By performing the above processing, an information holding video to which the importance and the virtual time code are given is generated from the landscape video taken by the in-vehicle camera CM and the traveling speed of the automobile M acquired by the speed sensor SS.

In this embodiment, it has been described that the traveling speed of the automobile M, the importance, and the virtual time code are given to the landscape video after the landscape video is shot.

However, the traveling speed of the automobile M, the importance, and the virtual time code may be assigned to the landscape video while shooting the landscape video. In the case of performing while shooting, each time a frame image of a landscape moving image is supplied, the traveling speed of the automobile M at the time of the frame image is given, and the importance is determined and given. In addition, every time the recording time of a landscape movie exceeds a predetermined time for calculating the average value of importance, the average value of importance is calculated and a virtual time code is assigned.

Next, the virtual time code assigned to the information-holding video will be described with reference to FIG.

FIG. 6 shows an example of an elapsed time from the start of shooting of the image processing device 10 according to the embodiment of the present application, a frame number of a plurality of frames at a predetermined time, and a basic time code of the plurality of frames. Further, the average value of the importance for each predetermined time calculated by the moving image generation unit 63, the magnification of the virtual time code with respect to the basic time code, and the assigned virtual time code are shown.

In this embodiment, in the 600-second landscape video shot by the in-vehicle camera CM, the shooting frame rate is set to 30 fps, the predetermined time for calculating the average value of importance is set to 60 seconds, and the importance is divided according to the traveling speed. A case where the degree is set in four stages of 0 to 3 will be described.

In addition, the magnification of the virtual time code is the same as the basic time code when the importance is 3, 1/2 of the basic time code when the importance is 2, and 1/3 of the basic time code when the importance is 1. In the case of degree 0, the setting will be described as a setting in which a virtual time code is assigned at a time code interval of 1/5 of the basic time code.

Since the basic time code and the virtual time code progression in the case of importance 3 proceed according to the frame rate of the captured landscape video, the progression speed is 1/30 second (about 0.033 seconds) per frame. Have. Further, the virtual time code progression in the case of importance 2 has a progress speed of 1/60 second (about 0.017 seconds) per frame, and the virtual time code progression in the case of importance 1 has a progress speed of 1/60 second (about 0.017 seconds) per frame. It has a progress speed of 1/90 second (about 0.011 second), and the virtual time code progress at the importance of 0 has a progress speed of 1/150 second (about 0.007 second) per frame.

That is, importance 3 is the same time lapse speed as the basic time code, importance 2 is about twice the time lapse speed of the basic time code, and importance 1 is about 3 times the time lapse speed of the basic time code. At 0, a virtual time code having a time elapsed speed about 5 times that of the basic time code is given.

In FIG. 6, the average value of the calculated importance is 3 in the elapsed time from the start of shooting to 180 seconds. Therefore, the virtual time code is given at a progress speed of 1/30 second per frame, which is equivalent to the basic time code.

In addition, the average value of the calculated importance is 1 during the elapsed time of 180 to 300 seconds from the start of shooting. Therefore, the virtual time code is given at a progress speed of 1/90 second per frame, which is a progress speed of 1/3 of the basic time code. That is, in 120 seconds of the elapsed time of 180 to 300 seconds from the start of shooting, the basic time code is the same progress speed as the actual shooting time of this period, but the virtual time code is the actual shooting time of 120 seconds in this period. On the other hand, it is given at a progress rate of 40 seconds.

In addition, the average value of the calculated importance is 2 in the elapsed time of 300 to 420 seconds from the start of shooting. Therefore, the virtual time code is given at a progress speed of 1/60 second per frame, which is a progress speed of 1/2 of the basic time code. That is, in 120 seconds of the elapsed time of 300 to 420 seconds from the start of shooting, the virtual time code is given at a progress speed of 60 seconds with respect to the actual shooting time of 120 seconds in this period.

Further, the average value of the calculated importance is 0 during the elapsed time of 420 to 480 seconds from the start of shooting. Therefore, the virtual time code is given at a progress speed of 1/150 second per frame, which is a progress speed of 1/5 of the basic time code. That is, in 60 seconds of the elapsed time of 420 to 480 seconds from the start of shooting, the virtual time code is given at a progress speed of 12 seconds with respect to the actual shooting time of 60 seconds in this period.

In addition, the average value of the calculated importance is 2 during the elapsed time of 480 to 600 seconds from the start of shooting. Since the progress speed of the virtual time code during this period is the same as that of the above-mentioned case of importance 2, the description thereof will be omitted.

That is, in a 600-second landscape video shot by the in-vehicle camera CM at 30 fps, the basic time code is constant at 1/30 second per frame, so the basic time code is 0 for the number of frames with frame numbers 1 to 18000. A time code of ~ 599.967 seconds is given. On the other hand, in the virtual time code, the progress speed of the time code changes according to the importance of each predetermined time, so that the virtual time code is 0 to 351.967 seconds for the number of frames of frame numbers 1 to 18000. A time code is given.

FIG. 7 shows the relationship between the frame numbers of a plurality of frames constituting the information-holding video in the example of the information-holding video shown in FIG. 6, the virtual time code assigned to each frame number, and the relationship between the extracted thumbnail images. It is a figure which shows.

The horizontal axis shows the frame numbers of a plurality of frames constituting the information-holding moving image. In addition, the elapsed time (basic time code) from the start of shooting corresponding to each frame number is shown. The vertical axis represents the virtual time code, and the horizontal axis represents the virtual time code corresponding to the frame number.

In this embodiment, a case where 10 thumbnail images are extracted from the information-holding moving image shown in FIG. 6 will be described.

As shown in FIG. 7, since the calculated importance of the section of frame numbers 1 to 5400 (elapsed time from the start of shooting to 0 to 180 seconds) is 3, the virtual time code per frame is the basic time code. Granted with the same elapsed time as.

Since the calculated importance of the section of frame numbers 5401 to 9000 (elapsed time 180 to 300 seconds from the start of shooting) is 1, the virtual time code per frame is 1/3 of the basic time code. Granted by.

Since the calculated importance of the section of frame numbers 9001 to 12600 (elapsed time from the start of shooting 300 to 420 seconds) is 2, the virtual time code per frame is half the elapsed time of the basic time code. Granted by.

Since the calculated importance of the section of frame numbers 12601 to 14400 (elapsed time 420 to 480 seconds from the start of shooting) is 0, the virtual time code per frame is 1/5 of the basic time code. Granted by.

Since the calculated importance of the section of frame numbers 14401 to 18000 (elapsed time from the start of shooting 480 to 600 seconds) is 2, the virtual time code per frame is half the elapsed time of the basic time code. Granted by.

As a result, for the information-holding video of this embodiment, the relationship between the plurality of frames constituting the information-holding video and the virtual time code is shown as a solid line in the figure.

The virtual time code is assigned so that the lower the importance, the faster the progress speed of the virtual time code. Therefore, as shown in FIG. 7, the lower the importance, the shorter the passage of time per frame, and the smaller the slope of the virtual time code.

The thumbnail generation unit 35 extracts a frame image for each predetermined time of the virtual time code as a thumbnail image based on the virtual time code.

Therefore, when 10 thumbnail images are generated from the landscape video of this embodiment, the virtual time code of the information-holding video is assigned a time code of 0 to 351.967 seconds, so that the thumbnail generation unit 65 is virtual. Ten frame images are extracted every 35.2 seconds from 0 seconds of the time code. That is, the thumbnail generation unit 65 extracts frame images at regular time intervals of about 35.2 seconds on the vertical axis in the figure.

The frame image extracted as a thumbnail image starts from 0 seconds when the virtual time code is 0 seconds, 35.2 seconds, 70.4 seconds, ..., 281.6 seconds and 316.8 seconds. Frame images at regular time intervals of approximately 35.2 seconds are selected. At this time, since the progress speed of the virtual time code changes according to the importance of each predetermined time, the basic time code and frame number of the extracted thumbnail image are not every fixed number.

Therefore, as shown in FIG. 7, the lower the importance, the smaller the number of thumbnail images extracted per unit shooting time.

That is, according to the present invention, since the progress speed of the virtual time code is changed according to the importance of each predetermined time, it is possible to change the number of thumbnail images extracted per unit shooting time according to the importance. It becomes.

FIG. 8 shows the time code, frame number, and importance thereof associated with the thumbnail images SM1 to SM10 in the 10 thumbnail images SM1 to SM10 extracted from the landscape moving image of this embodiment.

The upper part of FIG. 8 shows the information of the thumbnail image generated by the frame image extracted based on the conventional basic time code as a comparative example, and the lower part of FIG. 8 shows the frame extracted based on the virtual time code according to the present invention. It shows the information of the thumbnail image generated by the image. In the lower part of FIG. 8, the basic time code corresponding to the frame number of the thumbnail image extracted based on the virtual time code is also shown.

As shown in the upper part of FIG. 8, when the thumbnail image is extracted based on the basic time code, the thumbnail generation unit 65 has the basic time code of 0 seconds, 60 seconds, 120 seconds, ..., 480 seconds. And the frame image at regular time intervals of 60 seconds at 540 seconds is extracted as a thumbnail image. Further, the frame numbers of the extracted thumbnail images are for each fixed number of frames from frame numbers 1 to 1800 of frame numbers 1, 1801, 3601, ..., 14401, 16201. The importance of the extracted frame images is 3 for the 1st to 3rd frame images, 1 for the 4th to 5th frame images, and 1 for the 6th to 7th frames. The importance of the frame image is 2, the importance of the 8th frame image is 0, and the importance of the 9th to 10th frame images is 2.

Further, as shown in the lower part of FIG. 8, when the thumbnail image is extracted based on the virtual time code, the thumbnail generation unit 65 uses the virtual time code of 0 seconds, 35.2 seconds, 70.4 seconds, and so on. ..., Frame images at regular time intervals of 35.2 seconds at 281.6 seconds and 316.8 seconds are extracted as thumbnail images. The frame numbers of the thumbnail images to be extracted are the frame images of frame numbers 1, 1056, 2112, 3168, 4224, 5280, 8211, 10586, 12845 and 15890. Further, as for the importance corresponding to the extracted frame image, the importance of the 1st to 6th frame images is 3, the importance of the 7th frame image is 1, and the importance of the 8th frame image is 1. The importance is 2, the importance of the 9th frame image is 0, and the importance of the 10th frame image is 2.

That is, the thumbnail generation unit 65 extracts a small number of thumbnail images in the low-importance time section determined to have low highlights in the captured landscape video, and the high-importance time section determined to have high highlights. In, many thumbnail images are extracted.

In addition, as shown in the basic time code corresponding to the frame image of the thumbnail image extracted based on the virtual time code, the interval of the shooting time of the extracted thumbnail image becomes long in the section of low importance, and the importance is high. In the section, the interval between shooting times becomes shorter. That is, in the low-importance section, the extraction time interval of the thumbnail image or the frame image used to generate the thumbnail image becomes long, and in the high-importance section, the thumbnail image or the frame image used to generate the thumbnail image is generated. The extraction time interval of is shortened.

As a result, many thumbnail images in the vicinity of the desired scene, which is of high importance, are extracted from the captured landscape video, so that the desired scene can be efficiently searched from the captured landscape video.

In addition, instead of setting the importance of the landscape image according to the traveling speed of the automobile M, the moving image generation unit 63 shows the shooting position and the viewing degree of the scenery associated with the position on the map information corresponding to the shooting position. , The importance of the landscape image may be set based on the shooting direction based on the moving direction of the automobile M or the radio wave reception status of the satellite system. Alternatively, the moving image generation unit 63 may use the traveling speed, the shooting position, the viewing degree of the landscape associated with the position on the map information corresponding to the shooting position, the shooting direction based on the moving direction of the automobile M, and the radio wave of the satellite system. The importance of the landscape image may be set based on the combination of a plurality of elements of any one of the reception conditions.

FIG. 9 shows a flowchart FC2 when generating a thumbnail image of the image processing device 10 according to the embodiment of the present application.

First, the reproduction control unit 67 determines whether or not the reproduction of the landscape moving image is instructed. This determination may be made, for example, depending on whether or not the operation for instructing the reproduction of the landscape moving image is accepted on the touch panel 17 (step S201).

Next, the reproduction control unit 67 reads out the information-holding moving image recorded in the storage unit 50 and instructed to reproduce (step S202).

As an interval determination step, the thumbnail generation unit 65 determines the extraction interval of the virtual time code for extracting the thumbnail image from the virtual time code of the information holding video read by the playback control unit 67 and the quantity setting of the thumbnail images to be generated ( Step S203).

Next, as a frame image determination step, the thumbnail generation unit 65 determines a frame number to be a thumbnail image from the thumbnail image extraction interval of the determined virtual time code (step S204).

Next, as an extraction step, the thumbnail generation unit 65 extracts the frame image of the determined frame number as a thumbnail image (step S205).

After that, the thumbnail generation unit 65 arranges the extracted thumbnail images in chronological order and supplies them to the output unit 30 (step S206).

After the processing in step S206 is completed, the thumbnail generation unit 65 ends the processing, and then the reproduction control unit 67 executes the reproduction processing of the information-holding moving image.

By the above processing, it is possible to generate a thumbnail image group in which there are many thumbnail images in the high-importance section and few thumbnail images in the low-importance section.

In other words, the image processing method of the present invention is an image processing method in which the image processing device 10 extracts a thumbnail image from an information-holding moving image to which a virtual time code is assigned, and is based on the virtual time code and the number of thumbnail images generated. An interval determination step of determining an extraction interval for extracting thumbnail images, and a frame image determination step of determining a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting an information-holding moving image based on the extraction interval. Includes an extraction step of extracting a plurality of frame images to be extracted as thumbnail images from an information-holding moving image to generate a thumbnail group image.

10A and 10B are diagrams showing a display example of the touch panel 17 at the time of reproducing the information-holding moving image shown in FIGS. 6, 7 and 8.

FIG. 10A shows the operation reception display of the touch panel 17 during playback in the section of importance 3, and FIG. 10B shows the operation reception display of the touch panel 17 during playback in the section of importance 1. The length of the arrow in the figure indicates the scrolling speed. The time displayed at the lower left of the thumbnail images SM1 to SM10 is a playback time (basic time code) corresponding to each frame of the displayed thumbnail images SM1 to SM10.

The reproduction control unit 67 controls the reproduction speed, the reproduction time display of the time display unit TM, and the scroll speed of the thumbnail image sequence SR at the time of reproduction based on the basic time code.

For example, the scroll speed during playback in the section of the thumbnail image SM3 having an importance of 3 in FIG. 10A is 105.6 seconds from the beginning of the thumbnail image SM3 to which the basic time code of 70.4 seconds is given. It takes about 35 seconds to scroll to the beginning of the thumbnail image SM4 to which the time code is attached.

Further, the scroll speed at the time of reproduction in the section of the thumbnail image SM6 having the importance of FIG. 10B is the basic 352.9 seconds from the beginning of the thumbnail image SM7 to which the basic time code of 273.7 seconds is given. The interval to the beginning of the thumbnail image SM8 to which the time code is assigned is scrolled in about 79 seconds.

In other words, when the frame image based on the thumbnail image SM4 is displayed as a still image constituting the moving image displayed on the display 15, the thumbnail image SM4 is approached on the bar BA indicating the current playback time position. When the base frame image of the thumbnail image SM8 is displayed as a still image constituting the moving image displayed on the display 15, the thumbnail image SM8 is placed on the bar BA indicating the current playback time position. , The scroll speed of the thumbnail image string SR is controlled.

That is, the playback time (basic time code interval) of the moving image corresponding to the thumbnail image having a low importance is longer than the playback time corresponding to the thumbnail image having a high importance. Therefore, the scroll speed of the thumbnail image string SR during video reproduction in the less important section becomes slower, and the scroll speed of the thumbnail image string SR during video reproduction in the higher importance section becomes faster.

Further, in the thumbnail image displayed in the thumbnail image string SR, the thumbnail images are displayed less at long time intervals than the recording time of the captured landscape video in the section of low importance, and in the section of high importance. Is displayed many thumbnail images at short time intervals with respect to the recording time of the captured landscape video.

When the user tries to search for a desired scene from the captured landscape video, for example, the user has an idea of where the desired scene is recorded in the landscape video, and the user has an idea. The suitable shooting time interval between the plurality of thumbnail images displayed as the thumbnail image string SR is different from that in the case where the thumbnail image string SR is not used. In the former case, the thumbnail image string SR, which is a thumbnail image around the shooting time that has already been guessed and the shooting time interval between the thumbnail images is relatively short, is effective. In the latter case, which is the desired scene first? Since it is necessary to guess whether or not the images are recorded around, the thumbnail image string SR having a relatively long shooting time interval between the thumbnail images is effective.

In order to cope with this, the image processing device 10 is configured so that the shooting time interval between adjacent thumbnail images of a plurality of thumbnail images displayed in the thumbnail image string SR as a part of the operation reception display can be changed by the user. It is preferable to have.

Specifically, the image processing device 10 includes an operation icon for designating the time scale of the thumbnail image string SR by the user in the operation reception display of the touch panel 17, and the thumbnail generation unit 65 displays the operation reception display of the user. The extraction interval of the virtual time code for extracting the thumbnail image to be displayed as the thumbnail image string SR is determined according to the time scale specified by the operation. This makes it possible to arbitrarily change the time scale of the thumbnail image sequence to search for a desired scene from a limited time range, or to search for a desired scene from a relatively long time range. It is possible to generate a valid thumbnail image string in.

In the image processing device 10 of the above-described embodiment, a method of extracting a frame image from an information-holding moving image and generating a thumbnail image after a reproduction instruction is given has been described. However, the generation of thumbnail images is not limited to this. For example, after the shooting of the landscape video is completed and the video generation unit 63 assigns a virtual time code to the landscape video to generate the information-holding video (steps S101 to S106 in FIG. 5), a thumbnail is provided for the information-holding video. The generation unit 65 may execute the processes of steps S202 to S205 of FIG. 9 to record the information-holding moving image and the thumbnail image in the storage unit 50. As a result, by generating and recording thumbnail images in advance, it is possible to quickly execute the reproduction process of the landscape moving image at the time of reproduction.

In the embodiment, the importance of the landscape video taken from the in-vehicle camera CM mounted on the automobile M is determined according to the traveling speed and the shooting position of the automobile M, and the thumbnail generation unit 65 determines the importance according to the importance. The image processing apparatus 10 for variably changing the generation quantity of thumbnail images per unit shooting time has been described. However, the image processing device according to the present invention is not limited to landscape moving images from automobiles.

For example, a moving image in which importance is given to a moving image taken by a smartphone, a handy camera, or the like may be generated. Further, the captured information-holding moving image may be recorded in an external recording device such as a USB memory or HDD, and processed and reproduced by a device such as a PC.

Further, in this embodiment, the method of calculating the average value of importance for each predetermined section of the basic time code and assigning the virtual time code for each predetermined section has been described, but the method of assigning the virtual time code is described. Not limited to this. For example, the passage of time for each frame may be assigned according to the importance to each of the plurality of frames constituting the landscape moving image to which the importance is given. As a result, it is possible to efficiently generate a thumbnail image of a section having a high importance even when the importance changes significantly in a predetermined section of the basic time code.

Further, in this embodiment, an image processing device 10 that generates one moving image data in which a virtual time code is added to a landscape moving image and performs moving image processing of the moving image data has been described. However, the playback control method is not limited to generating one moving image data. For example, a landscape video to which the importance and the virtual time code are not assigned, and information data indicating the frame number and the virtual time code for the frame number are generated from the landscape video. The thumbnail generation unit 65 may determine a frame number to be extracted as a thumbnail image from the information data, and may extract the frame image of the frame number as a thumbnail image from the landscape moving image. This makes it possible to reduce the file data capacity of landscape moving images.

Further, in the present embodiment, the method of determining the importance from the traveling speed of the automobile M and generating the virtual time code from the importance has been described, but the method of generating the virtual time code is not limited to this. For example, a plurality of threshold values may be set for the running average speed of the automobile M for each predetermined time, and a virtual time code corresponding to the plurality of thresholds of the running average speed may be set. This makes it possible to directly generate a virtual time code from the traveling speed of the automobile M at predetermined time intervals.

That is, it suffices that information on the importance indicating a specific highlight can be added to each of the plurality of frame images constituting the captured image, and the number of thumbnail images generated per unit shooting time can be variably changed according to the importance. ..

10 Image processing device 15 Display 17 Touch panel 20 Input unit 30 Output unit 40 Operation input unit 50 Storage unit 60 Video control unit 63 Video generation unit 65 Thumbnail generation unit 67 Playback control unit

Claims (12)

An acquisition unit that acquires landscape videos taken from a moving body,
A video generation unit that generates an information-holding video that is attached to the landscape video with importance information based on information about the shooting status at each time point during the shooting period of the landscape video.
A thumbnail group image generation unit that extracts a plurality of thumbnail images based on the importance information from a plurality of frame images constituting the landscape moving image and generates a thumbnail group image in which the thumbnail images are arranged in chronological order. Have,
The thumbnail group image generation unit is an image processing device characterized in that the number of thumbnail images extracted per unit shooting time is changed according to the importance information. The moving image generation unit sets a series of virtual images based on information on the shooting situation when each of the plurality of frame images is shot for each of the plurality of frame images constituting the shot landscape movie. A virtual time code, which is a time code, is added as the importance information to generate the information holding video.
The image processing apparatus according to claim 1, wherein the thumbnail group image generation unit extracts thumbnail images based on the virtual time code. The moving image generation unit determines the importance of each of the plurality of frame images based on the information regarding the shooting situation, and assigns the virtual time code corresponding to the importance to the plurality of frame images. The image processing apparatus according to claim 2. The third aspect of the present invention is characterized in that the moving image generation unit assigns the virtual time code so that the lower the importance, the faster the passage of time indicated by the virtual time code in the group of frame images. The image processing apparatus described. The image processing apparatus according to any one of claims 2 to 4, wherein the moving image generating unit determines the importance based on the moving speed of the moving body. The image processing apparatus according to any one of claims 2 to 4, wherein the moving image generating unit determines the importance based on the GNSS positioning information of the moving body or the shooting position. The image processing apparatus according to any one of claims 2 to 4, wherein the moving image generating unit determines the importance based on a position on a map in comparison with a shooting position of the moving body. .. The image processing apparatus according to any one of claims 2 to 4, wherein the moving image generating unit determines the importance based on a shooting direction based on a moving direction of the moving body. An information acquisition unit that acquires information on the shooting status at each time point during the shooting period of the landscape moving image, and an information acquisition unit.
A video recording unit that records the information-holding video generated by the video generation unit, and a video recording unit.
A playback control unit that reads the information-holding video from the video recording unit, controls the landscape video to be played, fast-forwarded, or fast-rewinded, and outputs the video.
A thumbnail image string in which the thumbnail images generated by the thumbnail group image generation unit are arranged in the order of playback time is displayed, and the operation of selecting the thumbnail image included in the thumbnail image string or the operation of scrolling the thumbnail image string is included. Including an operation reception unit that accepts instruction operations,
The image processing apparatus according to any one of claims 1 to 8, wherein the reproduction control unit performs the control based on the instruction operation. An image processing device is an image processing method for extracting thumbnail images from an information-holding video to which importance information is given.
An interval determination step of determining an extraction interval for extracting thumbnail images based on the importance information and the number of thumbnail images generated, and
A frame image determination step of determining a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting the information-holding moving image based on the extraction interval.
An image processing method comprising an extraction step of extracting a plurality of frame images to be extracted as thumbnail images from the information-holding moving image to generate thumbnail group images. An image processing program executed by an image processing device provided in a computer.
An interval determination process for determining the extraction interval for extracting thumbnail images based on importance information and the number of thumbnail images generated, and
A frame image determination step of determining a plurality of frame images to be extracted as thumbnail images from a plurality of frame images constituting an information-holding moving image based on the extraction interval.
An image processing program characterized by executing an extraction step of extracting a plurality of frame images to be extracted as thumbnail images from the information-holding moving image to generate thumbnail group images. A computer-readable recording medium comprising storing the image processing program according to claim 11.

Images