JP7052613B2 - Display control device, display control method and program - Google Patents

Description

The present invention relates to a display control device, a display control method and a program.

For example, there is a device such as a drive recorder that stores an image of an in-vehicle camera that stores and displays an image captured while moving. For example, Patent Document 1 describes a display control system that displays thumbnail images related to an image recorded on a drive recorder.

Japanese Unexamined Patent Publication No. 2013-235395

An image captured while moving may have a large amount of data due to reasons such as being captured for a long time. In such a case, it may be difficult to detect the scene required by the viewer. For example, even if thumbnail images are displayed side by side as in Patent Document 1, since the thumbnail images are merely arranged in a matrix, the thumbnail images cannot be distinguished from each other, and there is a possibility that a required scene cannot be detected. be. Therefore, it is required to be able to appropriately detect the scene desired by the viewer from the captured image having a large amount of data.

In view of the above problems, it is an object of the present invention to provide a display control device, a display control method, and a program for appropriately detecting a scene desired by a viewer from a captured image.

The display control device according to one aspect of the present invention has an image information acquisition unit that acquires information of a plurality of images captured while moving, and a set time designation unit that specifies a plurality of set times based on the image information. A thumbnail acquisition unit that acquires the image captured at the set time as a thumbnail, and a plurality of the acquired thumbnails are arranged in a predetermined direction in the order of the captured time series, and in the predetermined direction. It has a display image generation unit that generates a display image to be displayed on the display unit so that the images are displayed side by side based on the position information indicating the position where the image is captured in different directions.

The display control device according to one aspect of the present invention has a thumbnail acquisition unit that acquires a thumbnail of an image that is a 360-degree image, and a spherical display area in a virtual three-dimensional space in which the acquired thumbnail is displayed on the display unit. A display image generation unit that generates a display image displayed on the display unit so that the thumbnail is displayed while changing the angle by displaying the thumbnail on the outer peripheral surface and rotating the display area. Have.

The display control method according to one aspect of the present invention includes an image information acquisition step of acquiring information of a plurality of images captured while moving, and a set time designation step of designating a plurality of set times based on the image information. A thumbnail acquisition step of acquiring the image captured at each of the set times as a thumbnail, and a plurality of the acquired thumbnails arranged in the order of the time series captured in the predetermined direction, and in the predetermined direction. It has a display image generation step of generating a display image to be displayed on the display unit so that the images are displayed side by side based on the position information indicating the position where the image is captured in a direction different from the above.

The program according to one aspect of the present invention includes an image information acquisition step of acquiring information of a plurality of images captured while moving, a set time designation step of designating a plurality of set times based on the image information, and a set time designation step. The thumbnail acquisition step of acquiring the image captured at each of the set times as a thumbnail, and the plurality of acquired thumbnails are arranged in the order of the time series captured in the predetermined direction, and are different from the predetermined direction. In the direction, the computer is made to execute a display image generation step of generating a display image displayed on the display unit so that the images are displayed side by side based on the position information indicating the captured position.

According to the present invention, it is possible to appropriately detect the scene desired by the viewer from the captured image.

FIG. 1 is a schematic view of a vehicle according to the present embodiment. FIG. 2 is a diagram showing an example of movement of a vehicle. FIG. 3 is a schematic block diagram of the display control system according to the present embodiment. FIG. 4 is a diagram for explaining the set time. FIG. 5 is a diagram showing an example of a display image. FIG. 6 is a diagram showing an example of an image in the display image mode. FIG. 7 is a diagram showing another example of the displayed image. FIG. 8 is a diagram showing an example of a display image. FIG. 9 is a diagram showing another example of the displayed image. FIG. 10 is a flowchart illustrating a display method of the displayed image. FIG. 11 is a block diagram showing another example of the display control system according to the present embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The present invention is not limited to the embodiments described below.

(Configuration of display control system)
FIG. 1 is a schematic view of a vehicle according to the present embodiment. The display control system 1 according to the present embodiment is a system for controlling the display of the image captured by the image pickup device 10. As shown in FIG. 1, the display control system 1 includes an image pickup device 10, an in-vehicle device 12, an external device 14, and a display terminal 16.

FIG. 2 is a diagram showing an example of movement of a vehicle. The image pickup device 10 is an image pickup device mounted on the vehicle V to capture an image. The image pickup device 10 captures an image while moving together with the vehicle V. Therefore, it can be said that the image pickup apparatus 10 captures an image for each position while moving, that is, changing the position. FIG. 2 is a view of an example of the movement of the vehicle V from above in the vertical direction, in which the direction X0 is the horizontal direction in the absolute coordinates and the direction Y0 is the horizontal direction orthogonal to the direction X0 in the absolute coordinates. Here, the moving direction of the vehicle V, that is, the moving direction of the vehicle V in absolute coordinates is defined as the traveling direction V1. The vehicle V moves while changing the traveling direction V1. In the example of FIG. 2, the traveling direction V1 is switched from the direction Y0 side to the direction X0 side, further switched to the direction Y0 side, and then switched to the direction X0 side. However, the change in the traveling direction V1 in FIG. 2 is an example. The image pickup device 10 is not limited to being mounted on the vehicle V as long as it captures an image while moving in the traveling direction V1. For example, the image pickup device 10 may be carried by a walking user and capture an image while moving as the user walks.

Further, the image pickup apparatus 10 continuously captures images at predetermined frame rates to capture a moving image (a plurality of continuous images) as an image. However, the image pickup apparatus 10 is not limited to capturing a moving image, and may capture a still image at predetermined timing intervals. The image pickup apparatus 10 according to the present embodiment is a 360-degree camera that captures a wide-area image, that is, a 360-degree image. The 360-degree image refers to a 360-degree panoramic image, and is an image obtained by capturing the outside of the radial direction of a predetermined axis for one round along the circumferential direction with respect to the predetermined axis, that is, for 360 degrees. Furthermore, the 360-degree image may be either a spherical panoramic image, a hemispherical panoramic image, or a 360-degree horizontal panoramic image.

As shown in FIG. 1, in the present embodiment, the image pickup apparatus 10 includes lenses 10a and 10b provided opposite to each other, and each of them captures a hemispherical panoramic image to obtain a hemispherical panoramic image. The whole celestial sphere panoramic image is taken by synthesizing. In this embodiment, the lenses 10a and 10b are ultra-wide-angle lenses, that is, fisheye lenses. The lens 10a is provided in the vehicle V and captures a 360-degree image in the vehicle V. The lens 10b is provided outside the vehicle V and captures a 360-degree image around the vehicle V. That is, it can be said that the image pickup apparatus 10 images the inside of the vehicle V and the outside of the vehicle V. However, the image pickup apparatus 10 is not limited to forming an omnidirectional panoramic image by synthesizing the captured images of the two lenses 10a and 10b in this way, and may form a 360-degree image with the image of one lens. good.

Further, the image pickup apparatus 10 is installed on the ceiling of the vehicle V, the lens 10a is provided on the inside of the vehicle, and the lens 10b is provided on the outside of the vehicle. However, the image pickup device 10 is not limited to being provided on the ceiling of the vehicle V, and may be provided at an arbitrary position of the vehicle V. For example, the image pickup apparatus 10 may be installed in the window of the vehicle V, the lens 10a may be provided toward the inside of the vehicle, and the lens 10b may be provided toward the outside of the vehicle. That is, the lens 10b is provided so as to take an image of the outside of the vehicle from the window of the vehicle V. Further, in the present embodiment, the position where the image pickup apparatus 10 is attached is predetermined in the vehicle V. However, for example, the user may freely set the mounting position of the image pickup apparatus 10, and the mounting position may be changed for each use.

In this way, the image pickup apparatus 10 captures a 360-degree image, and captures the inside of the vehicle V and the outside of the vehicle V. However, the image pickup device 10 is not limited to a device that captures a 360-degree image as long as it captures an image while moving, and may, for example, capture an image having a normal angle of view. That is, the image pickup apparatus 10 according to the present embodiment is at least either one that captures an image while moving or one that captures a 360-degree image.

Returning to FIG. 1, the in-vehicle device 12 is a device provided in the vehicle V and is a storage device for storing an image captured by the image pickup device 10. That is, the in-vehicle device 12 is a drive recorder. Further, the external device 14 is a server provided outside the vehicle V, and acquires an image acquired by the in-vehicle device 12, that is, an image captured by the image pickup device 10. As will be described in detail later, the external device 14 generates a plurality of thumbnails from the image captured by the image pickup device 10 to generate a display image in which the thumbnails are arranged. The display terminal 16 is a display device that displays an image acquired by the external device 14, that is, an image captured by the image pickup device 10, and also displays a display image generated by the external device 14.

FIG. 3 is a schematic block diagram of the display control system according to the present embodiment. As described above, the image pickup apparatus 10 shown in FIG. 3 captures the inside of the vehicle V and the surroundings of the vehicle V while moving with the movement of the vehicle V. Hereinafter, the image captured by the image pickup apparatus 10 will be referred to as an image P. It can be said that the image P is an image captured while moving. The image pickup apparatus 10 may include a microphone (not shown) and may record a voice (for example, a voice in the vehicle V) together with the image P.

The in-vehicle device 12 includes an input unit 30, a display unit 32, a communication unit 34, a storage unit 36, and a control unit 38. The input unit 30 is a mechanism capable of inputting information from the user, such as a button or a touch panel. The display unit 32 is a display such as a touch panel that displays various information such as images. Further, the in-vehicle device 12 may be configured to be able to output sound by a speaker (not shown). In this case, the in-vehicle device 12 can reproduce the sound recorded by the image pickup device 10 on the speaker while displaying the image P captured by the image pickup device 10 on the display unit 32. Further, the control unit 38 operates a map application to acquire the current position in the absolute coordinates of the vehicle V by GPS (Global Positioning System), and the display unit 32 indicates the position of the vehicle V on the map. Navigation information can be displayed. The in-vehicle device 12 may be a mobile terminal such as a smartphone or a tablet.

The communication unit 34 is configured to transmit and receive data by communicating with other devices such as an image pickup device 10 and an external device 14 under the control of the control unit 38. The storage unit 36 is a memory that stores various information such as the calculation contents of the control unit 38 and the image P. The storage unit 36 includes, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), and an external storage device such as a flash memory (Flash Memory).

The control unit 38 is an arithmetic unit, that is, a CPU (Central Processing Unit). The control unit 38 has an image acquisition unit 40 and a position information acquisition unit 42. The image acquisition unit 40 and the position information acquisition unit 42 read out the software (program) stored in the storage unit 36, and the CPU performs a predetermined process to execute the process described later.

The image acquisition unit 40 acquires the image P captured by the image pickup apparatus 10 via the communication unit 34. The image acquisition unit 40 stores the acquired image P in the storage unit 36 in association with the time when the image was taken. The image acquisition unit 40 may acquire audio data recorded in the image pickup apparatus 10 in addition to the image P.

The position information acquisition unit 42 acquires the position information of the vehicle V. The position information acquisition unit 42 acquires the current position in the absolute coordinates of the vehicle V as the position information. The position information acquisition unit 42 acquires the current position of the vehicle V in the absolute coordinate system by communicating with the satellite via the communication unit 54, for example, as GPS. The position information acquisition unit 42 can sequentially detect the position of the moving vehicle V by acquiring the current position at predetermined time intervals.

Further, the position information acquisition unit 42 acquires the traveling direction V1 of the vehicle V as the position information. The position information acquisition unit 42 sequentially acquires the traveling direction V1 of the vehicle V based on the current position of the vehicle V at predetermined time intervals. That is, the position information acquisition unit 42 detects in which direction the vehicle V has traveled, that is, in which direction the traveling direction V1 has traveled, from the position of the vehicle V immediately before and the position of the current vehicle V. The position information acquisition unit 42 stores the position information, that is, the current position and the traveling direction V1 in the storage unit 36 in association with the time when the position information is acquired.

The external device 14 has an input unit 50, a display unit 52, a communication unit 54, a storage unit 56, and a control unit 58. The external device 14 is a server provided outside the vehicle V, that is, a computer. The input unit 50 is a mechanism capable of inputting information from the user, such as a button or a touch panel. The display unit 52 is a display that displays various information such as images. Further, the external device 14 may be configured to be able to output sound by a speaker (not shown). However, the external device 14 may have at least a communication unit 54, a storage unit 56, and a control unit 58 as a server.

The communication unit 54 is configured to transmit and receive data by communicating with other devices such as an external device 14 and a display terminal 16 under the control of the control unit 58. The storage unit 56 is a memory that stores various information such as the calculation contents of the control unit 58 and the image P. The storage unit 56 includes, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), and an external storage device such as a flash memory (Flash Memory).

The control unit 58 is an arithmetic unit, that is, a CPU (Central Processing Unit). The control unit 58 includes an image information acquisition unit 60, an image direction determination unit 62, an image analysis unit 64, a set time designation unit 66, a thumbnail acquisition unit 68, and a display image generation unit 70. The image information acquisition unit 60, the image direction determination unit 62, the image analysis unit 64, the set time designation unit 66, the thumbnail acquisition unit 68, and the display image generation unit 70 are software stored in the storage unit 56 ( By reading the program) and performing a predetermined process on the CPU, the process described later is executed.

The image information acquisition unit 60 acquires the information of the image P. In the present embodiment, the image information acquisition unit 60 acquires the image P and the position information from the vehicle-mounted device 12 as the information of the image P. The image information acquisition unit 60 stores the acquired image P and the position information in the storage unit 56. After the image P is taken by the image pickup device 10, the image information acquisition unit 60 acquires the image P and the position information and stores them in the storage unit 56. That is, the image information acquisition unit 60 collectively acquires the entire data of the image P. However, for example, when the image P is continuously captured as a moving image, the image P may be divided into a plurality of moving image data. In this case, the image information acquisition unit 60 may acquire the moving image data every time the moving image data is generated, instead of acquiring the entire data of the image P at once. Further, the image information acquisition unit 60 may acquire the audio data recorded in the image pickup apparatus 10 as the information of the image P and store it in the storage unit 56.

The image direction determination unit 62 determines (calculates) which direction the direction in the image P is in absolute coordinates based on the image P and the position information. The direction in the image P is a direction in the camera coordinate system of the image pickup apparatus 10, and means a direction in the image P such as an upward direction, a downward direction, a right direction, and a left direction. Hereinafter, a method of associating the direction in the image P with the direction in absolute coordinates will be described.

In the present embodiment, the image pickup apparatus 10 is attached to the vehicle V in a predetermined position and posture. Therefore, the posture of the image pickup apparatus 10, that is, the image pickup direction (camera angle, direction toward which the lens faces) is fixed with respect to the vehicle V. In other words, since the posture of the image pickup device 10 with respect to the vehicle V is preset, the image direction determination unit 62 acquires the set value of the posture of the image pickup device 10 with respect to the vehicle V, so that the image pickup device 10 with respect to the vehicle V Posture information can be obtained. The information on the posture of the image pickup device 10 with respect to the vehicle V can be associated with the orientation of the vehicle V and the direction in the image P. Further, the image direction determination unit 62 acquires the traveling direction V1 of the vehicle V as position information. Since the traveling direction V1 of the vehicle V is the direction in the absolute coordinates of the direction of the vehicle V (in front of the vehicle V), the direction of the vehicle V can be associated with the direction in the absolute coordinates. In this way, the information relating the direction of the vehicle V and the direction in the image P and the information relating the direction of the vehicle V to the direction in the absolute coordinates are used to obtain the direction in the image P and the direction in the absolute coordinates. Can be associated with. That is, the image direction determination unit 62 acquires the set value of the posture of the image pickup device 10 with respect to the vehicle V and the direction information in the absolute coordinates of the vehicle V, and based on the information, the posture of the image pickup device 10 in the absolute coordinates. Can be calculated. In other words, the image direction determination unit 62 associates the direction in the image P with the direction in the absolute coordinates based on the information on the posture of the image pickup device 10 with respect to the vehicle V and the information on the direction in the absolute coordinates of the vehicle V. Can be done. Further, the image direction determination unit 62 can associate the direction of the image P in the image with the direction of the vehicle V in the traveling direction V1 by associating the direction of the image P with the direction in absolute coordinates. In this way, the image direction determination unit 62 can set the direction of the absolute coordinates and the direction of the traveling direction V1 in the image P.

In the above description, the image direction determination unit 62 acquires information on the posture of the image pickup device 10 with respect to the vehicle V by acquiring preset values. However, the image direction determination unit 62 is not limited to acquiring information on the posture of the image pickup device 10 with respect to the vehicle V based on the set value. For example, even if an acceleration sensor (not shown) is provided in the image pickup device 10 and the image direction determination unit 62 acquires information on the posture of the image pickup device 10 with respect to the vehicle V based on the detection value of the acceleration sensor, that is, the acceleration of the image pickup device 10. good. Since the image pickup device 10 generates acceleration along the traveling direction V1, the image direction determining unit 62 associates the posture of the image pickup device 10 with the traveling direction V1 from the acceleration of the image pickup device 10 to take an image of the vehicle V. Information on the posture of the device 10 can be acquired. Further, for example, the image pickup device 10 is provided with a gyro sensor capable of detecting the posture at the absolute coordinates of the image pickup device 10, and the image direction determination unit 62 uses the detection value of the gyro sensor to obtain information on the posture of the image pickup device 10 with respect to the vehicle V. May be obtained as. By detecting the orientation of the image pickup device 10 and the like using the sensor in this way, the image orientation determination unit 62 information on the posture of the image pickup device 10 with respect to the vehicle V even when the mounting position of the image pickup device 10 is not fixed. Can be obtained, and the posture of the image pickup apparatus 10 in absolute coordinates can be calculated. As described above, the image direction determination unit 62 may associate the direction of the image P in the image with the direction in absolute coordinates and the traveling direction V1 by using an arbitrary method.

The image analysis unit 64 extracts the feature image from the plurality of images P by analyzing the information of the image P acquired by the image information acquisition unit 60, and sets the time at which the feature image is captured as the feature time (feature). Detect as a tag). In other words, it can be said that the feature image is the image P captured at the feature time. The image analysis unit 64 determines that, among the plurality of images P (moving images captured by the image pickup apparatus 10 for each frame in the present embodiment), the image P satisfying a predetermined predetermined condition is a feature image. .. The feature image is an image obtained by capturing a scene that is presumed to be highly likely to be viewed by a user, and the image analysis unit 64 extracts such a feature image based on a predetermined condition.

The image analysis unit 64 detects the feature time by determining whether the image P satisfies a predetermined condition, that is, by determining whether the parameter extracted from the image P satisfies the predetermined condition. For example, the image analysis unit 64 sets the image P as a feature image when the facial expression of the person shown in the image P is a parameter and the facial expression satisfies a predetermined feature amount such as a smile. Judgment is made, and the time when the image P is captured is detected as the feature time. Further, the image analysis unit 64 may extract a preset image P of a person as a feature image based on a feature amount such as a face, with the person shown in the image P as a parameter. Further, the image analysis unit 64 determines whether the sound recorded in the image pickup apparatus 10 (for example, the sound in the vehicle) satisfies a predetermined condition, that is, whether the parameter extracted from the sound satisfies the predetermined condition. The feature time may be detected by determining. For example, the image analysis unit 64 captures an image P at the time when the voice is recorded when the voice parameter, for example, the magnitude or frequency is out of a predetermined range such as being equal to or more than a predetermined threshold. Is determined to be a feature image, and the time when the image P is captured (that is, the time when the sound is recorded) is detected as the feature time. Further, for example, when the voice contains a predetermined keyword, the image analysis unit 64 may determine the image P captured at the time when the voice is recorded as a feature image. Further, the image analysis unit 64 detects the feature time by determining whether the position information satisfies a predetermined condition, that is, by determining whether the parameter extracted from the position information satisfies the predetermined condition. May be good. For example, the image analysis unit 64 detects whether the vehicle V is located at a predetermined location using the position of the vehicle V as a parameter, and features an image P captured at a time when the vehicle V is located (reached) at the predetermined location. The time when the image P is captured (that is, the time when the image P reaches a predetermined place) is detected as the feature time. The predetermined place here may be set by the user or may be set by the control unit 58. The predetermined place can be arbitrarily set, for example, an intermediate point or a branch point of the movement route of the vehicle V, a tourist attraction, or the like.

The image analysis unit 64 may use any one of the above-mentioned feature image determination methods, or may combine a plurality of them. That is, the image analysis unit 64 may determine whether the image is a feature image using at least one of the image P, the voice, and the position information, or the feature image using all of the image P, the voice, and the position information. You may make a judgment as to whether or not it is. Further, when the image analysis unit 64 has a plurality of parameters for determining the feature image, the image analysis unit 64 may weight the determination of the feature image by scoring. For example, the image analysis unit 64 may count the number of parameters satisfying a predetermined condition, and may determine the image P at the time when the number becomes a predetermined value or more as a feature image. For example, when the predetermined number is 2, when the facial expression of the person is a smile and there is a time when the voice exceeds the predetermined threshold value, there are two parameters satisfying the predetermined condition, so that the image P at that time is displayed. It is determined that the image is a feature image. Further, a predetermined condition, that is, a plurality of threshold values may be set stepwise for one parameter, and points may be set for each threshold value. In this case, the image analysis unit 64 determines, for example, the image P at the time when the total value of the points becomes a predetermined value or more as a feature image. Further, when a plurality of feature images are extracted, the feature images may be ranked by the number of parameters satisfying a predetermined condition or by the total value of points, that is, by the score. That is, the image P is an image that is presumed to be more likely to be viewed by the user as the score is higher. The score is, for example, the number of parameters satisfying a predetermined condition, the total value of the points, and the like.

FIG. 4 is a diagram for explaining the set time. The set time designation unit 66 designates a plurality of different set times from within the time when the image P is captured, based on the information of the image P. The thumbnail acquisition unit 68, which will be described later, uses the image P captured at the set time as the thumbnail 100. An example of setting the set time will be described with reference to FIG. The horizontal axis of FIG. 4 is time, and in the example of FIG. 4, the time from time ts to time te is the time when the image P is captured (imaging time). That is, the imaging of the image P is started at the time ts, and the imaging of the image P is completed at the time te.

As shown in FIG. 4A, the set time designation unit 66 sets a plurality of temporary times (temporary tags) for each predetermined time between the time ts and the time te. This predetermined time, that is, the time between the provisional times, has a predetermined fixed length, for example, 5 minutes or more and 10 minutes or less. That is, the set time designation unit 66 sets a temporary time at regular time intervals during the imaging time. In the example of FIG. 4, the provisional times are t1, t2, t3, t4, and t5, and the time from t1 to t2, the time from t2 to t3, the time from t3 to t4, and the time from t4 to t5. Times are the same length as each other.

Further, the set time designation unit 66 acquires the feature time detected by the image analysis unit 64, that is, the time when the feature image is captured. In the example of FIG. 4, as shown in FIG. 4B, the time ta and the time tb are detected as feature times within the time when the image P is captured.

The set time designation unit 66 designates the set time based on this feature time, that is, the time when the feature image is captured. More specifically, in the present embodiment, the set time designation unit 66 designates the set time based on the provisional time and the feature time. As shown in FIG. 4 (c), the set time designation unit 66 sets the provisional time and the feature time as the set time. Further, the set time designation unit 66 replaces the tentative time closest to the feature time among the plurality of tentative times with the feature time, and sets the replaced feature time as the set time. The set time designation unit 66 sets a tentative time that is not replaced with the feature time, that is, a tentative time that is not the tentative time closest to the feature time as the set time. That is, in the example of FIG. 4, the tentative time closest to the time ta, which is the feature time, is time t2, and the tentative time closest to the time tb, which is another feature time, is time t4. Therefore, the times t2 and t4 are replaced with the feature times ta and tb, and the times ta and tb are set to the set time. Further, the other temporary times t1, t3, and t5 are set to the set time. That is, in the example of FIG. 4, the times t1, ta, t3, tb, and t5 are designated as the set times.

If the tentative time is used as it is, the time between the set times will be constant. However, when the set time is specified based on the feature time in this way, the length between the set times, that is, between the set times continuous in the time series is different for each set time. That is, what is the time between time t1 and time ta, the time between time ta and time t3, the time between time t3 and time tb, and the time between time tb and time t5? , Different from each other. However, not all the times between the set times are different from each other, and at least one of the times between the set times may be different from the others. That is, the time between the set time set based on the feature time and the set time that is continuous in time series with the set time is at least one of the other set times, and the time between the other set times. Different from.

In the above description, the set time designation unit 66 sets the feature time as the set time, but if the set time is set based on the feature time, the feature time cannot be set as it is. May be good. That is, the set time designation unit 66 may set a time deviated from the feature time by a predetermined time as the set time. For example, the set time designation unit 66 may set a time earlier than the feature time by a predetermined time as the set time. By setting the set time in this way, the image reproduced from the set time can be an image including before and after the feature image.

As described above, the set time designation unit 66 according to the present embodiment sets the set time based on the provisional time and the feature time, but decides to set the set time based on the provisional time and the feature time. Not limited. The set time designation unit 66 may set the set time based on the feature time without using the provisional time. Further, the set time designation unit 66 may set the set time based on the provisional time without using the feature time. That is, the set time designation unit 66 may set the set time so that the length between the set times is constant. Further, the set time designation unit 66 may set the set time so that the lengths between the set times are different from each other even when the feature time is not used. That is, the set time designation unit 66 may set a plurality of set times between the imaging time of the image P, that is, the time ts and the time te. Therefore, the set time designation unit 66 can set the set time if there is information on the length of the imaging time of the image P (that is, the time from the time ts to the time te). Therefore, it can be said that the set time designation unit 66 sets the set time based on the information of the image P (here, the information of the length of the imaging time of the image P).

Returning to FIG. 3, the thumbnail acquisition unit 68 acquires the image P captured at the set time as the thumbnail 100. The thumbnail refers to the reduced data of the image P, and in the present embodiment, refers to the image P captured by the image pickup apparatus 10 at a set time. In the present embodiment, the thumbnail acquisition unit 68 extracts the image P captured at the set time as the thumbnail 100 from the image P acquired by the image information acquisition unit 60 and stored in the storage unit 56. Since the thumbnail acquisition unit 68 acquires the thumbnail 100 at each set time, the thumbnail 100 is acquired for the number of set times. The thumbnail acquisition unit 68 may acquire the data as the thumbnail 100 after performing processing such as reducing the data of the image P captured at the set time, or the image P captured at the set time. The data may be acquired as the thumbnail 100 as it is without any processing. Further, the thumbnail 100 in the present embodiment is a still image at a set time, but may be, for example, a moving image captured between the set time and a predetermined time.

The display image generation unit 70 generates an image including the thumbnail 100 acquired by the thumbnail acquisition unit 68 as the display image P1. In the present embodiment, the display image generation unit 70 transmits the generated display image P1 to the display terminal 16 and causes the display unit 82 of the display terminal 16 to be described later to display the generated display image P1. However, the display image generation unit 70 may display the generated display image P1 on another display unit of the display control system 1 (for example, the display unit 32 of the in-vehicle device 12 or the display unit 52 of the external device 14). The display image P1 will be described later.

As described above, in the present embodiment, the external device 14 acquires the information of the image P by the image information acquisition unit 60, designates the set time by the set time designation unit 66, and acquires the thumbnail 100 by the thumbnail acquisition unit 68. Then, the display image generation unit 70 generates the display image P1. Therefore, in the present embodiment, the external device 14 constitutes the display control device.

As shown in FIG. 3, the display terminal 16 includes an input unit 80, a display unit 82, a communication unit 84, a storage unit 86, and a control unit 88. The display terminal 16 is a device for displaying the display image P1. For example, in the present embodiment, the display terminal 16 is a mobile terminal such as a tablet or a smartphone carried by a user. Therefore, the display terminal 16 may be brought into the vehicle V by the user. However, the display terminal 16 may be any device as long as it is a device that displays the display image P1. The in-vehicle device 12 and the display terminal 16 may be different mobile terminals or may be the same mobile terminal.

The input unit 80 is a mechanism capable of inputting information from the user, such as a button or a touch panel. The display unit 82 is a display such as a touch panel that displays various information such as images. Further, the display terminal 16 may be configured to be able to output sound by a speaker (not shown). In this case, the display terminal 16 may reproduce the sound recorded by the image pickup device 10 on the speaker while displaying the image P captured by the image pickup device 10 on the display unit 82.

The communication unit 84 is configured to transmit / receive data by communicating with another device such as an external device 14 under the control of the control unit 88. The storage unit 86 is a memory that stores various information such as the calculation contents of the control unit 88 and the image P. The storage unit 86 includes at least one external storage device such as a RAM (Random Access Memory), a ROM (Read Only Memory), and a flash memory (Flash Memory).

The control unit 88 is an arithmetic unit, that is, a CPU (Central Processing Unit). The control unit 88 includes a display image acquisition unit 90 and a display control unit 92. The display image acquisition unit 90 and the display control unit 92 read the software (program) stored in the storage unit 86 and perform a predetermined process on the CPU to execute the process described later.

The display image acquisition unit 90 acquires an image to be displayed on the display unit 82. The display image acquisition unit 90 acquires the display image P1 generated by the display image generation unit 70 from the external device 14 as an image to be displayed on the display unit 82. Further, the display image acquisition unit 90 acquires the image P captured by the image pickup device 10 from the external device 14 as an image to be displayed on the display unit 82. The display control unit 92 causes the display unit 82 to display the image acquired by the display image acquisition unit 90.

(About the displayed image)
Hereinafter, the display image P1 generated by the display image generation unit 70 of the external device 14 will be described. FIG. 5 is a diagram showing an example of a display image. FIG. 5 shows an example in which the display image P1 is displayed on the display screen of the display unit 82. Further, FIG. 5 describes an example in which the display image P1 is displayed in a thumbnail mode, that is, a mode in which the display area Q described later is displayed while rotating. As shown in FIG. 5, the display image generation unit 70 generates the display image P1 so that the thumbnails 100 acquired by the thumbnail acquisition unit 68 are displayed side by side in the order of the captured time series. The display terminal 16 acquires the display image P1 generated by the display image generation unit 70 by the display image acquisition unit 90, and causes the display control unit 92 to display the acquired display image P1 on the display screen of the display unit 82.

Here, the thumbnail 100 is generated for each time series, that is, for each set time. The display image generation unit 70 generates an image in which thumbnails 100 are arranged in order for each captured time series (for each time series at a set time) as a display image P1. In the example of FIG. 5, thumbnails 100A, 100B, 100C, 100D, and 100E are displayed as thumbnails 100. Thumbnails 100A, 100B, 100C, 100D, and 100E are images taken in this order in chronological order. That is, the thumbnail 100A is an image captured at time t1 shown in FIG. 4, the thumbnail 100B is an image captured at time ta shown in FIG. 4, and the thumbnail 100C is captured at time t3 shown in FIG. The thumbnail 100D is an image captured at time tb shown in FIG. 4, and the thumbnail 100E is an image captured at time t5 shown in FIG. Therefore, in FIG. 5, thumbnails 100A, 100B, 100C, 100D, and 100E are arranged in this order.

The display image P1 will be described in more detail. Hereinafter, the display image P1 will be described based on the direction X, the direction Y, and the direction Z. The direction X is a direction along the display screen of the display image P1, and the direction Y is a direction along the display screen of the display image P1 and orthogonal to the direction X. Further, the direction Z is a direction orthogonal to the direction X and the direction Y, and is a depth direction of the display screen. That is, the directions X, Y, and Z are the directions at the coordinates set on the display screen. The display image generation unit 70 reproduces a virtual three-dimensional space having directions X, directions Y, and directions Z as three axes on the display screen, and reproduces the display image P1 in the virtual three-dimensional space. The display image generation unit 70 arranges a plurality of spherical display areas Q in the virtual three-dimensional space, that is, in the display image P1. Then, the display image generation unit 70 maps the thumbnail 100 to the outer surface of the spherical display area Q, so that the thumbnail 100 is displayed on the outer surface of the display area Q. In the present embodiment, since the image P, that is, the thumbnail 100 is a 360-degree image, the thumbnail 100 is mapped over the entire circumference of the display area Q in the circumferential direction. However, in the present embodiment, since the display screen is a two-dimensional surface along the X direction and the Y direction, the display image P1 actually displayed in the display area Q is the direction of the display area Q in the entire thumbnail 100. The image is mapped to the surface on the opposite side of Z (the front side of the display screen). Although the display area Q is spherical, it may be a sphere in which the distance from the center to the outer peripheral surface is constant, or an ellipsoid in which the distance from the center to the outer peripheral surface is not constant. That is, it can be said that the spherical shape here is a shape in which the outer peripheral surface is a curved surface and is continuous (not discontinuous).

Then, the display image generation unit 70 generates the display image P1 so as to rotate each display area Q in the thumbnail mode shown in FIG. The thumbnail 100 is displayed while changing the angle as the display area Q rotates, and the displayed image (the displayed image area of the entire thumbnail 100) changes. In other words, it can be said that the display image generation unit 70 displays the thumbnail 100 while virtually changing the camera angle (imaging direction) of the image pickup device 10 by rotating the display area Q. Furthermore, the thumbnail 100 displays an image of a part of the whole, and the displayed portion changes as the display area Q rotates. For example, the thumbnail 100 displays an image from 0 to 180 degrees out of 360 degree images at a certain timing, and an image from 1 degree to 181 degrees at the next timing. Images with different angles are displayed as the region Q rotates. As described above, the image P, that is, the thumbnail 100 is an image for one round along the circumferential direction with respect to the predetermined axis, and the rotation direction of the display area Q is, for example, the circumferential direction with respect to the predetermined axis. It may be in the direction along. However, since the display area Q is spherical, the thumbnail 100 can be appropriately displayed regardless of the direction of rotation. Therefore, it is preferable that the display image generation unit 70 randomly rotates the display area Q in an arbitrary direction. That is, the display image generation unit 70 may change the rotation direction of the display area Q. Further, the display image generation unit 70 may have the same rotation direction for each display area Q, or may have different rotation directions for each display area Q. Further, the display image generation unit 70 may change the rotation speed of the display area Q. Further, the display image generation unit 70 may have the same rotation speed for each display area Q, or may have different rotation speeds for each display area Q.

Further, in the thumbnail mode shown in FIG. 5, the display image generation unit 70 moves each display area Q in at least one of the direction X, the direction Y, and the direction Z in a predetermined locus. As a result, the display area Q, that is, the thumbnail 100, is visually recognized as floating on the display screen. It is preferable that the locus for moving the display area Q is randomly set. Further, the display image generation unit 70 may have the same locus for each display area Q, or may have a different locus for each display area Q. The locus may be different for each display area Q. It is preferable that the display area Q has a locus set so as to move in all directions of the direction X, the direction Y, and the direction Z. By setting the locus in this way, it becomes easier to visually recognize the locus as if it were floating. The locus may be any direction, not limited to the direction along each axis of the direction X, the direction Y, and the direction Z, and may be an arbitrary curve, not limited to a straight line. The display area Q may be moved at a constant speed along a predetermined locus, or may be moved while changing the speed.

In this way, the display image generation unit 70 may have at least one of the rotation direction, the rotation speed, and the movement locus different for each display area Q. In this case, it can be said that the display image generation unit 70 has a different display method for each display area Q. Further, the display image generation unit 70 may display the same for each display area Q.

Further, the display image generation unit 70 arranges the display area Q, that is, the thumbnail 100 in chronological order for each time (set time) when the thumbnail 100 displayed in the display area Q is captured (set time). Is being generated. More specifically, the display image generation unit 70 generates the display image P1 so that the display area Q (thumbnail 100) is arranged in the order of time series along the direction Z, that is, the depth direction of the screen. That is, the display image generation unit 70 arranges the display area Q (thumbnail 100) on the direction Z side (back side of the screen) as the time taken by the image is delayed. In the example of FIG. 5, the thumbnail 100A is displayed on the outer surface of the display area QA, the thumbnail 100B is displayed on the outer surface of the display area QB, the thumbnail 100C is displayed on the outer surface of the display area QC, and the outside of the display area QD. The thumbnail 100D is displayed on the surface, and the thumbnail 100E is displayed on the outer surface of the display area QE. In this case, as shown in FIG. 5, the display area QA, the display area QB, the display area QC, the display area QD, and the display area QE are directed to the direction Z, that is, toward the back side of the screen, in this order. They are lined up. Since the display screen is two-dimensional, it can be said that the display image generation unit 70 reduces the size of the sphere in the order of the display areas QA, QB, QC, QD, and QE, and the time series is Later, it can be said that the size of the display area Q, that is, the size of the thumbnail 100 is reduced. The direction in which the display areas Q are arranged for each time series is not limited to the direction Z, and may be any direction. That is, the display image generation unit 70 may generate the display image P1 so that the display area Q is displayed in a predetermined direction in the order of the time series in which the thumbnail 100 is captured.

Further, the display image generation unit 70 arranges the thumbnail 100, that is, the display area Q, in a direction different from the arrangement direction in the time series (here, the direction Z) based on the position where the image P as the thumbnail 100 is captured. .. Here, the first thumbnail (here, thumbnail 100A) captured at the first set time (for example, time t1 in FIG. 4) and the second set time (here, time ta) after the first set time are captured. The explanation will be given based on the second thumbnail (thumbnail 100B in this case). The display image generation unit 70 acquires information on the traveling direction V1 of the vehicle V as position information. The display image generation unit includes the first thumbnail so that the second thumbnail (thumbnail 100B) is located on the V1 side in the traveling direction from the first set time to the second set time with respect to the first thumbnail (thumbnail 100A). Line up with the second thumbnail. In the example of FIG. 5, since the traveling direction V1 from the first set time to the second set time is the Y direction (Y0 direction in FIG. 2), the second thumbnail (thumbnail 100B) is the first thumbnail (thumbnail). It is located on the Y direction side with respect to 100A). More specifically, the display image generation unit 70 associates the direction in absolute coordinates with the direction on the screen. That is, the display image generation unit 70 associates the direction X0 in the absolute coordinates shown in FIG. 2 with the direction X on the display screen, and associates the direction Y0 shown in FIG. 2 with the direction Y on the screen. This makes it possible to reproduce the traveling direction V1 on the display screen. The display image generation unit 70 sets the direction of the traveling direction V1 for each set time in the display image P1, and arranges the thumbnails 100, that is, the display area Q along the set traveling direction V1.

Further, the display image generation unit 70 generates the display image P1 so that the background image 110 is displayed in an area other than the thumbnail 100. That is, the display image P1 is an image in which the thumbnail 100 is displayed on the background image 110. The background image 110 may be an arbitrary image, but may be, for example, an image of a map of an area in which the vehicle A has traveled. In this case, the display image generation unit 70 acquires the position information of the vehicle V at the time when the thumbnail 100 is captured, and sets the position of the vehicle V at the time when the thumbnail 100 is captured on the map of the background image 110. It is preferable to display the thumbnail 100 (display area Q). In this way, by displaying the thumbnail 100 at the position where the thumbnail 100 is captured on the map, it is possible to appropriately recognize the position of the image captured. Further, for example, the background image 110 may be an image linked to the time when the thumbnail 100 is captured. In this case, for example, when the time when the thumbnail 100 is captured is night, the background image 110 becomes a night image, and when the time when the thumbnail 100 is captured is daytime, the background image 110 becomes a daytime image.

Further, for example, the display image generation unit 70 may change the number of thumbnails 100 to be displayed. In this case, for example, the display image generation unit 70 changes the number of thumbnails 100 by performing an operation (setting) for changing the time interval between the set times, which is the time when the thumbnails 100 are captured. That is, when the operation is performed so as to reduce the time interval between the set times, the display image generation unit 70 acquires the information to that effect, increases the number of thumbnails 100, and sets the time between the thumbnails 100. The display image P1 is updated so that the interval is reduced. Further, when the operation is performed so as to increase the time interval between the set times, the display image generation unit 70 reduces the number of thumbnails 100 and increases the time interval between the set times of the thumbnails 100 so that the display image P1 To update.

Further, in the example of FIG. 5, all the created thumbnails 100 are displayed on one screen, but a part of the thumbnails 100 is displayed on one screen, for example, an operation by the user (swipe operation of the screen, etc.). The display image P1 may be switched so that the thumbnail 100 that has not been displayed is displayed.

FIG. 6 is a diagram showing an example of an image in the display image mode. The display control unit 92 of the display terminal 16 displays the display image P1 on the display screen (display unit 82), and when the thumbnail 100 is selected by the user, the image to be displayed on the display screen is set to the thumbnail mode, that is, The display image P1 is switched to the image reproduction mode, that is, the image P shown in FIG. In other words, when the thumbnail 100 is selected by the user, the display terminal 16 acquires the image P from the external device 14 by the display image acquisition unit 90, and the display control unit 92 displays the acquired display image P1 in the display unit 82. Display on the display screen of. In this case, the display terminal 16 may sequentially acquire the image P from the external device 14 and perform streaming reproduction. Further, the display terminal 16 may acquire the entire data of the image P from the external device 14 and store it in the storage unit 86, and read out and reproduce the image P stored in the storage unit 86. The display control unit 92 of the display terminal 16 may display an image P0 indicating the imaging direction (camera angle, lens facing direction) of the display image P1 on the display screen of the display unit 82. FIG. 6 shows an example in which the image P images the traveling direction of the vehicle V, and here, the road on the traveling direction side of the vehicle V is imaged. Therefore, in the example of FIG. 6, the image P0 shows that the image P1 captures the traveling direction of the vehicle V. For example, the image P0 is a superposition of an image P0a, which is an icon of the vehicle V, and an image P0b, which is an image showing the imaging direction of the image P1. It can be said that the image P0a is a schematic image of the vehicle V viewed from above in the vertical direction. The image P0b is, for example, a range to be imaged (in the example of FIG. 6) in a region extending 360 degrees (a circular region of a broken line in the example of FIG. 6) centered on an axis along the vertical direction on the image P0a. It is an image showing the area of the shaded line). It can be said that the orientation of the image indicating the range to be imaged in the image P0b indicates the imaging direction. When the image pickup direction of the image P1 is changed by a swipe operation or the like, the display control unit 92 also changes the image pickup direction indicated by the image P0, for example, by changing the direction of the image P0b, that is, the direction of the image indicating the image pickup range. , It is desirable to change it together.

More specifically, when the thumbnail 100 is selected in the display screen, the display control unit 92 reproduces the image P from the time when the thumbnail 100 is captured on the display screen. That is, the display control unit 92 continuously reproduces the image P from the imaging time of the selected thumbnail 100. As a result, the user can cue and reproduce the image P from the thumbnail 100. The display image generation unit 70 displays the display image so that when the thumbnail 100 is selected by the user, the display area Q to which the thumbnail 100 is mapped pops and cracks, and then the display image is switched to the reproduction of the image P. It is preferable to generate P1. However, the method of switching screens is not limited to this, and is arbitrary. For example, the display control unit 92 displays such that the thumbnail 100 selected by the user is enlarged or expanded to expand the entire screen, or the thumbnail 100 selected by the user is on the opposite side of the Z direction (front side of the screen). The display image P1 may be generated so as to switch to the reproduction of the image P after the display or the like is performed so as to jump toward the image P. The user may select the thumbnail 100 by touching the place where the thumbnail 100 (display area Q) is displayed, or operate the button or the like to select the thumbnail 100. It is also good. That is, the method of selecting the thumbnail 100 is arbitrary.

FIG. 7 is a diagram showing another example of the displayed image. Here, the thumbnail 100 extracted as the feature image, that is, the display area Q to which the thumbnail 100 at the set time set based on the feature time is mapped is referred to as the feature display area. Further, the normal display area is the display area Q to which the thumbnail 100 not extracted as the feature image, that is, the thumbnail 100 at the set time not set based on the feature time (here, set based on the provisional time) is mapped. It is described as. In this case, as shown in FIG. 7, the display image generation unit 70 is mapped to the feature display area by making the display method of the feature display area different from the display method of the normal display area in the thumbnail mode. The thumbnail 100, that is, the feature image may be highlighted. In the example of FIG. 7, the display areas QB and QD are feature display areas, and the display areas QA, QC and QE are normal display areas. In the example of FIG. 7, the display image generation unit 70 displays the display areas QB and QD on the side opposite to the Z direction (front side of the screen) from the display areas QA, QC and QE, thereby displaying the display areas QB and QD. , The display area is displayed larger than QA, QC, and QE. That is, in this case, the display areas QB and QD, which are the feature display areas, are displayed on the front side of the screen regardless of the order in the time series. That is, in the example of FIG. 7, the feature display area is displayed on the front side of the screen and enlarged, so that the display method of the feature display area is different from the display method of the normal display area. However, the method of making the display different is not limited to this. For example, the display image generation unit 70 may make the movement locus of the feature display area different from the movement locus of the normal display area, make the movement locus of the feature display area longer than the movement locus of the normal display area, or make the movement locus of the feature display area longer. The display method may be different by increasing the moving speed or frequency to be higher than the moving speed or frequency of the normal display area. Further, for example, the feature display area may be highlighted by adding a note to the feature display area with characters, pictures, or the like so that the display method is different.

Further, as described above, the feature images may be ranked by the total value of the number of parameters and the points satisfying a predetermined condition, that is, by the score. When the feature images are ranked in this way, the display image generation unit 70 may display the feature display areas (display areas QB, QD) in different ways for each rank of the feature images. For example, when the ranking of the thumbnail 100D of the display area QD is higher than the ranking of the thumbnail 100B of the display area QB, that is, when the score of the thumbnail 100D is higher than the score of the thumbnail 100B, the display image generation unit 70 may use the display area QD. Is arranged on the front side of the screen with respect to the display area QB, and the display area QD is made larger than the display area QB. Further, for example, the locus of movement of the display area QD may be larger than the locus of movement of the display area QB. By changing the display of the display area Q for each order of the feature images in this way, it becomes easier for the user to detect the feature image, which is a scene in which the user is likely to request viewing.

The display image P1 that emphasizes and displays the feature display area as shown in FIG. 7 is displayed in place of the display image P1 shown in FIG. 5, but is different from the display image P1 as shown in FIG. , The display image P1 as shown in FIG. 7 may be switched and displayed. Further, the display image P1 that emphasizes and displays the feature display area as shown in FIG. 7 may be used for the fixed thumbnail mode of FIG. 8 described later.

FIG. 8 is a diagram showing an example of a display image. As shown in FIG. 5, the display image generation unit 70 generates the display image P1 in the thumbnail mode in which the display area Q is rotated. However, the display image generation unit 70 may also generate the display image P1 in a display mode other than the thumbnail mode. FIG. 8 shows the display image P1 in the fixed thumbnail mode. As shown in FIG. 8, the display image P1 in the fixed thumbnail mode is different from the display image P1 in the thumbnail mode in that the display area Q is not rotated and the rotation of the display area Q is stopped. In this case, the thumbnail 100 displayed in the display area Q is not displayed so as to change the angle, so that the thumbnail 100 is displayed as an image in which the angle is fixed in a predetermined direction. In other words, the display image generation unit 70 generates the display image P1 so that the angle of the thumbnail 100 is in a fixed predetermined direction in the fixed thumbnail mode.

More specifically, it is preferable that the display image generation unit 70 generates the display image P1 so that the angles of all the displayed thumbnails 100 are in the same predetermined direction in the fixed thumbnail mode. That is, the display image generation unit 70 arranges the angles of the thumbnails 100 in a fixed direction, that is, in the same predetermined direction. By arranging the angles of the thumbnail 100 in the same predetermined direction in this way, the thumbnail 100 in the fixed thumbnail mode is displayed as an image captured in the same direction, that is, in the same predetermined direction. The direction of the angle of the thumbnail 100, that is, a predetermined direction can be arbitrarily set, but for example, the traveling direction V1 of the vehicle V is preferable. By setting the angles of all the thumbnails 100 as the traveling direction V1 of the vehicle V, all the thumbnails 100 become images displaying the traveling direction V1 side. Therefore, the display image P1 is an image in which thumbnails 100 captured on the traveling direction V1 side are arranged in chronological order, and the user who visually recognizes the thumbnails can relive the movement by the vehicle V. Further, the direction of the angle may be opposite to the traveling direction V1. In such a case, it can be said that the direction of the angle is set based on the traveling direction V1, so that all the thumbnails 100 are images displaying the direction set based on the traveling direction V1. Further, the direction of the angle may be a predetermined direction in absolute coordinates (for example, one of the north, south, east, and west directions). By setting the angles of all the thumbnails 100 to the predetermined directions in the absolute coordinates in this way, all the thumbnails 100 become images displaying the predetermined directions in the absolute coordinates. Further, in the fixed thumbnail mode, the display image generation unit 70 may stop the movement of the display area Q in the directions X, Y, and Z, that is, float, or keep it floating. good. In the fixed thumbnail mode, the image P0 may be displayed together with the thumbnail 100. In this case, the image P0 is an image indicating the direction of the angle of the thumbnail 100, and for example, the direction of the image P0b is the direction of the angle of the thumbnail 100.

Switching between the thumbnail mode and the fixed thumbnail mode is performed, for example, by a user operation (setting). When the display image P1 in the thumbnail mode is displayed on the display unit 82 and the switching setting is made, the display image generation unit 70 generates the display image P1 in the fixed thumbnail mode and displays the display image P1 on the display terminal 16. Send. The display terminal 16 acquires the display image P1 in the fixed thumbnail mode by the display image acquisition unit 90, and switches the display image P1 in the thumbnail mode to the display image P1 in the fixed thumbnail mode by the display control unit 92. .. Switching from the fixed thumbnail mode to the thumbnail mode is performed in the same manner.

FIG. 9 is a diagram showing another example of the displayed image. As shown in FIG. 9, the display image generation unit 70 may generate the display image P1 in the sort mode, which is a display mode other than the thumbnail mode. As shown in FIG. 9, in the sort mode, the thumbnails 100 are arranged in chronological order in one direction (here, the X direction). Then, the thumbnails 100 are arranged in the order of the scores calculated at the time of extracting the feature image in a direction different from one direction (here, the Y direction). That is, the higher the score, the more the thumbnail 100 is located on the side different from one direction (here, the Y direction). That is, in the sort mode, the thumbnail 100 is sorted according to the score, that is, the degree to which it is estimated that the user is likely to request viewing. In FIG. 9, a time arrow along the X direction and a score arrow along the Y direction are displayed, but these arrows may not actually be displayed. Further, in the sort mode, sorting may be performed based on parameters other than the score. In the sort mode, the display area Q is not rotated, but the display area Q may be rotated. Further, even in the sort mode, when the display area Q is not rotated, the image P0 may be displayed together with the thumbnail 100. That is, the display image generation unit 70 may display the image P0 in the display mode in which the display area Q does not rotate, and may not display the image P0 in the display mode in which the display area Q rotates. ..

Switching between the thumbnail mode and the sort mode is performed, for example, by a user operation (setting). When the display image P1 in the thumbnail mode is displayed on the display unit 82 and the switching setting is made, the display image generation unit 70 generates the display image P1 in the sort mode and transmits it to the display terminal 16. do. The display terminal 16 acquires the display image P1 in the sort mode by the display image acquisition unit 90, and switches the display image P1 in the thumbnail mode to the display image P1 in the sort mode by the display control unit 92. Switching from the thumbnail mode to the sort mode is performed in the same manner. Further, the fixed thumbnail mode and the sort mode can be switched in the same manner. However, the display image generation unit 70 is not limited to generating the display image P1 in all modes of the thumbnail mode, the fixed thumbnail mode, and the sort mode, and may use at least one of the thumbnail mode, the fixed thumbnail mode, and the sort mode. Just generate it.

The display method of the display image P1 described above will be described with reference to the flowchart. FIG. 10 is a flowchart illustrating a display method of the displayed image. When displaying the display image P1, first, the image pickup device 10 captures the image P while moving, and the vehicle-mounted device 12 acquires the position information for each movement. As shown in FIG. 10, the display control system 1 acquires the information of the image P by the image information acquisition unit 60 of the external device 14 (step S10). In the present embodiment, the image information acquisition unit 60 acquires the image P (image data) and the position information as the information of the image P from the in-vehicle device 12. After acquiring the information of the image P, the display control system 1 associates the direction in the image P with the direction in absolute coordinates by the image direction determination unit 62 of the external device 14 (step S12). That is, the image direction determination unit 62 calculates which direction the direction in the image P points to in absolute coordinates.

After that, the display control system 1 analyzes the information of the image P by the image analysis unit 64 of the external device 14 and detects the feature time (step S14). That is, the image analysis unit 64 extracts a feature image that is presumed to be highly likely to be viewed by the user from the image P, and detects the time when the feature image is captured as the feature time. After detecting the feature time, the display control system 1 designates the set time by the set time designation unit 66 of the external device 14 (step S16). In the present embodiment, the set time designation unit 66 designates the provisional time and the feature time as the set time. After setting the set time, the display control system 1 acquires the image P captured at the set time as the thumbnail 100 by the thumbnail acquisition unit 68 of the external device 14 (step S18), and the display image generation unit 70 obtains the image P. An image in which thumbnails 100 are arranged in time series is generated as a display image P1. After the display image P1 is generated, the display control system 1 causes the display unit 82 of the display terminal 16 to display the display image P1 (step S20).

As described above, the display control device (here, the external device 14) according to the present embodiment includes an image information acquisition unit 60, a set time designation unit 66, a thumbnail acquisition unit 68, and a display image generation unit 70. Have. The image information acquisition unit 60 acquires information on a plurality of images P captured while moving. The set time designation unit 66 designates a plurality of set times different from each other from the time when the image P is captured, based on the information of the image P. The thumbnail acquisition unit 68 acquires the image P captured at the set time as the thumbnail 100. In the display image generation unit 70, the plurality of acquired thumbnails 100 are arranged in the order of the time series in which they are imaged in a predetermined direction (Z direction in this embodiment), and the image P is imaged in a direction different from the predetermined direction. The display image P1 to be displayed on the display unit (display unit 82 in the present embodiment) is generated so that the displayed positions are displayed side by side based on the position information indicating the position.

Since the image P is captured while moving, the amount of data is large. Therefore, it may be difficult for the user to detect an image of a scene that he / she wants to see from the image P. On the other hand, in the display control device according to the present embodiment, the images captured at each set time are used as thumbnails 100, and the thumbnails 100 are arranged in chronological order and are arranged based on the captured positions. Generate P1. Therefore, according to this display control device, the user can appropriately identify each of the plurality of thumbnails 100 by visually recognizing the thumbnails 100 arranged in chronological order and for each of the captured positions, and the images are captured. The desired scene can be appropriately detected from the image P. Further, the display image P1 displays the scenes for each movement in order by arranging the thumbnails 100 in chronological order. Therefore, the user can relive the movement by visually recognizing the thumbnail 100, and can look back on the trip in the vehicle V and have a comfortable experience. Further, the display image P1 arranges thumbnails 100 based on the position. That is, the image information acquisition unit 60 acquires the position information (here, the position information of the vehicle V) in which the image P is captured, and the display image generation unit 70 acquires the position information to obtain a plurality of thumbnails 100. The images P are arranged based on the positions where the images P are captured in directions different from the predetermined directions (X direction and Y direction in this embodiment). Therefore, by visually recognizing the thumbnail 100, the user can relive the movement route, so that the image of the scene to be seen can be detected more appropriately.

Further, the position information includes information on the traveling direction V1 when capturing the image P, and the thumbnail 100 includes a first thumbnail of the first set time and a second set time which is a set time after the first set time. Has a second thumbnail of. The display image generation unit 70 arranges the first thumbnail and the second thumbnail so that the second thumbnail is located on the V1 side in the traveling direction from the first set time to the second set time with respect to the first thumbnail. This display control device arranges the thumbnails 100 in chronological order and also arranges them in the traveling direction V1. Therefore, by visually recognizing the thumbnail 100, the user can more preferably relive the movement route, so that the image of the scene to be seen can be detected more appropriately.

Further, the set time designation unit 66 designates the set time based on the time (feature time) at which the feature image satisfying the predetermined condition among the plurality of images P is captured. As a result, the set time designation unit 66 makes the lengths between the set times continuous in the time series different. This display control device specifies a set time based on the time when the feature image is captured. As a result, the thumbnail 100 can display a feature image that the user may want to see, and the user can more appropriately detect the image of the scene that the user wants to see.

Further, the image P is a 360-degree image. Then, the display image generation unit 70 generates the display image P1 so that the thumbnail 100 is displayed on the outer peripheral surface of the spherical display area Q in the virtual three-dimensional space displayed on the display unit. Then, the display image generation unit 70 generates the display image P1 so that the thumbnail 10 is displayed while changing the angle with the rotation of the display area Q by rotating the display area Q. In this way, the display image P1 is displayed so that the thumbnail 100 changes its angle by rotating the spherical display area Q. Therefore, the display image P1 can display the entire 360-degree image as a thumbnail 100. That is, according to this display control device, the thumbnail 100 of the 360-degree image can be appropriately displayed.

Further, the display image generation unit 70 generates a display image P1 in which the rotation of the display area Q is stopped so that each thumbnail 100 is an image displaying the same predetermined direction side. According to this display control device, the thumbnail mode in which the display area Q is rotated and the entire 360-degree image is displayed as the thumbnail 100, and the rotation of the display area Q is stopped and the one-way angle image is displayed as the thumbnail 100. You can switch between the fixed thumbnail mode and the fixed thumbnail mode. Therefore, according to this display control device, by switching the display method of the thumbnail 100, the user can more appropriately detect the image P of the scene that he / she wants to see.

Further, the display control device (here, the external device 14) according to the present embodiment has a thumbnail acquisition unit 68 for acquiring a thumbnail 100 of an image P which is a 360-degree image, and a display image generation unit 70. In the display image generation unit 70, the acquired thumbnail 100 is displayed on the outer peripheral surface of the spherical display area Q in the virtual three-dimensional space displayed on the display unit, and the display area Q is rotated so that the thumbnail 100 is displayed. A display image P1 to be displayed on the display unit is generated so that the display area Q is displayed while changing the angle as the display area Q rotates. According to this display control device, the entire 360-degree image can be displayed as a thumbnail 100. That is, according to this display control device, the thumbnail 100 of the 360-degree image can be appropriately displayed. Therefore, according to this display control device, the user can appropriately detect the desired scene from the captured image P.

Further, the display control system 1 according to the present embodiment includes a display control device, an image pickup device for capturing an image P, and a display unit (display unit 82 in the present embodiment) for displaying the display image P1. According to the display control system 1, by displaying the display image P1 by the display unit, the user can appropriately detect the desired scene from the captured image P.

Further, the image pickup device is provided in the vehicle V and images the inside and the outside of the vehicle V. According to this display control system 1, the user can appropriately detect the desired scene from the images P captured inside and outside the vehicle, and can relive the movement in the vehicle V. ..

In the present embodiment, the display unit that displays the display image P1 is the display unit 82 of the display terminal 16. However, the display image P1 may be displayed on any device of the display control system 1, and may be displayed on, for example, the display unit 32 of the in-vehicle device 12 or the display unit 52 of the external device 14.

Further, in the present embodiment, the external device 14 constitutes the display control device and generates the display image P1, but the device that constitutes the display control device and generates the display image P1 is the display control system 1. Any device may be used. For example, as shown in FIG. 11, the display terminal 16A may configure a display control device. FIG. 11 is a block diagram showing another example of the display control system according to the present embodiment. In the following description of the display control system 1A of FIG. 11, the description of the parts having the same configuration as the display control system 1 shown in FIG. 3 will be omitted.

The display control system 1A of the example shown in FIG. 11 has an external device 14A and a display terminal 16A. The control unit 58A of the external device 14A has an image information acquisition unit 60, and acquires the information of the image P, here the image P and the position information, from the in-vehicle device 12. The control unit 88A of the display terminal 16A displays an image information acquisition unit 90A, an image direction determination unit 62A, an image analysis unit 64A, a set time designation unit 66A, a thumbnail acquisition unit 68A, and a display image generation unit 70A. It has a control unit 92.

The image information acquisition unit 90A acquires the information of the image P acquired by the external device 14A, here, the image P and the position information. The image direction determination unit 62A associates the direction in the image P with the direction in absolute coordinates in the same manner as the image direction determination unit 62 shown in FIG. The image analysis unit 64A detects the feature time by the same method as the image analysis unit 64 shown in FIG. 3, and the set time designation unit 66A uses the same method as the set time designation unit 66 shown in FIG. 3 to set the set time. To specify. Then, the thumbnail acquisition unit 68A acquires the thumbnail 100 by the same method as the thumbnail acquisition unit 68 shown in FIG. 3, and the display image generation unit 70A uses the same method as the display image generation unit 70 shown in FIG. The display image P1 is generated. The display control unit 92 causes the display unit 82 to display the display image P1 generated by the display image generation unit 70A.

As described above, in the display control system 1A shown in FIG. 11, the display terminal 16A constitutes the display control device. That is, the display terminal 16A displays the generated display image P1 while generating the display image P1. In FIG. 11, the display control system 1A does not include the external device 14, and the in-vehicle device 12 may transmit the information of the image P to the display terminal 16A.

Further, for example, the in-vehicle device 12 may configure the display control device. That is, in this case, the display control system 1 may transmit the display image P1 generated by the vehicle-mounted device 12 from the vehicle-mounted device 12 to the display terminal 16 without including the external device 14. Further, in this case, the vehicle-mounted device 12 may display the generated display image P1 on its own display unit 32. Even in this case, the display control system 1 includes the external device 14, transmits the display image P1 generated by the in-vehicle device 12 to the external device 14, and displays the display image P1 from the external device 14 to the display terminal 16. May be sent.

As described above, if the display control device has functions of an image information acquisition unit, a thumbnail acquisition unit, a set time designation unit, and a display image generation unit to generate a display image P1, the display control system 1 Of these, any device may configure the display control device. Further, the display control device may be configured by a plurality of devices. For example, the following configuration is also possible. That is, the external device 14 stores the image P, extracts the feature image, and detects the feature time. The external device 14 transmits the feature time as the information of the image P to the display terminal 16. The display terminal 16 sets the set time based on the feature time, and transmits the information of the set time to the external device 14. The external device 14 generates thumbnails 100 based on the set time, and generates a display image P1 in which the thumbnails 100 are arranged. The display terminal 16 acquires the display image P1 and displays it on the display unit 82. In this case, since the display terminal 16 does not acquire image data or position information as the information of the image P, the amount of data to be communicated can be suppressed.

Although the embodiments of the present invention have been described above, the embodiments are not limited by the contents of these embodiments. Further, the above-mentioned components include those that can be easily assumed by those skilled in the art, those that are substantially the same, that is, those in a so-called equal range. Furthermore, the components described above can be combined as appropriate. Further, various omissions, replacements or changes of the components can be made without departing from the gist of the above-described embodiment.

1 Display control system 10 Imaging device 12 In-vehicle device 14 External device (display control device)
16 Display terminal 60 Image information acquisition unit 62 Image direction determination unit 64 Image analysis unit 66 Setting time specification unit 68 Thumbnail acquisition unit 70 Display image generation unit 82 Display unit 90 Display image acquisition unit 92 Display control unit 100 Thumbnail P image P1 Display image Q Display area V Vehicle V1 Travel direction

Claims (5)

An image information acquisition unit that acquires information on multiple images captured while moving,
A set time specification unit that specifies a plurality of set times based on the information in the image,
A thumbnail acquisition unit that acquires the image captured at the set time as a thumbnail, and
The plurality of acquired thumbnails are arranged in a predetermined direction in the order of the captured time series, and are displayed side by side in a direction different from the predetermined direction based on the position information indicating the position where the image is captured. As such, it has a display image generation unit that generates a display image displayed on the display unit.
The image is a 360 degree image and is
The set time designation unit specifies the set time based on the time when the feature image satisfying the predetermined condition among the plurality of images is captured, so that the set time can be set during the time series. Different lengths,
The display image generation unit causes the thumbnail to be displayed on the outer peripheral surface of a spherical display area in the virtual three-dimensional space displayed on the display unit, and the thumbnail extracted as the feature image and the feature image The display image is generated so that the moving trajectory of the display area is different from that of the thumbnails that have not been extracted, and the thumbnails are displayed while changing the angle.
Display control device. The position information includes information on the traveling direction when the image is captured.
The thumbnail has a first thumbnail of a first set time and a second thumbnail of a second set time which is a set time after the first set time.
The display image generation unit has the first thumbnail and the first thumbnail so that the second thumbnail is located on the traveling direction side from the first set time to the second set time with respect to the first thumbnail. The display control device according to claim 1, wherein the two thumbnails are arranged side by side. The display image generation unit according to claim 1 or 2 , wherein the display image generation unit generates the display image in which the rotation of the display area is stopped so that each of the thumbnails is an image displaying the same predetermined direction side. The display control device described. An image information acquisition step to acquire information on multiple images captured while moving, and
A set time specification step that specifies a plurality of set times based on the information in the image, and
A thumbnail acquisition step of acquiring the image captured at each of the set times as a thumbnail, and
The plurality of acquired thumbnails are arranged in the order of the time series in which the images are captured in a predetermined direction, and are displayed side by side in a direction different from the predetermined direction based on the position information indicating the position where the image is captured. Has a display image generation step, which generates a display image to be displayed on the display unit.
The image is a 360 degree image and is
In the set time designation step, by designating the set time based on the time when the feature image satisfying the predetermined condition among the plurality of images is captured, the set time is continuously specified during the set time. Different lengths,
In the display image generation step, the thumbnail is displayed on the outer peripheral surface of a spherical display area in the virtual three-dimensional space displayed on the display unit, and the thumbnail extracted as the feature image and the feature image are displayed. The display image is generated so that the moving trajectory of the display area is different from that of the thumbnail not extracted as, and the thumbnail is displayed while changing the angle.
Display control method. An image information acquisition step to acquire information on multiple images captured while moving, and
A set time specification step that specifies a plurality of set times based on the information in the image, and
A thumbnail acquisition step of acquiring the image captured at each of the set times as a thumbnail, and
The plurality of acquired thumbnails are arranged in the order of time series captured in a predetermined direction, and are displayed side by side based on the position information indicating the position where the image is captured in a direction different from the predetermined direction. It is a program that causes a computer to execute a display image generation step of generating a display image displayed on the display unit.
The image is a 360 degree image and is
In the set time designation step, by designating the set time based on the time when the feature image satisfying the predetermined condition among the plurality of images is captured, the set time is continuously specified during the set time. Different lengths,
In the display image generation step, the thumbnail is displayed on the outer peripheral surface of a spherical display area in the virtual three-dimensional space displayed on the display unit, and the thumbnail extracted as the feature image and the feature image are displayed. The display image is generated so that the moving trajectory of the display area is different from that of the thumbnail not extracted as, and the thumbnail is displayed while changing the angle.
program.

Images