JP6388532B2 - Image providing system and image providing method - Google Patents

Description

  The present invention relates to an image providing system and an image providing method.

  There is a technique for associating position information of a shooting position with a subject image, specifying a subject image to be displayed, and displaying the specified subject image. For example, there has been proposed a technique for searching for an image in which a desired search object is reflected from a database that stores a shooting position and image data in association with each other.

  Also, techniques have been proposed to identify objects acquired by digital images based on location data for automatic annotation of digital images and for latching images to create collages (eg, (See Patent Documents 1 and 2).

International Publication No. 2013/114473 Special table 2010-509668 gazette

  As one aspect, an object of the present invention is to display a series of images of a section in which a subject is captured from a plurality of images with a simple operation.

In one embodiment, the image providing system, an image providing system including a server and a display terminal, wherein the server, based on the information about the object which the display terminal is specified, identifies the position of existence point of the object A generating unit that generates specific data for specifying a series of surrounding images of a section including the subject, based on a shooting position where the subject is shot and a position of the location where the subject exists, and the specific data And a transmission unit that transmits a series of surrounding images of a section including the subject based on the specific data from a storage device that stores a plurality of surrounding images. And a display unit that displays an image of the subject in a series of surrounding images extracted by the extraction unit based on the specific data.

  According to one aspect, a series of images of a section in which a subject is captured from a plurality of images can be displayed with a simple operation.

It is a figure which shows an example of an image provision system. It is a figure which shows an example of the relationship between the camera mounted in the vehicle and a visual field. It is a functional block diagram which shows an example of the control part contained in a vehicle. It is a figure which shows an example of vehicle-mounted image metadata. It is a flowchart which shows an example of the process which a control part performs. It is a flowchart which shows an example of a process of an image storage server and a metadata server. It is a sequence chart which shows an example of the process which a display terminal and an image search server perform. It is a flowchart which shows an example of a production | generation of synthetic | combination image metadata. It is a figure which shows an example of the area of vehicle-mounted image metadata. It is a figure which shows an example of the bounding box surrounding a to-be-photographed object. It is a figure which shows an example of the polygon data containing a to-be-photographed object. It is a figure which shows an example of the relationship between vehicle-mounted image metadata and synthetic | combination video metadata. It is a figure which shows an example of composite image metadata. It is a figure which shows an example of a selection screen. It is a flowchart which shows an example of the process which sets and displays a visual field. It is a figure which shows an example of the omnidirectional image to which the texture was set. It is a figure which shows an example of the image which caught the to-be-photographed object. It is a figure which shows an example of the virtual visual field of the virtual camera at the time of changing a viewpoint and a visual field. It is a figure which shows an example of the moving image displayed on a display part. It is a figure which shows the example of a screen for performing the image selection based on visibility. It is a figure which shows an example of the hardware constitutions of an image search server. It is a figure which shows an example of the hardware constitutions of a display terminal.

<Inventor's knowledge>
For example, an image of the entire periphery is captured at a predetermined timing while a vehicle equipped with a camera capable of capturing the entire periphery moves. Therefore, the entire surrounding image captured by the camera is an image captured at a different position. When shooting while the vehicle passes near the subject, the omnidirectional image including the subject becomes a series of omnidirectional images.

  When the user's terminal (hereinafter referred to as a display terminal) is set to display an image of the field of view in the traveling direction of the vehicle among the all-around image, even if the subject is included in the all-around image, the display The subject may not be displayed on the terminal. For example, when there is a subject behind the traveling direction of the vehicle, the subject is not displayed on the display terminal even if the subject is included in the all-around image.

  In this case, the user manually sets the field of view by operating the display terminal. As a result, the display terminal can display a subject behind the vehicle in the traveling direction, and the display terminal can display a subject photographed by camera work centered on the subject. However, in this case, since the subject is displayed with camera work centered on the subject by manual operation of the user, the operability becomes complicated.

  In addition, when reproducing a moving image of a subject, it is difficult to reproduce the moving image with camera work centered on the subject while manually setting the field of view. Therefore, it is desirable that the camerawork image captured around the subject in the series of surrounding images including the subject is displayed without the user performing a field-of-view setting operation.

<Example of overall configuration of image providing system>
Hereinafter, embodiments will be described with reference to the drawings. FIG. 1 shows an example of an image providing system 1. In the image providing system 1, a plurality of vehicles 2 (2A, 2B,...), An image storage server 3, an image search server 4, a metadata server 5, a map server 6, and a display terminal 7 are connected via a network 8. Has been.

  The vehicle 2 is an automobile, for example. The vehicle 2 is equipped with a camera capable of photographing the entire periphery. In the embodiment, the plurality of vehicles 2 captures an all-around image having the entire view as a field of view at a predetermined timing while moving. Since the plurality of vehicles 2 capture the entire surrounding image while moving, different vehicles 2 may capture the same subject.

  Each vehicle 2 gives an image ID to the captured all-around image and transmits it to the image storage server 3 via the network 8 at a predetermined timing. ID is an abbreviation for Identification. In addition, each vehicle 2 transmits data regarding the captured all-around image (hereinafter referred to as in-vehicle image metadata) to the metadata server 5 via the network 8.

  The image storage server 3 stores the all-around image transmitted from each vehicle 2 in association with the image ID. Further, the image storage server 3 extracts a series of all-around images including the subject designated from the display terminal 7 and transmits the extracted images to the display terminal 7. The image storage server 3 is an example of a storage device.

  The image search server 4 includes a server communication unit 11, a subject position specifying unit 12, a composite image metadata generation unit 13, and a calculation unit 14. The image search server 4 is an example of a server. The server communication unit 11 performs data communication with the network 8. The server communication unit 11 is an example of a transmission unit.

  The subject position specifying unit 12 specifies the position of the designated subject. In the embodiment, the subject position specifying unit 12 specifies the position of the subject corresponding to the keyword from the map server 6 based on the keyword related to the subject transmitted from the display terminal 7. The subject position specifying unit 12 is an example of a specifying unit.

  The composite image metadata generation unit 13 generates composite image metadata. The composite image metadata is data that identifies the all-around image of the section including the subject specified by the display terminal 7 in the series of all-around images of the in-vehicle image metadata. The composite image metadata is an example of specific data. The composite image metadata generation unit 13 is an example of a generation unit.

  In the embodiment, whether or not the subject is included in the surrounding image is determined based on whether or not the distance between the position of the subject and the shooting position is equal to or less than a predetermined threshold. However, the criterion for determining whether or not the surrounding image includes a subject is not limited to the above example.

  The calculation unit 14 calculates the value of each parameter included in the composite image metadata. For example, the calculation unit 14 calculates the distance between the above-described shooting position and the subject position. Further, the calculation unit 14 calculates the azimuth and elevation angle of the position of the subject viewed from the shooting position.

  The metadata server 5 stores the above-described vehicle-mounted image metadata and composite image metadata. The metadata server 5 stores a plurality of in-vehicle image metadata and a plurality of composite image metadata.

  The map server 6 stores a map position and a keyword in association with each other. The keyword is information for specifying the subject. For example, the keyword may be a facility name or address. The position on the map may be specified by, for example, longitude and latitude. The map stored in the map server 6 may be a three-dimensional map.

  The display terminal 7 is a terminal owned by the user. For example, a personal computer, a smart phone, a mobile phone, a tablet terminal, or the like may be used. The display terminal 7 includes a terminal communication unit 21, an image extraction unit 22, a visual field setting unit 23, an image processing unit 24, and a display unit 25.

  The terminal communication unit 21 performs communication via the network 8. The image extraction unit 22 extracts a plurality of omnidirectional images from the image storage server 3 based on the composite image metadata of the section including the subject.

  The visual field setting unit 23 sets a visual field (also referred to as a virtual visual field) in the entire surrounding image based on the composite image metadata received from the image search server 4. The image processing unit 24 performs processing for generating a field-of-view image set by the field-of-view setting unit 23 among the entire surrounding images.

  The display unit 25 displays the image processed by the image processing unit 24. For example, the display unit 25 may be a touch panel display. When the display unit 25 is a touch panel display, the display unit 25 has an operation function. When the display terminal 7 has an operation unit (not illustrated) (for example, a button), the display unit 25 may not have an operation function.

  In the example of FIG. 1, the image storage server 3, the image search server 4, and the metadata server 5 are provided as separate servers, but these servers may be a single server. For example, the image search server 4 may have the functions of the image storage server 3 and the metadata server 5.

<Example of vehicle>
FIG. 2 shows an example of a vehicle 2 equipped with a camera capable of photographing the entire periphery. Cameras C <b> 1 to C <b> 4 are provided on the front, rear, left and right of the vehicle 2. The visual fields corresponding to the cameras C1 to C4 are V1 to V4.

  Each of the cameras C1 to C4 uses a wide-angle lens, and the visual fields V1 to V4 are three-dimensionally wide. The visual field V1 and the visual field V4 have a visual field range that overlaps the visual field V2 and the visual fields V2 and V3. When the images captured by the cameras C1 to C4 are combined, an all-around image with the entire view centered on the vehicle 2 as a field of view is generated.

  In the embodiment, the omnidirectional image can be generated based on images of four channels taken by the cameras C1 to C4. However, the present invention is not limited to the example of FIG. For example, a single camera using a 360-degree omnidirectional lens may be installed on the roof of the vehicle 2 and this camera may shoot an all-around image.

  In the embodiment, the image generated by the camera mounted on the vehicle 2 is described as an all-around image, but is not limited to the all-around image. For example, a camera mounted on the vehicle 2 may capture a surrounding image of 350 degrees.

  FIG. 3 shows an example of the control unit 30 included in the vehicle 2. The images taken by the cameras C1 to C4 are time synchronized. The four images synchronized in time are stored in the primary storage unit 31. The primary storage unit 31 may be a memory, for example.

  The four images that are time-synchronized are referred to as an all-around image regardless of whether they are synthesized. In the embodiment, it is assumed that the omnidirectional image is an image that has not been synthesized until the image processing unit 24 of the display terminal 7 performs image processing. However, the four images synchronized in time may be combined by the control unit 30.

  The compression encoding unit 32 reads the all-around image from the primary storage unit 31 and compress-encodes the read all-around image. For example, for each of four images (resolution: 640 × 480) of the omnidirectional image, compression is performed on the omnidirectional image (resolution: 1280 × 960) obtained by combining four omnidirectional images at different times. The encoding unit 32 may perform compression encoding. Examples of compression encoding include H.264. H.264 or the like, to which the compression coding unit 32 performs compression coding, is stored in the in-vehicle storage unit 36.

  The time measuring unit 33 measures time. The position information acquisition unit 34 acquires information on the current position. For example, the position information acquisition unit 34 may be a Global Positioning System (GPS). The position information acquisition unit 34 and the all-around image stored in the primary storage unit 31 are time-synchronized.

  The in-vehicle image metadata generation unit 35 generates in-vehicle image metadata. The in-vehicle image metadata is data related to the all-around image captured by the cameras C1 to C4. The in-vehicle image metadata shown in the example of FIG. 4 includes parameters such as an image ID, a shooting time, a shooting position, camera specific information, and camera installation information.

  The image ID is an identifier that identifies the entire surrounding image. The photographing time is a time measured by the time measuring unit 33 when the cameras C1 to C4 perform photographing. The shooting position is position information acquired by the position information acquisition unit 34 when an all-around image is acquired.

  The camera specific information is information relating to the resolution, lens distortion, and the like of the cameras C1 to C4. The camera installation information is information related to the installation positions and postures of the cameras C1 to C4. The camera specific information and the camera installation information are known information.

  The in-vehicle image metadata may have parameters other than those shown in the example of FIG. Cameras C <b> 1 to C <b> 4 mounted on the vehicle 2 capture an all-around image at a predetermined timing. The in-vehicle image metadata generation unit 35 assigns a unique image ID to generate in-vehicle image metadata each time an all-around image is generated.

  The image ID of the in-vehicle image metadata and the all-around image corresponding to the image ID are associated and stored in the in-vehicle storage unit 36. The vehicle communication unit 37 transmits the in-vehicle image metadata to the metadata server and transmits the entire surrounding image to the image storage server 3 at every predetermined timing.

  In addition, it is desirable that the transmitted information is deleted from the all-around image and the in-vehicle image metadata stored in the in-vehicle storage unit 36. Thereby, the information amount memorize | stored in the vehicle-mounted memory | storage part 36 can be reduced.

<Flowchart showing an example of processing of the control unit>
Next, a flowchart of processing performed by each unit of the control unit 30 will be described with reference to FIG. The cameras C1 to C4 mounted on the vehicle 2 perform shooting (step S1). The four images captured by the cameras C1 to C4 are stored in the primary storage unit 31.

  These four images are time-synchronized, and images taken by the cameras C1 to C4 are stored in the primary storage unit 31 (step S2). As described above, in the embodiment, the four images synchronized in time are collectively referred to as an all-around image.

  The compression encoding unit 32 performs compression encoding of the entire surrounding image (step S3). The control unit 30 assigns an image ID to the omnidirectional image that has been compression-encoded (step S4). The control part 30 memorize | stores the omnidirectional image to which image ID was provided in the vehicle-mounted memory | storage part 36 (step S5).

  In parallel with the processes of steps S1 to S5, the following processes of steps S6 to S9 are performed. The position information acquisition unit 34 acquires position information when the cameras C1 to C4 are photographed (step S6). The in-vehicle image metadata generation unit 35 associates the time measured by the time measurement unit 33 with the position information acquired by the position information acquisition unit 34 (step S7).

  The in-vehicle image metadata generation unit 35 acquires an image ID corresponding to the position information acquired from the position information acquisition unit 34 (step S8). Thereby, the vehicle-mounted image metadata including the time information and the position information is associated with the all-around image and stored in the vehicle-mounted storage unit 36 (step S9).

  While the vehicle 2 is moving, the camera 2 uses the cameras C <b> 1 to C <b> 4 to take an image at a predetermined timing. Therefore, a series of omnidirectional images over time taken at different points is generated. Then, for each time, the all-around image and the in-vehicle image metadata are associated with each other and stored in the in-vehicle storage unit 36.

  The vehicle communication unit 37 determines whether or not a connection with the image storage server 3 has been established (step S10). When the connection is established (YES in step S10), the vehicle communication unit 37 transmits the all-around image to the image storage server 3 (step S11). On the other hand, when the connection is not established (NO in step S10), the all-around image is not transmitted.

  The vehicle communication part 37 determines whether the connection with the metadata server 5 is established (step S12). When the connection is established (YES in step S12), the vehicle communication unit 37 transmits the in-vehicle image metadata to the metadata server 5 (step S13). On the other hand, when the connection is not established (NO in step S12), the in-vehicle image metadata is not transmitted.

  The control unit 30 may delete the transmitted all-around image and in-vehicle image metadata from the in-vehicle storage unit 36. Since the cameras C1 to C4 shoot at a predetermined timing, the all-around image and the in-vehicle video metadata increase every time shooting is performed. Therefore, the amount of information stored in the in-vehicle storage unit 36 can be reduced by deleting the entire surrounding image and in-vehicle image metadata that have been transmitted by the control unit 30 from the in-vehicle storage unit 36.

<Example of processing of image storage server and metadata server>
FIG. 6A shows an example of processing of the image storage server 3. The image storage server 3 determines whether or not a connection is established with the vehicle communication unit 37 of the vehicle 2 (step S21).

  When the connection is established (YES in step S21), the image storage server 3 determines whether or not an all-around image has been received from the vehicle communication unit 37 (step S22). If it is determined that the omnidirectional image has been received (YES in step S22), the image storage server 3 stores the received omnidirectional image in association with the image ID (step S23). As a result, the all-around image is accumulated in the image accumulation server 3.

  If the connection with the vehicle communication unit 37 is not established (NO in step S21) or if no image is received (NO in step S22), the image storage server 3 does not perform the process of step S23.

  FIG. 6B shows an example of processing of the metadata server 5. The metadata server 5 determines whether or not a connection is established with the vehicle communication unit 37 of the vehicle 2 (step S24).

  When the connection is established (YES in step S24), the metadata server 5 determines whether or not the in-vehicle video metadata is received from the vehicle communication unit 37 (step S25). When it is determined that the in-vehicle video metadata has been received (YES in step S25), the metadata server 5 stores the received in-vehicle video metadata (step S23).

  If the connection with the vehicle communication unit 37 is not established (NO in step S24), or if the in-vehicle video metadata is not received (NO in step S25), the metadata server 5 performs the process of step S26. Absent.

<Example of processing performed between display terminal and image search server>
Next, an example of processing performed by the display terminal 7 and the image search server 4 will be described with reference to the sequence chart of FIG. The user of the display terminal 7 inputs information on the subject to be displayed on the display terminal 7. For example, when the display unit 25 has an operation function, the user uses the display unit 25 to input information on a subject to be processed.

  The display terminal 7 transmits information on the input subject (subject specified by the user) to the image search server 4. In the embodiment, it is assumed that the subject information designated by the user is a keyword.

  As described above, the keyword may be the name or address of the facility. Accordingly, the terminal communication unit 21 transmits the input keyword to the image search server 4 (step SC1). Note that the subject information specified by the user is not limited to keywords. For example, the subject information may be position information indicating the longitude and latitude of the subject.

  The image search server 4 transmits the received keyword to the map server 6. The map server 6 transmits the position information associated with the keyword to the image search server 4. Thereby, the image search server 4 acquires position information based on the keyword (step SC2). The acquired position information is the position information of the subject specified by the user.

  The image search server 4 extracts, from the metadata server 5, in-vehicle video metadata including the acquired position information (shooting position in the example of FIG. 4) from among the plurality of in-vehicle video metadata stored in the metadata server 5. (Step SC3). There may be a plurality of in-vehicle video metadata to be extracted.

  The composite image metadata generation unit 13 of the image search server 4 specifies the image of the section including the subject from the in-vehicle video metadata, and generates composite image metadata (step SC4). An example of the generation of the composite image metadata will be described with reference to the flowchart in FIG.

  The calculation unit 14 calculates a distance D between the shooting position and the subject position included in the extracted in-vehicle video metadata (step S31). As the distance D, the following formula (1) for calculating the distance between two points may be used.

In the following formula (1), the shooting position is (longitude x1, latitude y1), and the subject position is (longitude x2, latitude y2). The shooting position is included in the in-vehicle video metadata, and the position of the subject is specified by the position information acquired in step SC2. r is the equator radius of the earth.
“D = r × cos ⁻¹ (sin (y1) × sin (y2) + cos (y1) × cos (y2) × cos (x2−x1))” (Formula 1)

  Next, the calculation unit 14 calculates a section Tseg in which the distance D between the shooting position obtained by the above equation (1) and the subject position is equal to or less than the threshold value Dth. The section Tseg indicates a section in which the subject is included in the entire surrounding image.

  FIG. 9 shows an example in which the section Tseg (the hatched section) is specified in the in-vehicle image metadata. If the distance between the shooting position and the subject position is short, the subject appears clearly in the entire surrounding image. On the other hand, if the distance between the shooting position and the subject position is long, the sharpness of the subject in the all-around image decreases.

  Therefore, the above-mentioned threshold value Dth is used as a reference for determining whether or not a subject is included in the all-around image. An arbitrary value may be set as the threshold value Dth. A threshold value Dth may be input to the image search server 4.

  When the bounding box BB of the subject shown in FIG. 10 to be described later is obtained, the distance necessary for the subject to appear in the entire surrounding image at a predetermined pixel or more is determined based on the camera-specific information included in the in-vehicle image metadata. It may be Dth.

  In the case of FIG. 9, among the series of all-around images starting from time T1, the shooting position and the subject position in the section from time T4 to T7 satisfy “D ≧ Dth”. There is a possibility that the entire surrounding image at the time T3 or the time T8 also includes the subject. However, in the example of FIG. 9, since the time T3 and the time T8 do not satisfy “D ≧ Dth”, the composite image metadata generation unit 13 determines that the subject is not included in the all-around image.

  The shooting position changes with time as shown in the example of FIG. In the example of FIG. 9, the shooting position is position P4 at time T4, and is position P5 at time T5. That is, since the cameras C1 to C4 mounted on the vehicle 2 perform shooting at a predetermined timing, a series of all-around images for each shooting timing is generated.

  The distance at time T is Dt. The above equation (1) may be Dt = F (X, Pt). X is the position of the subject, and Pt is the shooting position at time t. Then, the distance Dt at time T can be obtained by the function F of the above equation (1).

  Further, as described above, if the start time when “D ≧ Dth” is Ts and the end time Te, the section Tseg can be expressed as “Ts, Te”. In the example of FIG. 9, the start time Ts of the section Tseg is T4, and the end time Te is T7.

As shown in the example of FIG. 8, after the section Tseg is calculated, the composite image metadata generation unit 13 calculates the azimuth information θ (step S33). The azimuth information θ is the azimuth when the subject position is photographed from the photographing position. The azimuth information θ is obtained by the following equation (2), for example. Note that x1, x2, y1, and y2 are the same as those in the above-described formula (1).
“Θ = 90−arctan2 (sin (x2−x1), cos (y1) × tan (y2) −sin (y1) × cos (x2−x1))” (2)

  Thereby, the azimuth information θ of the subject viewed from the photographing position is obtained. Since the shooting position changes each time the vehicle 2 moves, the azimuth information θ also changes depending on the shooting position. Since the shooting position changes with the shooting time, the azimuth information θ also changes with the time. Therefore, the azimuth information θt at time t can be expressed as θt = G (X, Pt). That is, if Expression (2) is expressed as a function G, θt = G (X, Pt).

  The computing unit 14 calculates elevation angle information φ (step S34). The elevation angle information φ indicates the elevation angle of the subject relative to the horizontal plane direction. Since the elevation angle information φ also changes with time, the elevation angle information φt can be obtained. The elevation angle information φt can be obtained, for example, based on the bounding box BB indicated by the broken line in FIG.

The elevation angle information φt is obtained by the following equation (3) based on the center coordinates Bc of the eight vertices B1 to B8 of the bounding box BB when a rectangular parallelepiped bounding box BB surrounding the subject three-dimensionally is set. Can be obtained. The center coordinate Bc is (xc, yc, zc). xc, yx, and zx indicate the center coordinates of the x-axis, y-axis, and z-axis of the bounding box BB.
“Φt = arctan (zc / Dt)” (Formula 3)

  The computing unit 14 calculates the visual field information FOV (step S35). The visual field information FOV defines a bounding box BB shown as an example in FIG. The visual field information FOV is defined by a horizontal viewing angle (fovH) and a vertical viewing angle (fovV).

  For example, the visual field information FOV may be set to just include the bounding box BB, or may be a fixed value. When the visual field information FOV is a fixed value, the calculation unit 14 does not perform calculation for obtaining the visual field information FOV. Next, the calculating part 14 calculates visibility V (step S36).

  Since the visibility V also changes with time, the visibility V at time t is defined as visibility Vt. Visibility Vt indicates the degree to which a three-dimensional area (for example, a bounding box BB) of a subject can be seen from the shooting position. That is, the visibility Vt indicates the rate of the subject that appears in the all-around image at time t.

For example, as shown in FIG. 10, the number of line-of-sight vectors when a line-of-sight vector is emitted with a predetermined angular resolution from the shooting position (Pt) toward the bounding box BB is N. The example of FIG. 11 is an example of polygon data including a subject. The calculation unit 14 calculates the rate at which the line-of-sight vector collides with the subject. When the total number of hits (collisions) on the polygons of the subject is M, the calculation unit 14 calculates the visibility Vt by the following expression (4).
“Vt = M / N” (Formula 4)

  For example, if there is any shielding object between the shooting position and the subject, the line-of-sight vector that hits the shielding object among the line-of-sight vectors emitted from the shooting position does not hit the subject. For this reason, since M decreases, visibility Vt also decreases.

The computing unit 14 computes the visibility goodness Q (step S37). The visibility goodness Q is a value that is comprehensively evaluated from the visibility Vt in the section Tseg obtained in step S32. For example, the visibility goodness Q may be calculated as an average value of all the visibility V in the section Tseg as in the following formula (5).

In equation (5), fs indicates the total number of frames. For example, in the case of the example of FIG. 12 showing an example of the relationship between the in-vehicle image metadata and the composite video metadata, the visibility goodness Q in the section Tseg is expressed by the following formula (6).
“Q = (V4 + V5 + V6 + V7) / 4” (Formula 6)

  Thus, the process of step SC4 in FIG. 7 ends, and composite image metadata is generated. As shown in the example of FIG. 12, in the in-vehicle image metadata, a section from time T4 to T7 is a section Tseg (section where hatching is performed).

  The calculation unit 14 calculates the azimuths θ4 to θ7, the elevation angles φ4 to φ7, the visual fields FOV4 to FOV7, and the visibility V4 to V7 during the section Tseg. FIG. 13 is an example of parameters included in the composite image metadata.

  The composite image metadata ID is an identifier for identifying composite image metadata. The image ID, shooting time, shooting position, camera-specific information, and camera installation information are data included in the in-vehicle image metadata that is a source for generating the composite image metadata.

  The section information indicates the start time and end time of the section Tseg for the composite image metadata. Since the section information is “Ts, Te” as described above, the section information is “T4, T7” in the example of FIG.

  The composite image metadata generation unit 13 generates composite image metadata for each time included in the section Tseg. In the case of the example in FIG. 12, the composite image metadata generation unit 13 generates composite image metadata for four times from time T4 to T7.

  As described above, a plurality of composite image metadata including a designated subject may be generated. The server communication unit 11 transmits the composite image metadata ID for specifying each of the generated composite image metadata to the display terminal 7 in a list format (step SC5). The terminal communication unit 21 of the display terminal 7 receives the list of composite image metadata IDs.

  The display terminal 7 acquires a plurality of composite image metadata specified by the composite image metadata ID included in the list from the metadata server 5 (step SC6). For this purpose, the terminal communication unit 21 transmits the composite image metadata ID included in the list to the metadata server 5.

  The metadata server 5 transmits a plurality of composite image metadata specified by the composite image metadata ID to the display terminal 5. Thereby, the display terminal 7 acquires a plurality of composite image metadata.

  The display terminal 7 displays a screen (hereinafter referred to as a selection screen) for selecting which composite image metadata is to be displayed among the plurality of obtained composite image metadata (step SC7). Note that when the composite image metadata received by the display terminal 7 is one, the process of step SC7 may not be performed.

  FIG. 14 shows an example of the selection screen. The selection screen displays the visibility goodness Q for each composite image metadata ID. In the example of FIG. 14, the shooting time and thumbnail are displayed for one composite image metadata ID.

  The visual goodness Q is an integrated evaluation value of the visibility V for one entire surrounding image in the section Tseg, and indicates an average quality when a moving image is displayed. The user selects a desired composite image metadata ID from the selection screen displayed on the display unit 25. Therefore, the user can view a high-quality moving image including the designated subject by selecting the composite image metadata ID having a high visibility goodness Q value.

  In the example of FIG. 14, the display unit 25 may display the maximum value of the visibility V instead of the visibility goodness Q. In this case, the user can select a still image with high visibility V among a plurality of still images specified by the composite image metadata ID. Thereby, the display unit 25 can display a high-quality still image including the designated subject.

  The terminal communication unit 21 of the display terminal 7 transmits the selected composite image metadata ID to the metadata server 5. The metadata server 5 transmits the selected composite image metadata to the display terminal 7. Thereby, the display terminal 7 acquires the selected composite image metadata (step SC8).

  As shown in the example of FIG. 7, when the composite image metadata ID is selected, the image extraction unit 22 extracts a series of images based on the selected composite image metadata ID from the image storage server 3 (step SC9). ). The composite image metadata includes an image ID and section information. In the example of FIG. 12, the section information is information indicating time T4 to time T7.

  Accordingly, the display terminal 7 transmits the image ID and the section information included in the selected composite image metadata to the image storage server 3. The image storage server 3 extracts a series of all-around images specified by the section information for all-around images specified by the image ID from among the plurality of stored all-around images. The image storage server 3 transmits the extracted series of all-around images to the terminal communication unit 21. A series of all-around images extracted at this time becomes a series of all-around images in the section Tseg.

  The terminal communication unit 21 receives a series of all-around images in the section Tseg. As shown in the example of FIG. 7, the display terminal 7 sets a field of view in the entire surrounding image, and the image processing unit 24 performs image processing with the set field of view. Then, the image processed image is displayed on the display unit 25 (step SC10).

  FIG. 15 shows an example of the process of step SC8. As described above, the all-around image received by the display terminal 7 is four images captured by the cameras C1 to C4. The image processing unit 24 synthesizes the acquired all-around image and sets a three-dimensional curved surface centered on the vehicle 2 (step S41). The three-dimensional curved surface may be defined by a polygon, for example.

  The image processing unit 24 performs texture setting (step S42). For example, the image processing unit 24 calculates a line-of-sight vector facing the polygon vertex of the three-dimensional curved surface from the camera arranged in the virtual space based on the camera specific information and the camera installation information included in the composite image metadata.

  Then, the image processing unit 24 calculates the pixel position of the camera corresponding to the line-of-sight vector based on the camera specific information. Thereby, the image processing unit 24 can calculate the correspondence between the polygon vertex and the texture pixel position.

  An example of the all-around image in which the texture is set is shown in FIG. In FIG. 16, the omnidirectional image 41 is shown as an ellipse. However, as described above, the omnidirectional image is a three-dimensional curved surface. For example, the all-around image in which the texture is set has a bowl-shaped shape that is curved three-dimensionally.

  The display terminal 7 receives the composite image metadata corresponding to the omnidirectional image in the example of FIG. Therefore, the visual field setting unit 23 can set the virtual visual field of the virtual camera 42 for the entire peripheral image in the example of FIG. 16 based on the orientation information θ and the elevation angle information φ included in the composite image metadata. In FIG. 16, the virtual visual field is indicated by a one-dot chain line. This virtual visual field includes a subject 44.

  The image processing unit 24 performs a drawing process with the set virtual visual field (step S44). Thereby, an image (an image including the subject 44) capturing the subject 44 as shown in the example of FIG. 17 is displayed on the display unit 25. The subject 44 in the example of FIG. 17 shows a building.

  As described above, as the vehicle 2 moves, the shooting location changes, and the viewpoint and field of view of the subject change. That is, in the section Tseg, the shooting positions of a series of all-around images change every moment. Therefore, the visual field setting unit 23 changes the setting of the viewpoint and the visual field for the omnidirectional image at the next time in the section Tseg based on the composite image metadata corresponding to the omnidirectional image (step S45).

  FIG. 18 shows an example of the virtual field of view of the virtual camera 42 when the viewpoint and field of view are changed. The visual field setting unit 23 changes the virtual visual field based on the azimuth information θ and the elevation angle information φ at the next time of the section Tseg.

  A series of all-around images in the section Tseg has different shooting positions. Therefore, the azimuth information θ and the elevation angle information φ of the series of all-around images in the section Tseg vary depending on the all-around image. That is, camera work differs depending on the entire surrounding image.

  The image processing unit 24 determines whether or not to end the display (step S46). For example, when the image display based on the entire surrounding image in the section Tseg is completed (YES in step S46), the display unit 25 ends the display. Alternatively, when the display terminal 7 is operated to end the display, the display unit 25 ends the display.

  On the other hand, when the display is not terminated (NO in step S46), the processes in steps S43 to S45 are repeated. Accordingly, the processes in steps S43 and S44 are repeated during the section Tseg. The field-of-view setting unit 23 displays an image captured by camera work that captures the subject as needed based on the orientation information θ and the elevation angle information φ included in the composite image metadata.

  Thereby, the user can view a series of images of the section Tseg captured by camera work capturing the subject without performing a special operation. The visual field setting unit 23 changes the position of the virtual camera 42 based on the camera work information including the azimuth information θ and the elevation angle information φ.

  Therefore, when viewing a series of images in the section Tseg, each image is an image that is always captured around the subject. For example, when the display terminal 7 displays a series of images in the section Tseg as a moving image, the user can simply view a camerawork moving image that always captures the subject with a simple operation simply by specifying a keyword.

  FIG. 19 shows an example of a moving image displayed on the display unit 25. FIG. 19A is an example of an image based on the entire surrounding image captured by the cameras C1 to C4 before reaching the position where the subject 44 is captured from the front.

  In the case of FIG. 19A, a part of the subject 44 is hidden by the shield 45. Examples of the shield 45 include a utility pole and a tree. In the example of FIG. 19A, the visibility of the subject 44 is V = 0.7 due to the shield 45.

  FIG. 19B is an example of an image based on an all-around image obtained by photographing the subject 44 from the front. In this case, the camera work has changed, and the visibility V of the subject 44 is not significantly affected by the shield 45. In the case of FIG. 19B, the visibility V is V = 0.9.

  FIG. 19C is an example of an image based on an all-around image captured at a position passing through the subject 44 from the front. In this case, the camera work has changed, and the visibility V of the subject 44 is reduced due to the shield 45. In the case of FIG. 19C, the visibility V is V = 0.7.

  FIG. 20 shows an example of a screen for performing image selection based on the visibility V. The display unit 25 of the display terminal 7 includes an image display area 51, a visibility graph 52, and a slider bar 53. The image display area 51 is an area for displaying an image including the subject 44.

  The visibility graph 52 is a graph showing the visibility V that changes with time in the section Tseg. The slider bar 53 is a bar for designating an arbitrary time T of the visibility graph 52. The user can move the slider bar 53 along the direction of time T.

  Accordingly, by positioning the bar of the slider bar 53 at a position where the visibility V is the highest, a still image with the highest visibility V is displayed in the image display area 51. Therefore, as described above, it is possible to give an index to the user as to which part of the image to which camerawork is applied that is capturing the subject 44 well.

<Example of hardware configuration of image search server>
Next, an example of the hardware configuration of the image search server 4 will be described with reference to the example of FIG. As illustrated in the example of FIG. 21, a CPU 111, a RAM 112, a ROM 113, an auxiliary storage device 114, a medium connection unit 115, and a communication interface 116 are connected to the bus 100.

  The CPU 111 is an arbitrary processing circuit. The CPU 111 executes the program expanded in the RAM 112. As a program to be executed, a program for performing the processing of the embodiment can be applied. The ROM 113 is a non-volatile storage device that stores programs developed in the RAM 112.

  The auxiliary storage device 114 is a storage device that stores various information. For example, a hard disk drive, a semiconductor memory, or the like can be applied to the auxiliary storage device 114. The medium connection unit 115 is provided so as to be connectable to the portable recording medium 118.

  As the portable recording medium 118, a portable memory or an optical disk (for example, Compact Disk (CD), Digital Versatile Disk (DVD), etc.) can be applied. A program for processing performed by the image search server 4 of the embodiment may be recorded on the portable recording medium 118.

  Each unit other than the server communication unit 11 of the image search server 4 may be realized by the CPU 111. The server communication unit 11 may be realized by the communication interface 116. The RAM 112, the ROM 113, and the auxiliary storage device 114 are all examples of a tangible storage medium that can be read by a computer. These tangible storage media are not temporary media such as signal carriers.

<Example of hardware configuration of display terminal>
Next, an example of the hardware configuration of the display terminal 7 will be described with reference to FIG. As illustrated in the example of FIG. 22, a CPU 211, a RAM 212, a ROM 213, an auxiliary storage device 214, a medium connection unit 215, and a communication interface 216 are connected to the bus 200.

  The CPU 211, RAM 212, ROM 213, auxiliary storage device 214, and medium connection unit 215 are the same as those described above. The program executed by the CPU 211 may be a program that performs the processing of the embodiment. The communication interface 216 performs communication with the outside.

  The input / output interface 217 is an interface for inputting / outputting data to / from the display unit 25, for example. The terminal communication unit 21 may be realized by the communication interface 216. Of the display terminal 7, each unit other than the terminal communication unit 21 and the display unit 25 may be realized by the CPU 211.

  The RAM 212, the ROM 213, and the auxiliary storage device 214 are all examples of a tangible storage medium that can be read by a computer. These tangible storage media are not temporary media such as signal carriers.

<Others>
Therefore, in the embodiment, the image search server 4 generates composite image metadata that specifies a series of all-around images of a section including the subject designated by the display terminal 7, and the display terminal 7 generates an image based on the composite image metadata. The entire surrounding image is extracted from the accumulation server 3. Thereby, the display terminal 7 can display a series of images of the section Tseg including the subject with a simple operation.

  In particular, only by specifying a keyword on the display terminal 7, the display terminal 7 can display a series of images in the section Tseg including the specified subject as a moving image. Therefore, the user can view the moving image of the section including the subject with a simple operation. The same applies when still images with different shooting positions are displayed continuously.

  The field of view setting unit 23 sets the field of view of the series of images in the section Tseg so that the subject is always located at the center of the image based on the orientation information θ and the elevation angle information φ included in the composite image metadata. Can do. Thereby, for example, for a series of images in the section Tseg, the display terminal 7 can reproduce a moving image in which the subject is always located at the center.

  The azimuth information θ and the elevation angle information φ are examples of camera work information. The camera work information may be only the orientation information θ. In this case, the elevation angle is fixed. Therefore, the subject is not always located at the center of the image. When the display terminal 7 displays the subject by camera work based on the azimuth information θ and the elevation angle information φ, the display terminal 7 can display a moving image or a still image in which the subject is always located at the center.

  The composite image metadata includes visibility V. As shown in the example of FIG. 20, the user can selectively display an image with good visibility V by operating the slider bar 53.

  In addition, when the cameras C1 to C4 of a large number of vehicles 2 photograph the subject specified by the display terminal 7, the metadata server 5 stores a large amount of composite image metadata regarding the subject specified by the display terminal 7.

  For this reason, as shown in FIG. 14, the display terminal 7 can selectively extract a high-quality image with a high visibility goodness Q by displaying a selection screen to be selected from the list of the visibility goodness Q. . Thereby, the composite image metadata ID of the image (for example, moving image) to display can be narrowed down.

  In the embodiment described above, the example in which the visibility goodness Q is a value that evaluates the visibility V in an integrated manner has been described, but the visibility goodness Q may be a value that indicates the degree of visibility of the subject. Good. For example, among images captured by the cameras C1 to C4 (that is, all-around images), a value based on the distance from the center where the subject is captured in the image may be used as the visibility goodness Q.

  When the subject is shown in the center of the image, the degree of visibility of the subject increases. For this reason, the value of visibility goodness Q also becomes high. On the other hand, when the subject is separated from the center of the image, the degree of visibility of the subject is low, and the visibility goodness Q is also low.

  Therefore, when the user selects a composite video metadata ID having a high visual quality Q, the image extraction unit 22 can extract only the entire surrounding image in which the subject is captured at the center of the image.

  The present embodiment is not limited to the above-described embodiment, and various configurations or embodiments can be taken without departing from the gist of the present embodiment.

DESCRIPTION OF SYMBOLS 1 Image provision system 2 Vehicle 3 Image storage server 4 Image search server 5 Metadata server 6 Map server 11 Server communication part 12 Subject position specification part 13 Composite image metadata production | generation part 14 Computation part 21 Terminal communication part 22 Image extraction part 23 View field Setting unit 24 Image processing unit 25 Display unit 30 Control unit 33 Time measurement unit 34 Position information acquisition unit 35 In-vehicle image metadata generation unit 36 In-vehicle storage unit 111, 211 CPU
112, 212 RAM
113, 213 ROM

Claims (8)

An image providing system including a server and a display terminal,
The server
Based on the information about the object which the display terminal is specified, a specification unit for specifying a position of presence point of the object,
A generating unit that generates specific data for identifying a series of surrounding images of a section including the subject, based on a photographing position where the subject is photographed and a position of the existence point of the subject;
A transmission unit for transmitting the specific data to the display terminal;
With
The display terminal is
An extraction unit that extracts a series of surrounding images of a section including the subject based on the specific data from a storage device that stores a plurality of surrounding images;
A display unit for displaying an image of the subject in a series of surrounding images extracted by the extraction unit based on the specific data;
An image providing system comprising: The generating unit, the distance between the position of existence point shooting position and the object of the peripheral image when less than a predetermined distance, is identified as a peripheral image including the object,
The image providing system according to claim 1. It said specific data includes direction information indicating the orientation of the position of existence point of the object as viewed from the imaging position,
The display unit displays an image of a visual field based on the orientation information among the surrounding images.
The image providing system according to claim 1 or 2. The specific data further includes elevation angle information indicating an elevation angle of the position of the subject existing from the shooting position,
The display unit displays an image of a visual field based on the azimuth information and the elevation angle information among the surrounding images.
The image providing system according to claim 3. The specific data includes visibility indicating a rate of the subject in the surrounding image,
The display terminal displays a screen on which any one of the visibility of the plurality of specific data can be selected;
The image providing system according to any one of claims 1 to 4. The specific data includes visibility goodness indicating an integrated value of the visibility of each surrounding image of a series of surrounding images of a section including a subject specified by the display terminal,
The display terminal displays a screen on which any one of the visual goodnesses of the plurality of specific data can be selected;
The image providing system according to claim 5. The extraction unit extracts a series of surrounding images in which the subject is reflected in the center of the surrounding image among a series of surrounding images of a section including the plurality of subjects.
The image providing system according to any one of claims 1 to 6. An image providing method in which a server and a display terminal communicate with each other,
The server
Based on the information about the object which the display terminal is specified, to locate the presence point of the object,
Based on the shooting position at which the subject is shot and the position of the location of the subject , specific data for identifying a series of surrounding images of a section including the subject is generated,
Transmitting the specific data to the display terminal;
The display terminal is
From a storage device that stores a plurality of surrounding images, a series of surrounding images of a section including the subject is extracted based on the specific data,
On the basis of the specific data, and displays an image of the object among the extracted the series of surrounding images,
Image providing method.