JP7507946B2 - IMAGE PROCESSING APPARATUS AND IMAGE DISPLAY METHOD - Google Patents

Description

The present invention relates to an image processing device and an image display method for displaying an image on a display.

Advances in networks and information processing devices have made it easier to access necessary information regardless of time or place. For example, a map application allows you to immediately view information about facilities at a specified location on the map, as well as photographed images of that location, making it possible to understand the information in relation to the location information inherent in the map. In addition, by using panoramic (360°) photographed images to display images in any direction, or by sequentially switching the display to images taken from the viewpoint in the direction of travel, it is possible to create a sense of realism as if you are looking around the world of the image or walking through it.

To provide a stable environment like the one described above, it is desirable to prepare as many captured images as possible, regardless of location. For this reason, measures are sometimes taken in which users share images over a network, such as by soliciting image data from the general public or allowing individual users to associate images with locations on a map. However, as the number of images increases, it becomes more likely that images that show completely different aspects of the same location will be treated equally, confusing the viewer and requiring a great deal of effort to arrive at the desired image. In particular, when the displayed images are switched sequentially, as in the technology for shifting the viewpoint of a panoramic image described above, the difference in the appearance of the images before and after the switch can reduce the sense of realism or make it difficult to recognize the location as the same.

The present invention was made in consideration of these problems, and its purpose is to provide a technology that can display images corresponding to locations in an appropriate and efficient manner.

One aspect of the present invention relates to an image processing device. This image processing device includes an input information acquisition unit that accepts a user operation related to an image to be displayed on a display device, and an image display unit that displays on the display device one of a plurality of images taken in different circumstances at the same location in response to the user operation, and further displays a situation selection icon for selecting the shooting situation of the same location as the image being displayed, and when the input information acquisition unit accepts an operation on the situation selection icon, displays on the display device the image taken in the selected situation, and is characterized in that when a user operation is performed to continuously indicate within a range regarded as the same position as the position where the situation selection icon is displayed for a predetermined period of time or more, the image display unit displays a situation selection icon representing another option, or expands the area of the situation selection icon to increase the number of options displayed.

Another aspect of the present invention relates to an image display method, which includes the steps of an image processing device receiving a user operation related to an image to be displayed on a display device, displaying on the display device one of a plurality of images taken in different circumstances at the same location in response to the user operation, displaying a situation selection icon for selecting a shooting situation at the same location as the image being displayed, and receiving an operation on the situation selection icon and displaying on the display device an image taken in the selected situation, wherein the step of displaying the situation selection icon is characterized in that when a user operation is performed to continuously indicate within a range regarded as the same position as the position at which the situation selection icon is displayed for a predetermined period of time or more, a situation selection icon showing another option is displayed, or the area of the situation selection icon is expanded to increase the number of options shown.

In addition, any combination of the above components, and any conversion between the expression of the present invention as a method, device, system, computer program, recording medium on which a computer program is recorded, etc., are also valid aspects of the present invention.

The present invention makes it possible to display images corresponding to locations in an optimal and efficient manner.

FIG. 1 is a diagram showing a configuration example of an image display system to which the first embodiment can be applied. 1 is a diagram showing an internal circuit configuration of an image processing device according to a first embodiment; 2 is a diagram showing a configuration of functional blocks of an image processing device and an image providing server according to the first embodiment; FIG. 4 is a diagram showing an example of a functional block configuration of an image providing server when constructing an image database in the first embodiment. FIG. FIG. 4 is a diagram showing an example of a screen displayed on a display device in the first embodiment. FIG. 13 is a diagram showing an example of a screen when accepting selection of a shooting situation in the first embodiment. 4 is a diagram illustrating an example of a database structure stored in an image database in the first embodiment. FIG. 1 is a diagram illustrating an example of a structure of classification information of shooting conditions used in the first embodiment. FIG. 10 is a flowchart showing a processing procedure in which the image processing device displays a captured image corresponding to a location in cooperation with the image providing server in the first embodiment. 13 is a diagram illustrating an example of the structure of data stored in an image database in embodiment 2. FIG. 11 is a diagram for explaining data stored in an image database when a position in a hierarchy is further associated with the image data column in the image data information of FIG. 10. FIG. FIG. 13 is a diagram showing an example of an image displayed on a display device in the second embodiment; FIG. 13 is a diagram showing another manner for accepting an operation for selecting a floor number in the present embodiment. 13 is a flowchart showing a processing procedure in which an image processing device displays a photographed image corresponding to a location in cooperation with an image providing server in the second embodiment.

First embodiment
1 shows an example of the configuration of an image display system to which the present embodiment can be applied. The image display system 1 includes image processing devices 10a and 10b that display images in response to user operations, and an image providing server 20 that provides image data used for display. Input devices 14a and 14b for user operations and display devices 16a and 16b for displaying images are connected to the image processing devices 10a and 10b, respectively. The image processing devices 10a and 10b and the image providing server 20 establish communication via a wide area communication network such as the Internet, or a local network such as a LAN (Local Area Network). Alternatively, the image data may be provided from the image providing server 20 to the image processing devices 10a and 10b via a recording medium.

The image processing devices 10a, 10b may be connected to the display devices 16a, 16b and the input devices 14a, 14b either wired or wirelessly. Alternatively, two or more of these devices may be formed integrally. For example, in the illustrated example, the image processing device 10b forms a mobile terminal integrally with the input device 14b, which is a touchpad that covers the screen of the display device 16b. Furthermore, the display device 16a is not limited to a flat display, and may be a wearable display such as a head-mounted display, a projector, or the like.

As such, the external shape and connection form of the illustrated devices are not limited. The number of image processing devices 10a, 10b connected to the image providing server 20 is also not limited. Hereinafter, the image processing devices 10a, 10b will be collectively referred to as the image processing device 10, the input devices 14a, 14b as the input device 14, and the display devices 16a, 16b as the display device 16.

The image processing device 10 causes the display device 16 to display an image corresponding to a user operation on the input device 14. Specifically, at least among images taken at various locations, the image processing device 10 displays an image taken at a location specified by the user. The display purpose is not particularly limited here, and in order to achieve this purpose, the image processing device 10 may simultaneously carry out general processing such as progressing an electronic game or displaying images from an internet site using a browser. In this embodiment, an electronic map application, which has a high affinity as a means of specifying a location, will be described as an example.

The input device 14 may be any one or a combination of common input devices such as a controller, keyboard, mouse, touchpad, joystick, etc., and in this embodiment, in particular, it accepts from the user the specification of a location, movement of the viewpoint, change of line of sight, as well as the selection of the image capture conditions. The display device 16 may be a common display such as a liquid crystal display, plasma display, or organic EL display, and displays the image output from the image processing device 10. As described above, the display device 16 may also be a head-mounted display or a projector.

The image providing server 20 establishes communication with the image processing device 10 and transmits captured image data in response to a request from the image processing device 10. For this reason, the image providing server 20 holds a database that associates captured image data with the location and circumstances of the image capture. It then accepts designation of the location and circumstances of the image capture from the image processing device 10 and transmits data of the matching image. Here, "capture circumstances" may include, for example, the time circumstances of the image capture, such as the date and time of capture, time period, day of the week, month, season, year, and decade, as well as the subject circumstances, such as the name of an event taking place at the time of capture, the classification of the event, the number of people in the image, and the weather. However, specific items are not particularly limited as long as they are factors that change the appearance or impression of the location.

Figure 2 shows the internal circuit configuration of the image processing device 10. The image processing device 10 includes a CPU (Central Processing Unit) 23, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are connected to each other via a bus 30. An input/output interface 28 is also connected to the bus 30. The input/output interface 28 is connected to a communication unit 32 that is made up of a peripheral device interface such as USB or IEEE 1394, or a network interface for a wired or wireless LAN, and that establishes communication with the image providing server 20, a storage unit 34 such as a hard disk drive or non-volatile memory, an output unit 36 that outputs data to the display device 16, an input unit 38 that inputs data from the input device 14, and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory.

The CPU 23 controls the entire image processing device 10 by executing an operating system stored in the storage unit 34. The CPU 23 also executes various programs that are read from a removable recording medium and loaded into the main memory 26, or downloaded via the communication unit 32. The GPU 24 has the functions of a geometry engine and a rendering processor, performs drawing processing according to drawing commands from the CPU 23, and outputs to the output unit 36. The main memory 26 is composed of RAM (Random Access Memory), and stores programs and data required for processing. The internal circuit configuration of the image providing server 20 may be similar.

Figure 3 shows the functional block configuration of the image processing device 10 and the image providing server 20. Each functional block shown in this figure and in Figure 4 described later can be realized in hardware terms by the various circuits shown in Figure 2, and in software terms by programs that are loaded from a recording medium into main memory and perform various functions such as image analysis function, information processing function, image drawing function, and data input/output function. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination of these, and are not limited to any one of them.

The image processing device 10 includes an input information acquisition unit 50 that acquires information related to user operations, a data request unit 52 that requests data corresponding to the user operations from the image providing server 20, a data acquisition unit 54 that acquires data transmitted from the image providing server 20, and an image display unit 56 that generates a display image and displays it on the display device 16. The input information acquisition unit 50 is realized by the input unit 38 in FIG. 2, the CPU 23, etc., and acquires information related to user operations from the input device 14. Specifically, as described above, in addition to designating the location where the image was taken and the shooting conditions, it acquires operation contents such as starting and ending the display process, changing the line of sight to the panoramic image, and moving the viewpoint. When associated with a map application, the input information acquisition unit 50 also acquires the contents of various operations related to map display.

The data request unit 52 is realized by the CPU 23, communication unit 32, and the like in FIG. 2, and requests image data corresponding to a user operation from the image providing server 20. The data acquisition unit 54 is realized by the CPU 23, communication unit 32, main memory 26, and the like in FIG. 2, and acquires image data transmitted in response to a request from the image providing server 20. The data acquisition unit 54 may include a recording medium drive unit 40, and may read image data from a recording medium. The data acquisition unit 70 may also acquire option information from the image providing server 20 indicating what kind of image shooting conditions can be selected for a specified location. The data acquisition unit 70 decodes and expands the acquired image data as necessary.

The image display unit 56 is realized by the CPU 23, GPU 24, main memory 26, etc. in FIG. 2, generates a display image including the image acquired by the data acquisition unit 54, and outputs it to the display device 16. In detail, the image display unit 56 includes an icon display unit 64, an image switching unit 66, and a viewpoint change processing unit 68. The icon display unit 64 displays at least a situation selection icon for selecting the shooting situation of the image to be displayed. By superimposing the situation selection icon on the image currently being displayed, it is possible to switch the display to an image of a different shooting situation while keeping the image displayed. Note that since the shooting situations that can be selected vary depending on the images prepared for the specified location, the icon display unit 64 determines the configuration of the situation selection icon based on the option information transmitted from the image providing server 20 as described above.

In addition, the icon display unit 64 displays a movement icon for moving the viewpoint in the panoramic image display, superimposed on the image being displayed as necessary. The image switching unit 66 switches the display object to at least an image of the shooting situation selected by operating the situation selection icon. The image switching unit 66 also appropriately performs operations such as displaying thumbnail images, enlarging an image selected from the displayed thumbnail images, and displaying a map or information of a searched location. In the panoramic image display, the viewpoint change processing unit 68 switches the displayed image in response to an operation on the movement icon so that the viewpoint moves in a corresponding direction in the image world, and changes the field of view of the displayed object in response to an operation to specify a direction up, down, left, right, or diagonally on the screen so that the corresponding direction is visible.

General techniques can be used for the control of display in response to changes in viewpoint or line of sight implemented by the viewpoint change processing unit 68. However, in this embodiment, when switching images by moving the viewpoint, an image with the same shooting conditions as before the switch is preferentially selected and displayed. Alternatively, the user can select the shooting conditions on the spot. Note that if the display target is not a panoramic image, the function of the viewpoint change processing unit 68 can be omitted.

The image providing server 20 includes a request acquisition unit 58 that acquires a request for image data from the image processing device 10, a data transmission unit 62 that transmits image data corresponding to the request to the image processing device 10, and an image database 60 that stores the image data in association with the shooting location and shooting conditions. The request acquisition unit 58 is realized by the communication unit 32 and CPU 23 in FIG. 2, and acquires a data request for an image specifying the location and shooting conditions from the image processing device 10.

The data transmission unit 62 is realized by the CPU 23, the communication unit 32, etc., and extracts image data and option information corresponding to the request from the image database 60 and transmits it to the image processing device 10. When realizing a map application, the target of the request from the image processing device 10 and the target of the transmission from the image providing server 20 may include data that is generally provided as a map application, such as map images and facility information. In this case, a database for such data is prepared separately, but is not shown in FIG. 3.

The image database 60 is realized by the storage unit 34 in FIG. 2, and stores data on images taken at various locations in association with the location and circumstances of the image. As described above, the circumstances of the image can be considered to be the time circumstances and the circumstances of the subject. As options for selecting the former, time ranges of various granularities are prepared, such as the time of day when the image was taken, the month or season when it was taken, and the year when it was taken. As options for selecting the latter, various granular classifications and number ranges are prepared for events that were held, the number of people, the weather, etc.

Figure 4 shows an example of the functional block configuration of the image providing server 20 when constructing the image database 60. The functional blocks of the illustrated image providing server 20a may be included in the image providing server 20 of Figure 3, or may be in a different device. The image providing server 20a includes a data acquisition unit 70 that acquires captured image data from an external device, an attribute acquisition unit 72 that acquires attributes of the captured image such as the shooting location and shooting conditions, and the image database 60 shown in Figure 3.

The data acquisition unit 70 is realized by the CPU 23, communication unit 32, input unit 38, etc. in FIG. 2, and acquires captured image data from an external device such as the image processing device 10. For example, a general user can specify a location on a map for a map application running on the image processing device 10 and upload image data that he or she has taken at that location. In this case, the data acquisition unit 70 acquires the uploaded data together with information about the specified location via the network. The image data may also be acquired from an imaging device or information processing device directly connected to the image providing server 20a, or from another image providing server connected via a network.

The attribute acquisition unit 72 acquires information related to the location and conditions of the image by a specified means. In the case of an image taken with a digital camera, the date and time of shooting, location information at the time of shooting, etc. are recorded as additional information such as Exif (Exchangeable Image File Format). Therefore, the attribute acquisition unit 72 reads the additional information from the image data acquired by the data acquisition unit 70 and acquires the necessary information. Alternatively, the attribute acquisition unit 72 may acquire information related to the attributes input by the user who provided the image in the image processing device 10. Alternatively, the attribute acquisition unit 72 may acquire information such as the number of people in the image and the weather by analyzing the captured image using a general method. Such information may be directly confirmed and input by the user operating the image providing server 20a.

The attribute acquisition unit 72 associates the information related to the shooting location and shooting conditions acquired in this manner with the captured image data and stores them in the image database 60. By performing such association on data of images that do not have uniformity, such as those collected from general users, and storing the data in a database, it is possible to easily extract from the abundant captured images the ones that are most suitable for the viewing conditions.

For example, if a panoramic image of a certain location is displayed and an attempt is made to move in the direction of travel in the image world by moving the viewpoint, if there is insufficient image data to begin with and there is no image captured from the corresponding viewpoint, it will be impossible to switch the display. On the other hand, even if there are plenty of images captured from the relevant viewpoint, if one of them is displayed out of context, even a slight movement is likely to cause a significant change in the appearance of the displayed image, leading to discomfort and confusion.

As described above, by creating an image database 60 in association with attributes other than location, and extracting images with the same shooting conditions in response to the operation of moving the viewpoint, it is possible to continue displaying images with a consistent feel, allowing the user to appreciate and confirm the atmosphere and structure of the location without confusion. Also, by allowing the user to select the shooting conditions, it is possible to continue presenting images that suit the individual purpose of the user. Note that when image data is provided via a recording medium, it is stored in the recording medium in the form of the image database 60.

Figure 5 shows an example of a screen displayed on the display device 16 in this embodiment. Note that the changes in the display screen described below are actually made through a processing procedure in which the input information acquisition unit 50 of the image processing device 10 acquires the contents of a user operation, the data request unit 52 makes a request to the image providing server 20, and the data acquisition unit 54 acquires the corresponding image data; however, a description of this will be omitted where appropriate.

(a) shows a screen in which an electronic map is displayed by a map application. In this example, the display screen includes a map field 80, a search field 82, and an information field 84. When the user enters an address or facility name in the search field 82, the image switching unit 66 of the image display unit 56 displays a map of the surrounding area in the map field 80. When a facility name is entered, or when a facility is designated by operating the cursor 88 on an already displayed map, the image switching unit 66 displays text information such as basic information about the facility and the URL of its homepage in the information field 84.

In the illustrated example, the facility name "City Center" is entered into the search field 82, and a map of the surrounding area is displayed in the map field 80, with basic information displayed in the information field 84. Furthermore, the image switching unit 66 displays thumbnails 86a, 86b of images taken near the facility. The images displayed as thumbnails 86a, 86b are extracted from image data stored in the image database 60 using the location of the facility as a key. When the user designates one of these images with the cursor 88 or the like, the screen transitions to the captured image display screen of (b). In this example, the captured image display screen includes an image field 90 and a thumbnail field 92.

The image switching unit 66 displays an enlarged image of the thumbnail 86a selected by the user on the map display screen (a) in the image column 90, and displays thumbnails of other images taken at the same location in the thumbnail column 92. Note that the "same location" here is not limited to the scale of the area, and can include the same store or room, as well as a building, facility, or site. When one of the thumbnails displayed in the thumbnail column 92 is selected, the image switching unit 66 switches the display in the image column 90 to an enlarged image of that thumbnail.

When a panoramic image is displayed in the image field 90, the icon display section 64 of the image display section 56 displays a movement icon 94 for moving the viewpoint on the image. When the movement icon 94 is pointed to by a cursor or the like, the viewpoint change processing section 68 moves the viewpoint in the direction of the arrow indicated by the icon (toward the back in the figure), and switches the display to the panoramic image captured at that point. When an arbitrary position on the screen is pointed to and slid, the viewpoint change processing section 68 moves the field of view in the opposite direction so that the image moves in conjunction with the slide. This allows the user to view the specified location from any viewpoint or line of sight. The screen configuration is not limited to that shown in FIG. 5, and the processing procedure for displaying the captured image is not limited.

In an environment in which a location is selected and images taken there are displayed in this way, in addition to appreciating the images themselves, the purpose is often to understand the state of the location in relation to facilities and locations. However, as mentioned above, the more images taken of the location, the more diverse the images that can be viewed, but the more difficult it becomes to access the desired image. Furthermore, when moving the viewpoint using the movement icon 94, if the state of the image changes significantly before and after the movement, it is thought that the user will not get a sense of movement and will have difficulty understanding the space.

As described above, therefore, images taken in a situation suited to the user's purpose are extracted and displayed. Figure 6 shows an example of a screen when the selection of a shooting situation is accepted. The image 100 shown in (a) may be either a thumbnail displayed on the map display screen in (a) or the shot image display screen in (b) of Figure 5, or an enlarged image displayed on the shot image display screen in (b). Alternatively, it may be a shot image displayed on a homepage or the like unrelated to a map. When such an image 100 is displayed and the user performs a specified operation on the image 100, the icon display unit 64 displays a situation selection icon 102 representing the shooting situation options superimposed on the image 100.

In general image display technology, it is often set up so that a displayed image 100 can be selected and some action can be initiated by pointing at the image with a cursor or the like. For example, in the example of FIG. 5, as described above, pointing to thumbnail 86a causes an enlarged image to be displayed in image field 90. Alternatively, pointing to an image displayed in image field 90 can be used to move the viewpoint or change the line of sight.

To be compatible with such general operations, the icon display unit 64 displays the situation selection icon 102 at the indicated position on the condition that an indication within a predetermined range that can be regarded as the same position on the image 100 continues for a time equal to or longer than a predetermined threshold value t1. Here, "indication" may refer to an operation of a displayed cursor or a touch operation on a touch pad. In the former case, the duration of the indication corresponds to, for example, the time that a button on a controller or mouse is pressed. In the latter case, the duration of the indication corresponds to the duration of contact. These operations are generally called "long press." However, the operation for calling up the situation selection icon 102 is not limited to a long press, and for example, another GUI (Graphical User Interface) may be displayed and the situation selection icon 102 may be displayed in response to an operation on that GUI.

As an example, the illustrated situation selection icon 102 is a circular icon with an area assigned to each of the shooting situation options, which are day and night ("day", "night") and the four seasons ("spring", "summer", "autumn", "winter"). In this case, for example, when a long press is determined, the situation selection icon 102 is displayed with that position as the center, and if the designated position is then slid to one of the areas, it is determined that the option in the area including the reached point has been selected. Also, if the designated position does not move even when the situation selection icon 102 is displayed and this state continues for a time equal to or greater than a predetermined threshold value t2, a situation selection icon 102 representing a different option may be displayed.

Instead of displaying a new situation selection icon 102, the icon may be expanded depending on the duration of the instruction, increasing the number of options. For example, when the situation selection icon 102 is first displayed, only the options located in the central area (the options "day" and "night" in the illustrated example) are displayed. If the instruction continues for a time equal to or greater than threshold value t2 without the instruction position moving even after the option is displayed, the options in the outer area (the options "spring", "summer", "autumn", and "winter" in the illustrated example) may appear. Then, when one of the shooting situations is selected, the image switching unit 66 switches the display to an image taken at the same location in the selected shooting situation, as shown in (b).

For example, by switching from an image taken at night to an image taken during the day, the number of people in the subject increases as shown in the figure. Of the shooting situation options shown by the situation selection icon 102, options for which no image data exists are indicated as unselectable by being displayed in a different color from the other options. The switched image may be displayed in the same position as the image before the switch, or in a different position depending on the purpose. For example, if the image 100 is the thumbnail 86a shown in the information column 84 on the map display screen in FIG. 5(a), the image of the selected shooting situation may be enlarged and displayed in the image column 90 on the captured image display screen in (b).

In this case, when the user moves the viewpoint by operating the movement icon 94, all of the images to be switched and displayed by the viewpoint change processing unit 68 may be those of the shooting situation initially selected. This unifies the shooting situations of the images that are switched sequentially in response to the movement, and as described above, it is possible to avoid the situation where a slight movement causes a completely different image to be displayed, which may cause confusion. On the other hand, even if the image 100 in FIG. 6 is an image that is enlarged and displayed in the image field 90, by calling the situation selection icon 102 and switching the display to the image of the selected shooting situation, it is possible to easily check what the same place looks like in a different situation.

For example, you can see how the same location changes between day and night or depending on the season, or see what the location was like in the past. You can also see what an event you plan to attend looked like the last time. By displaying the situation selection icon 102 on an image, you can switch images directly, eliminating the hassle and distraction of first displaying another image such as a search screen or thumbnail. You can also easily switch and compare images even with a limited display area.

Figure 7 illustrates an example of the database structure stored in the image database 60. In this example, the image data information 110 includes a location column 112, an address column 114, a facility ID column 116, an image data column 118, a shooting date and time column 120, and an event column 122. The location column 112 indicates location information such as the latitude and longitude of the shooting location, the address column 114 indicates the address of the shooting location, and the facility ID column 116 indicates identification information such as the name of the facility at the shooting location. The image data column 118 indicates the data of images taken at the location or facility or their identification information. The shooting date and time column 120 indicates the date and time the image was taken, and the event column 122 indicates the name of the event being held at the location at the time of the image being taken.

That is, the image data information 110 associates each image data shown in the image data column 118 with the "location information" described in the position column 112, address column 114, and facility ID column 116, as well as with the "shooting conditions" described in the shooting date and time column 120 and event column 122. However, it is not intended to limit the items of "location information" and "shooting conditions" or the illustrated example of the data structure of the image data information 110. Here, the shooting conditions indicated by the image data information 110 are information specific to the image data, and are obtained by Exif or manual input. In order to efficiently narrow down this using the situation selection icon 102 in Figure 6, shooting date and time and event classification information are prepared.

Figure 8 illustrates an example of the structure of the classification information of shooting conditions used in this embodiment. In this example, classification information 130 includes a location column 132, an address column 134, a facility ID column 136, a time period column 138, a season column 140, and an event classification column 142. The location column 132, the address column 134, and the facility ID column 136 indicate the same information as the location column 112, the address column 114, and the facility ID column 116 shown in Figure 7. The time period column 138, the season column 140, and the event classification column 142 respectively indicate the time period, season, and event classification that can be presented as options for the location.

For example, for the "City Center" shown in the figure, there are images taken during the day and at night, while there are images for the seasons "spring," "autumn," and "winter," but no images for "summer." There are also images of events categorized as "exhibitions" and "concerts." However, the classification items for the shooting conditions are not limited to these. By preparing classifications of various granularities for each of the time conditions and subject conditions, it is possible to efficiently narrow down the images. The classification information 130 is stored in the image database 60 together with the image data information 110 shown in FIG. 7.

Each time a new captured image is acquired, the attribute acquisition unit 72 of the image providing server 20 registers the data in the image data information 110 in FIG. 7 and updates the classification information 130 as necessary based on the attributes. When viewing an image, the icon display unit 64 of the image processing device 10 acquires the classification information 130 of the specified location from the image providing server 20 and displays the selectable shooting situation classifications as options on the situation selection icon 102.

When the user selects one of the options, the image switching unit 66 retrieves from the image providing server 20 the image data associated with the location in the image data information 110 that corresponds to the selected shooting conditions, and displays it. If there are multiple corresponding images, one of them may be displayed in an enlarged manner, and the other images may be displayed as thumbnails. Note that the image data information 110 shown in FIG. 7 and the classification information 130 shown in FIG. 8 may actually be a single table, or image data may be associated with each classification.

Next, the image display processing procedure of this embodiment that can be realized by the above configuration is shown. FIG. 9 is a flowchart showing the processing procedure in which the image processing device 10 displays a captured image corresponding to a location in cooperation with the image providing server 20. This flowchart assumes a mode in which the captured image is displayed as part of a map application, but as described above, the processing performed prior to image display is not limited to map display. Furthermore, the image displayed by the image display unit 56 is basically the image data requested by the data request unit 52 and transmitted from the image providing server 20 and acquired by the data acquisition unit 54, but this procedure is omitted.

First, when a map application is started by a user operation on the image processing device 10, the image switching unit 66 of the image display unit 56 displays an image of an electronic map on the display device 16 (S10). Until the user inputs the name of the facility they want to search for or indicates a location on the map, the system waits while performing a process (not shown) corresponding to the operation on the map (N in S12). Once a location has been specified (Y in S12), the image switching unit 66 displays a thumbnail of an image taken of the location together with information about the location, as shown in FIG. 5(a) (S14).

The image displayed here may be one selected according to a predetermined rule. In the example of the map display screen in FIG. 5(a), the thumbnail 86a serves as a GUI for switching the main display from the map to the captured image, so in some cases a GUI for switching may be displayed instead of the thumbnail. If no operation is performed on the displayed thumbnail image or GUI, the image display unit 56 waits while performing a process (not shown) in response to the operation on the map (N in S16).

If an operation is performed on the thumbnail image or GUI (Y in S16) and it is not a long press operation as described above (Y in S18), the image switching unit 66 executes the display set for that operation, such as enlarging and displaying the operated thumbnail in the image field 90 shown in FIG. 5B (S20). If the GUI is operated, an image selected according to a predetermined rule may be enlarged and displayed. If the operation on the thumbnail image or GUI is a long press (N in S18), the icon display unit 64 displays a situation selection icon 102 for selecting a shooting situation at the specified position, as shown in FIG. 6A (S22). Then, if a certain shooting situation is selected by sliding the specified position, the image switching unit 66 displays an image corresponding to that shooting situation in the image field 90 shown in FIG. 5B (S24).

If there is no need to end the display due to user operation, etc., the processes of S16 to S24 are repeated (N in S26). However, from the second round onwards, the operation target in S16 is not limited to the thumbnail image, and it may be the image enlarged and displayed in the image column 90. Furthermore, the process performed in S20 may be an enlarged display of the thumbnail by the image switching unit 66, or a process corresponding to a change in viewpoint or line of sight by the viewpoint change processing unit 68. Here, when switching the display to an image to which the viewpoint has been moved, an image with the same shooting conditions as the one most recently selected is selected and displayed. If it becomes necessary to end the display, all processes are terminated (Y in S26).

According to the present embodiment described above, when or while an image taken at a location specified by the user is displayed, a situation selection icon is displayed to select the shooting situation by a specific operation. Images taken in the situation selected by the icon operation are then displayed. This makes it possible to efficiently present images of the desired state even at the same location, so that inconveniences due to a large number of images are unlikely to occur even if images are collected without setting any policy or restrictions. As a result, the user can conveniently view images taken at any location and check the state of the location.

In addition, because the image itself is switched by the icon displayed on the image, it is not distracting and it is easy to compare images in an enlarged state, compared to displaying a separate selection menu or rearranging thumbnail images. Furthermore, in a mode in which the panoramic image allows the image of a place to be viewed from any viewpoint or line of sight, it is possible to continue to display a consistent image as the viewpoint moves, so the sense of realism is less likely to be lost and the surroundings can be confirmed without any sense of discomfort.

Embodiment 2
The first embodiment is a technology for easily selecting and unifying the shooting conditions of images taken at the same location. In the present embodiment, images of different floors among images taken at the same address (same point on the ground) can be distinguished and selected. Maps are generally represented as a bird's-eye view of the ground so that the horizontal positional relationship of each facility and road can be seen. Therefore, it is difficult to understand the number of floors of a building and the state of each floor from the map alone. Although maps by floor level are sometimes prepared, they are limited to large-scale facilities where there is a need for them.

As shown in the first embodiment, when the captured images are displayed based on the map, images associated with horizontal positions such as latitude and longitude and the name of the entire building are detected at the same time, and images of various floors are displayed in a mixed state, which requires a great deal of effort to reach the image of the desired floor. In particular, when creating a sense of realism with a panoramic image, it can feel strange and confusing as the user moves from one floor to a completely different floor by switching images. Therefore, by classifying the captured images in association with the floors of the building, the display can be managed by floor and the user can be allowed to move to different floors of the same building.

The system configuration in this embodiment may be the same as the image display system 1 shown in FIG. 1, and the configurations of the image processing device 10 and image providing server 20 may be the same as those shown in FIGS. 2 to 4. FIG. 10 illustrates an example of the structure of data stored in the image database 60 in this embodiment. In this example, image data information 150 includes a location column 152, an address column 154, a facility ID column 156, a floor column 158, and an image data column 160. The location column 152, the address column 154, and the facility ID column 156 show the same information as the location column 112, the address column 114, and the facility ID column 116 shown in FIG. 7. The floor column 158 indicates the floor number of the building at the corresponding location, and the image data column 160 indicates the data of the image taken on that floor or its identification information.

In this case, the attribute acquisition unit 72 shown in FIG. 4 may determine which floor the image was taken on from information such as altitude included in the additional information of the acquired image data, or may have the user who provided the image input the floor on which the image was taken, or may determine the floor from information such as the store name associated with the image by the user. The attribute acquisition unit 72 may also determine the total number of floors of the building shown in the facility ID column 156 from official information about the building, and associate the information separately. The image data information 150 shown in the figure may be integrated with the image data information 110 shown in FIG. 7 to form a single database. In this case, the first and second embodiments can be realized simultaneously.

Figure 11 is a diagram for explaining the data stored in the image database 60 when the image data column 160 in the image data information 150 in Figure 10 is further associated with a position in the hierarchy. A schematic diagram of the floors of a building is shown on the left side of the figure. In this example, the same positions (same horizontal positions on the floor) a, b, c, d, e, and f are defined on each of the first to third floors. Of these, black circles indicate positions where there is image data, and white circles indicate positions where there is no image data.

For example, there is no image data at positions c, e, and f on the second floor because there is no floor due to the atrium structure. There may be cases where there is a floor but no image data, such as position a on the third floor. When managing image data at defined positions, table 162 shown on the right side of Figure 11 is associated instead of image data column 160 shown in Figure 10. Table 162 indicates the presence or absence of image data for each level and position. Also, in this example, when image data is present, its identification information is directly associated, for example, "001", "002", ..., but the linking with the image data may be performed in a separate table.

By managing image data for each floor position in this way, when an operation to move from the displayed floor to another floor is performed, the image can be switched to the same position on the selected floor. For example, when an image of position a on the first floor is displayed, if an operation to move to the second floor is performed, the display will switch to the image of position a on the second floor. In this way, it is easy to know which position on the selected floor the switched image corresponds to, and there is no confusion when moving thereafter.

Figure 12 shows examples of images displayed on the display device 16 in this embodiment. (a) shows a state in which a captured image of a certain floor in a building is displayed. These images are displayed in the image field 90 of the captured image display screen in (b) of Figure 5, for example, when an electronic map is displayed using a map application and a specific building is specified by searching or cursor operation, as shown in Figure 5. However, the images are not limited to those switched from the map, as long as they are the result of specifying a location on the ground, such as a building, address, latitude and longitude, etc.

The image 170 on the left side of (a) is part of the panoramic image, and as in the case of (b) of FIG. 5, a movement icon 94 for moving the viewpoint is displayed. That is, when this is pointed to, the viewpoint moves in the direction of the arrow (towards the back in the figure), and the panoramic image beyond that point is displayed. Furthermore, in this embodiment, when the movement icon 94 is pressed and held down, a floor selection icon 172 for selecting a floor is displayed. As an example, the floor selection icon 172 shown in the figure is a rectangular icon with areas assigned to each of the options from the second basement floor ("B2F") to the third floor ("3F").

The icon display unit 64 determines the configuration of the floor selection icon 172 by obtaining information on the total number of floors of the building and information on the floors for which images exist and are selectable from the image providing server 20. For example, an area for all floors is provided on the floor selection icon 172, and the areas for floors that cannot be selected are made different from the others to indicate that they cannot be selected. As shown in the figure, the currently displayed image, in this example the first floor ("1F"), is also made a different color from the other floors on the floor selection icon 172, making it unselectable.

When managing image data by position as shown in FIG. 11, floors for which there is no image data at the same position as the displayed position may be included in the hierarchical levels that cannot be selected. Alternatively, for floors for which image data exists at other positions, a different display color may be used or text may be used to indicate this, making them selectable with the user's consent. The image providing server 20 refers to the table 162 of the image database 60 and transmits information regarding the selectability of such positions to the image processing device 10. The user selects the desired floor by selecting one of the options from the floor selection icons 172. However, the shape and configuration of the floor selection icon 172 are not intended to be limited to this. Furthermore, the operation for displaying the floor selection icon 172 is not limited.

Image 174 on the right side of (a) assumes a case in which a touchpad capable of multi-touch operation is provided as the input device 14 to cover the screen of the display device 16. In this case, the viewpoint change processing unit 68 of the image display unit 56 changes the direction of the line of sight with respect to the panoramic image in response to a swipe operation, for example, of sliding a finger that has been in contact with the screen with a single touch (one contact point). As with image 170, a movement icon 94 may be displayed, and the viewpoint may be advanced in response to a touch operation thereon.

Furthermore, in this embodiment, in response to a swipe operation in which a finger touching the screen with multi-touch (multiple contact points) is swiped in a direction that can be considered as up or down on the screen, the display switches to the next floor up or down that corresponds to that direction. In the illustrated example, as shown by the arrow, an upward swipe switches the display to the image of the next floor down. This allows for an intuitive operation that feels like moving the displayed floor up. By distinguishing between viewpoint or line of sight operations and floor change operations based on the number of contact points, it becomes possible to easily achieve both.

(b) shows an image during a transition period when the display is switched to a lower floor by the above-mentioned operation on image 170 or image 174. In this example, an image obtained by combining pre-switch image 176 and post-switch image 178 is shown as the transition period. Here, the images of each floor are arranged so as to correspond to the hierarchical relationship of the floors of the building. In other words, if the post-switch floor is lower than the pre-switch floor, the post-switch image is arranged below, and if the post-switch floor is higher, the post-switch image is arranged above. Then, by moving the boundary between the two so that the area of the post-switch image gradually increases, the post-switch image appears to appear from the direction corresponding to the floor.

For example, both images can be alpha blended together, and the area in which the post-switching image 178 is transparent (α=0) can be narrowed vertically. By appropriately controlling the α value, the boundary between the two images can be composited semi-transparently. However, the way in which the images are displayed during the transition period is not limited to this; for example, the images before and after the switch can be slid up or down, or both can be enlarged or reduced vertically. In any case, by making the image of the post-switching floor appear from a direction corresponding to the floor of the building, the user can get the feeling that he or she is moving to a different floor of the building.

After the transition period shown in (b), the image captured at the switched floor is displayed as shown in (c). If multiple images are associated with the switched floor or the same position on that floor, access to an image of a desired floor and desired situation can be facilitated by allowing the user to select the shooting situation as in embodiment 1. In addition, if the image being displayed includes an elevator, and the image of the switched floor also includes the same elevator, the user can easily determine the direction of movement of the viewpoint thereafter, as if they were actually traveling in an elevator.

Therefore, if the image being displayed includes an elevator, when the elevator image is pressed and held instead of the movement icon 94 shown in image 170, a floor selection icon 172 may be displayed. In this case, the input information acquisition unit 50 recognizes that the area of the elevator image has been pressed and held, and the image switching unit 66 determines the field of view so that the elevator image is included in the image after switching. In cases where the same equipment, such as not only elevators but also escalators, stairs, and toilets, is located in the same position on multiple floors, displaying an image of the corresponding equipment even after switching makes it easier to understand the location.

In this case, the equipment that accepts the display operation of the floor selection icon 172 is set in advance, and when any of the equipment is included in the displayed image, the input information acquisition unit 50 detects a pointing operation on the area of that image. The operation here may be a long press or any other specified operation. When switching the image to the floor selected by the floor selection icon, the image switching unit 66 determines the field of view so that the image of the same equipment is included. Note that while the illustrated example shows an image taken inside a building, the display target of this embodiment is not limited to this, and any image that can be associated with height data that is difficult to show on a two-dimensional map may be an image of the outside of a building or land with elevation differences.

Figure 13 shows another mode for accepting an operation to select a floor. In this example, the display image 180 shows the interior of an open-ceiling building. That is, floors other than the currently displayed floor (the floor that accepts a horizontal viewpoint movement operation) are also included in the field of view. In this case, it is more efficient and easier to understand if the destination floor can be directly selected. On the other hand, to avoid confusion with a horizontal viewpoint movement operation on the currently displayed floor, moving to another floor is also performed by a long press operation in this case. In the example shown, the third floor is shown to be long pressed with a finger, and when the input information acquisition unit 50 detects this operation, the image switching unit 66 switches the display to the third floor.

Because the destination was directly pressed and held, the image after switching in this case is preferably an image of the selected floor near the position pressed and held. This allows the line of sight to be moved freely within the building, including up and down. This is not limited to an atrium, but can be realized in a similar manner as long as a part of another floor appears in the image of the displayed floor, such as near an escalator or staircase. The image transition may be the same as that shown in FIG. 12, or the viewpoint may be gradually moved to the indicated position. Note that the operation of moving to another floor is not limited to a long press, as long as it can be distinguished from a horizontal movement operation on the same floor. For example, multi-touch or double tap may be used. This embodiment may be combined with an embodiment that displays an icon for selecting a floor, or may be realized independently.

Next, the image display processing procedure of this embodiment that can be realized by the above configuration is shown. FIG. 14 is a flowchart showing the processing procedure in which the image processing device 10 displays a captured image corresponding to a location in cooperation with the image providing server. This flowchart assumes a mode in which the captured image is displayed as part of a map application, but as described above, the processing performed prior to image display is not limited to map display. Furthermore, the image displayed by the image display unit 56 is basically the image data requested by the data request unit 52 and transmitted from the image providing server 20 and acquired by the data acquisition unit 54, but this procedure is omitted.

First, when a map application is started by a user operation on the image processing device 10, the image switching unit 66 of the image display unit 56 displays an image of an electronic map on the display device 16 (S30). Until the user inputs the name of the facility they want to search for or indicates a location on the map, the system waits while performing a process (not shown) corresponding to the operation on the map (N in S32). Once the location has been specified (Y in S32), the image switching unit 66 enlarges and displays an image selected according to a predetermined rule from among images of buildings at that location (S34), as shown in FIG. 5(b) .

At this time, as in the first embodiment, thumbnails may be displayed and the user may select an image to be enlarged, or the image providing server 20 may select from a database according to a predetermined rule. The icon display unit 64 then displays a viewpoint shifting icon for shifting the viewpoint superimposed on the captured image (S36). If no operation is performed on the displayed viewpoint shifting icon, the image display unit 56 waits while performing a process (not shown) in response to a change of line of sight or the selection of another image (N in S38). If an operation is performed on the viewpoint shifting icon (Y in S38) and this is not a long press operation (Y in S40), the viewpoint change processing unit 68 switches the display to the image to which the viewpoint has been shifted (S42).

If the operation on the viewpoint movement icon is a long press (N in S40), the icon display unit 64 displays the floor selection icon 172, as in image 170 in FIG. 12 (S44). Then, when a floor is selected from the floor selection icon 172, the image switching unit 66 displays an image of the selected floor (S46). At this time, as shown in FIG. 12(b), a transition period occurs in which the switched image appears from a direction corresponding to the hierarchical relationship of the floors, and then the switched floor image as shown in (c) is displayed. Also, if image data is managed by position, the image is displayed in the same position as before the switch.

If a multi-touch compatible touchpad is introduced, instead of the processes of S38, S40, and S44, a multi-touch swipe operation displays an image of the next floor in the opposite direction to the sliding direction. Also, when a long press operation on the image of a specific facility is accepted, instead of determining a long press operation on the viewpoint movement icon in S38 and S40, a pointing operation on the image of that facility is determined. Then, in S46, an image of the same facility is included in the display.

Furthermore, if a part of another floor appears in the image displayed in S34, instead of judging whether the icon for moving the viewpoint is pressed and held in S38 and S40, the system judges whether a pointing operation is performed on the image of the other floor. In this case, the process of displaying the icon for selecting the floor in S44 is omitted, and in S46, the image of the corresponding position in the image of the specified other floor is displayed.

If there is no need to end the display due to a user operation or the like, the processes of S36 to S46 are repeated (N at S48). At this time, by limiting the image after switching in the viewpoint movement operation of S42 and the candidate images displayed as thumbnails to the image of the most recently selected floor, it is possible to prevent an image of another floor from being displayed out of context. If it becomes necessary to end the display, all processes are terminated (Y at S48).

According to the present embodiment described above, while a captured image of a location specified by the user is being displayed, a floor selection icon for selecting a floor number in the building is displayed by a specific operation. The display is then switched to the image of the floor selected by operating the icon. This allows images corresponding to the same location on a two-dimensional map to be displayed separately for each floor, allowing the user to check the situation and understand the structure by narrowing down to the floor that interests them or that they are interested in.

In a mode in which the image of a place can be viewed from any viewpoint or line of sight using a panoramic image, it is possible to transition between floors with the same operational feeling as moving the viewpoint or changing the line of sight. Furthermore, by providing a transition period in which the image of the selected floor appears from a direction that follows the hierarchical relationship of the floors, a sense of realism as if one were exploring a building can be imparted. Furthermore, by limiting the images displayed after the selection of a floor to the same position on that floor, an image of an unrelated floor does not suddenly appear, and the sense of realism is less likely to be lost. Furthermore, by superimposing a floor selection icon on the displayed image, it is possible to easily transition between floors compared to displaying a separate selection menu.

Furthermore, in cases where common equipment is located in the same position on multiple floors, the system can accept pointing operations to that image and have that image included in the field of view before and after switching the displayed floor, making it easier to understand the direction even on the switched floor. Also, by being able to directly specify another floor or its location included in the displayed image, the user can easily move to the desired location, even if it is a different floor.

The present invention has been described above based on the embodiments. The above embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention.

1 Image display system, 10 Image processing device, 14 Input device, 16 Display device, 20 Image providing server, 23 CPU, 24 GPU, 26 Main memory, 32 Communication unit, 34 Storage unit, 36 Output unit, 38 Input unit, 40 Recording medium drive unit, 50 Input information acquisition unit, 52 Data request unit, 54 Data acquisition unit, 56 Image display unit, 58 Request acquisition unit, 60 Image database, 62 Data transmission unit, 64 Icon display unit, 66 Image switching unit, 68 Viewpoint change processing unit, 70 Data acquisition unit, 72 Attribute acquisition unit.

Claims (8)

an input information acquisition unit that accepts a user operation related to an image to be displayed on the display device;
an image display unit that displays on a display device one of a plurality of images taken in different situations at the same location in response to the user operation, and further displays a situation selection icon for selecting a shooting situation at the same location as the image being displayed, and when the input information acquisition unit accepts an operation on the situation selection icon, displays on the display device the image taken in the selected situation;
Equipped with
When a user operation is performed to continuously point within a range that is regarded as the same position as the position at which the situation selection icon is displayed for a predetermined period of time or more, the image display unit displays the situation selection icon representing a different option, or expands the area of the situation selection icon to increase the options represented. The image processing device according to claim 1, characterized in that when the input information acquisition unit receives an operation to move the viewpoint of the panoramic image being displayed, the image display unit switches the display to a panoramic image taken in the most recently selected situation at the destination viewpoint. The image processing device according to claim 1 or 2, characterized in that the image display unit displays the situation selection icon that represents at least one of a situation related to the shooting time and a situation related to the subject as a choice. An image processing device according to any one of claims 1 to 3, characterized in that the image display unit displays the situation selection icon on the thumbnail image being displayed, and enlarges and displays the image taken in the selected situation. An image processing device according to any one of claims 1 to 4, characterized in that the image display unit, when a user points on a displayed image and the pointing continues for a predetermined period of time within a range that is considered to be the same position, displays the situation selection icon superimposed on the pointed position on the image. An image processing device according to any one of claims 1 to 5, characterized in that the image display unit displays a map on the display device and displays an image of a location on the map corresponding to a position specified by the user. receiving a user operation related to an image to be displayed on a display device;
displaying on a display device one of a plurality of images taken of the same place under different conditions in response to the user operation;
displaying a situation selection icon for selecting a shooting situation in the same place as the image being displayed;
accepting an operation on the situation selection icon and displaying an image captured in the selected situation on the display device;
Including,
The step of displaying the situation selection icon is characterized in that, when a user operation is performed to continuously point within a range that is regarded as the same position as the position at which the situation selection icon is displayed for a predetermined period of time or more, the situation selection icon representing a different option is displayed, or the area of the situation selection icon is expanded to increase the options represented. A function of accepting a user operation related to an image to be displayed on the display device;
a function of displaying on a display device one of a plurality of images taken of the same place under different conditions in response to the user operation;
A function to display a situation selection icon to select shooting conditions in the same location as the displayed photographed image,
a function of accepting an operation on the situation selection icon and displaying an image captured in the selected situation on the display device;
This is realized on a computer.
A computer program characterized in that the function of displaying the situation selection icon displays the situation selection icon representing a different option, or expands the area of the situation selection icon to increase the options displayed, when a user operation is performed to continuously point within a range that is regarded as the same position as the position at which the situation selection icon is displayed for a predetermined period of time or more.

Images