JP7467262B2 - Image information generating device, method, and program - Google Patents

Description

Embodiments of the present invention relate to an image information generating device, method, and program used in a system for managing a facility, for example, using images captured of the three-dimensional space of the facility.

In recent years, technologies have been proposed for managing facilities such as business establishments, offices, and residences using captured images. For example, Patent Document 1 describes a technology in which the three-dimensional space inside a building is captured in all directions (360°) from multiple different positions, the captured images are recorded on a storage medium, and a three-dimensional (3D) image showing the inside of the facility is generated by connecting the recorded omnidirectional images. Using this technology, for example, a facility manager or user can remotely grasp the condition of the facility using 3D images without having to go to the site.

US Patent Application Publication No. 2018/0075652

Meanwhile, the interiors of construction sites and living spaces change over time, and there is a demand for the ability to manage these changes using images. However, conventionally, images of the same space corresponding to multiple specified dates and times are generally simply selected from a storage medium and displayed side by side. For this reason, if the shooting conditions of each image taken on the multiple dates and times, such as the shooting position, shooting direction, magnification, etc., are different, simply displaying these images side by side makes it difficult for the administrator or user to accurately grasp the changes in the space.

This invention was made in light of the above-mentioned circumstances, and in one aspect aims to provide a technology for generating image information that can appropriately represent changes in three-dimensional space.

In order to solve the above problem, one aspect of the image information generating device or image information generating method according to the present invention utilizes a storage device that stores omnidirectional images captured in a three-dimensional space at a plurality of shooting positions at a plurality of shooting opportunities in association with coordinates indicating the shooting positions. When information is input requesting a comparison between a first omnidirectional image and a second omnidirectional image in the three-dimensional space captured at a first shooting opportunity and a second shooting opportunity among the plurality of shooting opportunities, a second omnidirectional image whose shooting position coordinates are closest to the shooting position coordinates of the first omnidirectional image is selected from all the second omnidirectional images captured at the second shooting opportunity among the omnidirectional images stored in the storage device. When the shooting position coordinates of the second omnidirectional image selected from all the second omnidirectional images captured at the second shooting opportunity are away from the shooting position coordinates of the first omnidirectional image by a predetermined distance or more, the selected second omnidirectional image is excluded from the target for generating the composite image. Then, the display range and display orientation of the selected second omnidirectional image are adjusted to correspond to the display range and shooting direction of the first omnidirectional image, and a composite image in which the adjusted second omnidirectional image and the first omnidirectional image are arranged so as to be comparable is generated and output.

According to one aspect of the present invention, even if the shooting positions of the comparison reference image and the comparison target image are different, the comparison target image whose shooting position is closest to that of the comparison reference image is selected, and the angle of the selected comparison target image is adjusted to correspond to the comparison reference image, and then display image information for comparing the two images is generated. This makes it possible to generate image information that can appropriately express the state of change in three-dimensional space.

FIG. 1 is a schematic diagram showing the configuration of a system including a server device that operates as an image information generating device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a hardware configuration of a server device in the system shown in FIG. FIG. 3 is a block diagram showing an example of a software configuration of the server device in the system shown in FIG. FIG. 4 is a flowchart showing an example of the procedure and contents of the image information generating process performed by the server device shown in FIG. FIG. 5 is a diagram showing an example of the shooting positions of the omnidirectional image serving as a comparison reference. FIG. 6 is a diagram showing an example of the shooting positions of omnidirectional images to be compared. FIG. 7 is a diagram showing an example of the display image information.

The following describes an embodiment of the present invention with reference to the drawings.

[One embodiment]
(Configuration example)
(1) System FIG. 1 is a schematic diagram of a system including an image information generating device according to an embodiment of the present invention.
This system includes a server device SV that operates as an image information generating device, and is configured to enable data communication between the server device SV and user terminals MT, UT1 to UTn used by users via a network NW.

The user terminals MT, UT1 to UTn include a user terminal MT used by a user who registers omnidirectional images, and user terminals UT1 to UTn used by users who view the registered images, all of which are configured as mobile information terminals such as smartphones or tablet terminals. Note that the user terminals may be notebook or desktop personal computers, and the connection interface to the network NW may be wired or not wireless.

The user terminal MT is capable of transmitting data between the camera CM and the user terminal MT via, for example, a signal cable or a low-power wireless data communication interface such as Bluetooth (registered trademark). The camera CM is made up of a camera capable of capturing images in all directions, and is fixed, for example, to a tripod capable of maintaining a constant height position. The camera CM transmits captured omnidirectional image data to the user terminal MT via the low-power wireless data communication interface.

The user terminal MT also has a function for measuring the current location using signals transmitted from, for example, a GPS (Global Positioning System) or a wireless LAN (Local Area Network). The user terminal MT also has a function for the user to manually input the location coordinates that serve as a reference point in case the above-mentioned location measurement function cannot be used, for example, inside a building.

Each time the user terminal MT receives omnidirectional image data taken at a position from the camera CM, it calculates the position coordinates representing the shooting position based on the position coordinates of the reference point and the movement distance and movement direction measured by the built-in motion sensor (e.g., an acceleration sensor and a gyro sensor). Then, it transmits the received omnidirectional image data, together with the calculated shooting position coordinates and information representing the shooting date and time, to the server device SV via the network NW. These processes are executed by a dedicated application that has been installed in advance.

The user terminals UT1 to UTn each have, for example, a browser. The user terminals UT1 to UTn have a function of accessing the server device SV using the browser, downloading an image of a desired location at a desired date and time in a desired facility and on a desired floor in response to an input operation by the user, and displaying the image on the display.

The network NW is composed of an IP network including the Internet and an access network for accessing this IP network. Examples of access networks that can be used include a public wired network, a mobile phone network, a wired LAN (Local Area Network), a wireless LAN, and a CATV (Cable Television).

(2) Server device SV
2 and 3 are block diagrams showing the hardware and software configurations of the server device SV, respectively.
The server device SV is a server computer installed on the cloud or the Web, and includes a control unit 1 having a hardware processor such as a central processing unit (CPU). A storage unit 2 and a communication interface (communication I/F) 3 are connected to the control unit 1 via a bus 4.

The communication I/F 3 transmits and receives data between the user terminals MT, UT1 to UTn via the network NW under the control of the control unit 1, and may be, for example, an interface for a wired network.

The storage unit 2 uses, for example, a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) as a main storage medium, which can be written to and read from at any time. Note that a combination of read only memory (ROM) and random access memory (RAM) may also be used as the storage medium.

The memory area of the memory unit 2 includes a program memory area and a data memory area. The program memory area stores middleware such as an OS (Operating System), as well as programs necessary to execute various control processes according to an embodiment of the present invention.

The data storage area includes an omnidirectional image storage unit 21, a plan view data storage unit 22, and an adjusted image storage unit 23 as storage units required to implement one embodiment, and further includes a storage unit for operations required for various processes by the control unit 1.

The omnidirectional image storage unit 21 is used to store a group of omnidirectional images for each floor of the target facility acquired from the user terminal MT. The floor plan data storage unit 22 is used to store floor plan data for each floor of the target facility. The adjusted image storage unit 23 is used to store images adjusted by the comparison image adjustment process performed by the control unit 1.

The control unit 1 includes, as control processing functions according to one embodiment of the present invention, an omnidirectional image acquisition unit 11, an image viewing control unit 12, a comparison image selection unit 13, an image angle adjustment unit 14, and a comparison display image generation and output unit 15. All of these processing units 11 to 15 are realized by causing a hardware processor to execute a program stored in a program storage area in the storage unit 2.

The omnidirectional image acquisition unit 11 receives omnidirectional image data from the user terminal MT via the communication I/F 3 each time the data is sent, the data having been taken at each of a number of positions within a building, for example. Then, the omnidirectional image acquisition unit 11 performs processing to associate the received omnidirectional image data with information indicating the coordinates of the photographing position and the photographing date and time, which were received together with the image data, and store the data in the omnidirectional image storage unit 21.

When the image viewing control unit 12 receives an image viewing request sent from the user terminals UT1 to UTn via the communication I/F 3, it performs a process of downloading the corresponding omnidirectional image to the requesting user terminal UT1 to UTn according to the request content. In addition, when the image viewing control unit 12 receives an image comparison request from the user terminals UT1 to UTn, it performs a process of passing the image comparison request to the comparison target image selection unit 13.

An image comparison request may contain both information specifying the comparison criteria and information specifying the comparison target, or it may contain only information specifying the comparison target. The former is used when the user wishes to view the comparison images from the beginning, and the latter is used when the user is already viewing the comparison criteria image and only specifies the comparison target.

The comparison target image selection unit 13 first selects an omnidirectional image corresponding to the shooting date and time included in the information specifying the comparison target from among all omnidirectional images related to the specified facility name and target area stored in the omnidirectional image storage unit, and then selects from each selected omnidirectional image the omnidirectional image whose shooting position coordinates are closest to the shooting position coordinates of the comparison standard image being viewed or the shooting position coordinates included in the information specifying the comparison standard.

The image angle adjustment unit 14 compares the image specified by the information specifying the comparison standard with the omnidirectional image selected by the comparison target image selection unit 13, and adjusts the angle (e.g., display range and shooting direction) of the comparison target omnidirectional image so that it is the same as or close to the angle of the comparison target image. Then, the comparison target image after adjustment is temporarily stored in the adjusted image storage unit 23 together with the comparison target image.

The comparison display image generation and output unit 15 reads the adjusted comparison target image and comparison reference image stored in the adjusted image storage unit 23, and generates display image data in which the two images are arranged side by side by combining the two images. The comparison display image data thus generated is then transmitted to the requesting user terminals UT1 to UTn via the communication I/F 3.

(Example of operation)
Next, an example of the operation of the server device SV configured as above will be described with reference to the flowchart of FIG.

(1) Acquisition of Omnidirectional Images For example, when taking and recording omnidirectional images of multiple points on a desired floor of a desired building, the user first uses the floor plan data of the building to be registered and its floor to determine the position where the user wants to start photographing the floor as a reference point, and obtains the position coordinates of this reference point from the coordinate system of the floor plan data and inputs it to the user terminal MT. As a result, the position coordinates of the reference point of the target floor are set in the user terminal MT. The floor plan data of the building and floor to be registered is stored in advance in the floor plan data storage unit 22 of the server device SV, and the user terminal MT can obtain the floor plan data of the desired building and floor by downloading it from the server device SV.

The user then operates the camera CM to capture images in all directions at the reference point. The camera CM may be operated remotely by the user terminal MT. When the above-mentioned capture operation is performed, omnidirectional image data of the reference point captured by the camera CM is transmitted to the user terminal MT, and the omnidirectional image data is transmitted from the user terminal MT to the server device SV. At this time, the user terminal MT adds information indicating the position coordinates of the reference point and the capture date and time to the omnidirectional image data and transmits it.

When the user has finished photographing at the reference point, he or she moves to the next photographing position (photographing point) and similarly photographs all-directionally using the camera CM. When the user terminal MT receives from the camera CM the all-directional image data photographed at the new photographing point, it transmits the all-directional image data together with information indicating the photographing position coordinates and the photographing date and time to the server device SV. At this time, the photographing position coordinates are calculated from the position coordinates set at the reference point and the travel distance and travel direction from the reference point to the new photographing point measured by a distance sensor (e.g., an acceleration sensor and a gyro sensor) of the user terminal MT.

Similarly, from then on, each time the user moves to a new shooting point and takes an omnidirectional photograph using the camera CM, the user terminal MT transmits the omnidirectional image data sent from the camera CM to the server device SV together with the shooting position coordinates calculated based on the measurement values of the motion sensor each time and information indicating the shooting date and time.

In response to this, the server device SV monitors the start of image capture in step S10 under the control of the omnidirectional image acquisition unit 11. Then, when it receives a notification of the start of image capture from the user terminal MT, it proceeds to step S11 and executes the process of receiving and storing omnidirectional image data as follows.

That is, the omnidirectional image acquisition unit 11 receives omnidirectional image data transmitted from the user terminal MT via the communication I/F 3, and stores the received omnidirectional image data in the omnidirectional image storage unit 21 in a state in which the data is associated with information indicating the shooting position coordinates and shooting date and time received together with the data. At the same time, the omnidirectional image acquisition unit 11 plots the shooting position coordinates on the floor plan data of the corresponding floor stored in the floor plan data storage unit 22.

Similarly thereafter, the omnidirectional image acquisition unit 11 repeatedly executes the process of receiving and storing the omnidirectional image data in step S11 each time the omnidirectional image data is transmitted from the user terminal MT. Note that this process of receiving and storing the omnidirectional image data ends when the omnidirectional image acquisition unit 11 detects in step S12 that a notification of the end of image capture has been sent from the user terminal MT.

The image capturing described above may be performed by multiple people on the same date and time, or may be performed by the same or different people on different dates and times, and the images obtained by either image capturing are stored in the server device SV. Each time an image is captured, floor plan data is generated in which the capturing points are plotted, and stored in the floor plan data storage unit 22. It is not necessary to plot all of the capturing points, and at least one capturing point may be plotted.

(2) Three-dimensional viewing tour using omnidirectional images When viewing omnidirectional images stored in the server SV, the user starts a browser on his/her user terminal UT1 and accesses the server SV. The server SV then first transmits a home screen under the control of the image viewing control unit 12. In this state, when the user specifies the name of the facility that the user wishes to view and its floor number, the server SV transmits floor plan data of the corresponding floor by the image viewing control unit 12, and causes the data to be displayed on the user terminal UT1.

Figure 5 shows an example of the floor plan data downloaded at this time. As shown in Figure 5, the floor plan data displays the floor layout, shooting positions (shooting points), and shooting order in circled numbers. In this state, when the user specifies an arbitrary shooting point, for example, the omnidirectional image at that shooting point is downloaded from the server device SV to the user terminal UT1. Then, when the user changes the display direction of the image, for example by operating the mouse, the display range of the omnidirectional image changes over 360° in response to this operation. Also, when the shooting points are moved sequentially by operating the mouse, the omnidirectional images at each shooting point are downloaded sequentially and displayed. In this way, a three-dimensional viewing tour using omnidirectional images of each room on the floor is possible.

(3) Selection of a Comparison Image During the three-dimensional browsing tour, suppose that the user inputs a different photographing date and time in the user terminal UT1 in order to compare an image of a given room currently being displayed (a comparison reference image) with an image of the same room taken on a different past date and time (a comparison target image). In response, an image comparison request specifying the different photographing date and time is sent from the user terminal UT1 to the server device SV.

In response to this, when the server device SV receives the image comparison request in step S13, under the control of the comparison target image selection unit 13, first in step S14, it identifies the shooting position coordinates of the image being displayed in the 3D browsing tour from the omnidirectional image storage unit 21. Next, in step S15, by searching the omnidirectional image storage unit 21, it selects the omnidirectional image whose shooting position coordinates are closest to the shooting position coordinates of the comparison reference image from among all the omnidirectional images of the same floor at the different shooting date and time specified by the image comparison request.

For example, suppose that the user is currently viewing an all-directional image taken at photography point P1, "10" in Figure 5, at the angle (display range and direction) shown in Q1 in the figure. Also suppose that the photography point on the same floor at a different photography date and time specified by the user as a comparison target is the one shown in Figure 6. In this case, the comparison target image selection unit 13 of the server device SV selects, from all the photography points shown in Figure 6, photography point P2 that is the closest to the photography point P1 being viewed, shown in Figure 5.

(4) Angle Adjustment of Comparison Image Next, under the control of the image angle adjustment unit 14, in step S16, the server device SV reads out the omnidirectional image corresponding to the selected shooting point P2 from the omnidirectional image storage unit 21, and compares this read-out comparison object omnidirectional image with the display image of the comparison reference. Then, the display range and display direction of the comparison image are adjusted so that the angle Q2 of the comparison image approaches the angle Q1 of the comparison reference image. This adjustment process is performed, for example, by shifting the contrast position between the displayed comparison reference image and the comparison image in pixel units and searching for a shift position that minimizes the difference value between each corresponding pixel between the comparison reference image and the comparison image.

When the image angle adjustment unit 14 has completed the adjustment process of the display range and display direction for the comparison image, it associates the adjusted comparison image with the comparison reference image and temporarily stores the image in the adjusted image storage unit 23.

(5) Generation and transmission of comparative display image In step S17, the server device SV reads out the comparison reference image and the comparison target image from the adjusted image storage unit 23 under the control of the comparative display image generation and output unit 15, synthesizes these images in a horizontally arranged state, and generates comparative display image data. At the same time, the comparative display image generation and output unit 15 superimposes floor plan data of the floor on the comparative reference image of the comparative display image data and synthesizes it. The photographing point P1 of the image being viewed and its angle Q1 are displayed in this floor plan data.

In step S18, the comparison display image generation and output unit 15 transmits the generated comparison display image data from the communication I/F 3 to the requesting user terminal UT1.

When the user terminal UT1 receives the comparison display image data transmitted from the server device SV, it displays the comparison display image on the display in place of the comparison reference image that had been displayed up until that point. Figure 7 shows an example of this comparison display image, with the comparison reference image shown on the right side of the screen and the comparison target image on the left side. In the figure, RD is a plan view of the floor displayed on the comparison reference image.

(Action and Effects)
As described above, in one embodiment, omnidirectional images taken at multiple positions in the three-dimensional space for each floor of a facility are stored in the omnidirectional image storage unit 21. When two images of the same floor taken at different times are compared and displayed for comparison, the omnidirectional image whose shooting position coordinates are closest to the shooting position coordinates of the comparison reference image is selected from all the omnidirectional images to be compared, the angle of the selected comparison reference omnidirectional image is adjusted so that it corresponds to the angle of the comparison reference image, and comparison display image data in which the adjusted comparison reference image and the comparison reference image are arranged side by side is generated and transmitted to the user terminal UT1.

Therefore, according to one embodiment, even if the shooting positions of the comparison reference image and the comparison target image are different, the comparison target image whose shooting position is closest to the shooting position of the comparison reference image is selected, and the angle of the selected comparison target image is adjusted so that it corresponds to the angle of the comparison reference image, and then display image information for comparing the two images is generated. This allows the user to generate image information that can appropriately express the state of change in three-dimensional space.

Floor plan data is also overlaid on the comparison display image data, and this plan data shows the shooting point and shooting angle, so users can check at a glance from the plan view the shooting location and direction of the image they are currently viewing.

[Other embodiments]
(1) If there is no image in the omnidirectional image to be compared that corresponds to the shooting position of the comparison reference image, it is difficult to generate image data for comparing the comparison reference image and the comparison target image.

In such a case, when the comparison image selection unit 13 selects the comparison image with the closest shooting position coordinates, it determines whether the distance between the shooting position coordinates of the comparison reference image is greater than or equal to a preset threshold value. If it is determined that the distance between the shooting position coordinates is greater than or equal to the threshold value, the omnidirectional image at that time is not selected as the comparison image, and the comparison display image generation and output unit 15 is notified of this. The comparison display image generation and output unit 15 generates a message to the effect that no corresponding comparison image exists, and transmits this message to the user terminal UT1 for display. In this way, it is possible to prevent the problem of an inappropriate image being displayed as a comparison image.

(2) In the above embodiment, an example was given of generating comparative display image data that displays omnidirectional images of the same location at two different dates and times side by side, but comparative display image data may be generated that displays images at three or more different dates and times side by side.

(3) In the above embodiment, an example was given of generating comparative display image data that comparatively displays images taken at two different dates and times. However, data may be generated that comparatively displays omnidirectional images taken on the same day by two different users on the same floor.

(4) In the above embodiment, the functions of the image information generating device are provided in the server device SV, but they may be provided in an inter-network connection device such as an edge router or in a user terminal. In addition, the control unit and the storage unit may be provided separately in separate server devices or terminal devices, and these may be connected via a communication line or network.

(5) In addition, various modifications can be made to the configuration of the image information generating device, the procedure and processing content of the image generating process, the type of three-dimensional space, etc., without departing from the spirit and scope of this invention.

In short, this invention is not limited to the above-mentioned embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

SV: Server device MT, UT1 to UTn: User terminal NW: Network CM: Camera 1: Control unit 2: Storage unit 3: Communication I/F
Description of the Reference Numerals 4: bus 11: omnidirectional image acquisition unit 12: image viewing control unit 13: comparison image selection unit 14: image angle adjustment unit 15: comparison display image generation and output unit 21: omnidirectional image storage unit 22: plan view data storage unit 23: adjusted image storage unit

Claims (6)

An image information generating device connectable to a storage device that stores omnidirectional images obtained by photographing a three-dimensional space at a plurality of photographing positions on a plurality of photographing occasions in association with coordinates indicating the photographing positions, the image information generating device comprising:
a selection unit that, when information requesting a comparison between a first omnidirectional image and a second omnidirectional image in the three-dimensional space captured at a first photographing opportunity and a second photographing opportunity among the plurality of photographing opportunities is input, selects, from among all the second omnidirectional images captured at the second photographing opportunity among the omnidirectional images stored in the storage device, a second omnidirectional image whose photographing position coordinates are closest to those of the first omnidirectional image;
an adjustment unit that adjusts a display range and a display orientation of the selected second omnidirectional image so as to correspond to a display range and an imaging direction of the first omnidirectional image;
an image generating unit that generates and outputs a composite image in which the second omnidirectional image and the first omnidirectional image whose display range and display orientation have been adjusted are arranged so as to be comparable ;
The selection unit excludes the selected second omnidirectional image from targets for generating the composite image when the shooting position coordinates of the second omnidirectional image selected from all the second omnidirectional images captured at the second shooting opportunity are away from the shooting position coordinates of the first omnidirectional image by a preset distance or more.
Image information generating device. The image information generating device according to claim 1, wherein, when information is input requesting a comparison between the first omnidirectional image and the second omnidirectional image in the three-dimensional space taken at a first date and time and a second date and time, respectively, the selection unit selects, from among all the second omnidirectional images taken at the second date and time among the omnidirectional images stored in the storage device, a second omnidirectional image whose shooting position coordinates are closest to the shooting position coordinates of the first omnidirectional image. 2. The image information generating device according to claim 1, wherein the image generating unit generates and outputs a message indicating that the second omnidirectional image to be compared does not exist when the selected second omnidirectional image is excluded from the generation of the composite image. The image information generating device according to claim 1, wherein the image generating unit generates an image by further superimposing an image in which the coordinates of the shooting position of at least one of the first omnidirectional image and the second omnidirectional image are plotted on a plan view of the three-dimensional space on the composite image in which the first omnidirectional image and the second omnidirectional image are arranged so as to be comparable. 1. An image information generating method executed by a device connectable to a storage device that stores omnidirectional images obtained by photographing a three-dimensional space at a plurality of photographing positions on a plurality of photographing occasions in association with coordinates indicating the photographing positions, the method comprising:
a selection process for selecting, when information is input requesting a comparison between a first omnidirectional image and a second omnidirectional image in the three-dimensional space captured at a first photographing opportunity and a second photographing opportunity among the plurality of photographing opportunities, from among all the second omnidirectional images captured at the second photographing opportunity among the omnidirectional images stored in the storage device, a second omnidirectional image whose photographing position coordinates are closest to the photographing position coordinates of the first omnidirectional image ;
an adjustment process of adjusting a display range and a display orientation of the selected second omnidirectional image so as to correspond to a display range and a shooting direction of the first omnidirectional image;
an image generating step of generating and outputting a composite image in which the second omnidirectional image and the first omnidirectional image, the display range and the display orientation of which have been adjusted, are arranged so as to be comparable to each other ;
The selection step includes excluding the selected second omnidirectional image from the generation target of the composite image when the shooting position coordinates of the second omnidirectional image selected from all the second omnidirectional images captured at the second shooting opportunity are apart from the shooting position coordinates of the first omnidirectional image by a preset distance or more.
A method for generating image information. 5. A program for causing a processor included in the image information generating device to execute the processes of the selection unit, the adjustment unit, and the image generating unit included in the image information generating device according to claim 1 .