JP2022050293A - Information processing device, information processing method, information processing system, and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing device, an information processing method, an information processing system and an information processing program, with which it is possible to process information that pertains to a photograph of a space and a plan view of the space.

SOLUTION: An information processing device 3 comprises a storage unit 33 and a control unit 32. The storage unit 33 stores a trained model having been trained by machine learning using a convolutional neural network on the basis of a correspondence relation between the plan view of first N spaces and one or more photographs in which each of the first N spaces is photographed. The control unit 32 extracts, on the basis of the trained model and the photograph of a second space differing from the N spaces and the plan view of the second space, information in which the photograph of the second space is compressed, and concatting this and information in which the plan view of the second space is compressed, thereby estimating the photographing position and photographing direction at which the photograph of the second space in the plan view of the second space was photographed.

SELECTED DRAWING: Figure 3

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to an information processing apparatus, an information processing method, an information processing system, and an information processing program. As an example, the present invention relates to an information processing device, an information processing method, an information processing system, and an information processing program capable of processing information related to a "room photograph" and a "room layout" of a real estate property.

The property guide of a real estate usually contains a floor plan as a plan view of the real estate, and in addition, a photograph of a specific room or facility may be posted. In the past, when a room photo of a real estate property (for example, a photo of a corridor, living room, Western-style room, kitchen, toilet, entrance, veranda, etc.) was taken, the correspondence between the photo and the floor plan (for example, from which place) was taken. The real estate agent, etc. manually entered the information (whether it was taken or in which direction it was taken, etc.).

Nam Vo et al. CVPR2019, Composing Text and Image for Image Retrieval-An Imperial Odyssey (arXiv: 1812.01719) Shuai Liao et al. CVPR 2019, Physical Restriction: Regression Viewpoints, Surface Normals and 3D Rotations on n-Spheres (arXiv: 1904.05404)

As mentioned above, the room photo of the property has been posted in the property guide, etc., and the property description explaining which place or room the photo is in has been described. However, to date, attempts to automate the input of the shooting position and shooting direction have not been successful (other than the present inventors). The main reason for this was that the floor plan, which is an architectural drawing, and the room photograph, which was taken of the actual room such as a real estate room, are completely different dimensions. See, for example, FIGS. 1 and 2. Fig. 1 is an example of a so-called floor plan (floor design). In the property shown in Fig. 1, there is a storage in front of the entrance, and from the entrance, you can enter the toilet and the 10.9 tatami mat LDK (living / dining / kitchen). .. The LDK (living-dining kitchen) has a kitchen, which is adjacent to a 6.7 tatami mat room on the right side of the drawing and a 5.4 tatami mat room on the left side of the drawing. There will be 2 rooms and an LDK. On the other hand, FIG. 2 is a photograph of the inside of the property shown in FIG. 1 taken by a digital camera. It is a diagram with different dimensions and viewpoints from the floor plan (floor design).

In FIG. 1, the shooting position and shooting direction of the photograph are indicated by arrows. If humans observe it, it can be seen that there is a kitchen in front and multiple doors are located in the direction of the right hand, but such identification of the shooting position and shooting direction should be automatically and accurately input by a computer. Because I couldn't do it, I had to guess by a person at a real estate company and input it manually. The inventors who have discovered such problems are more convenient information processing devices related to information processing of floor plans and room photographs of real estate properties, which can solve these problems, and the information processing devices. The present invention was made by recognizing that an information processing method using an information processing device, an information processing system including the information processing device, and an information processing program for operating a computer as the information processing device are required. At that time, although they came into contact with prior art documents related to image recognition such as Non-Patent Documents 1 and 2, the techniques described in these documents could not solve this problem.

In one embodiment of the present invention, an information processing apparatus (invention) having a storage unit (corresponding to the storage unit 33 in the detailed description of the invention) and a control unit (corresponding to the control unit 32 in the detailed description of the invention). The storage unit corresponds to the information of the first N objects (corresponding to the information regarding the room of the learning model in the detailed description of the invention). , The trained model based on the correspondence with one or more photographs (corresponding to the room photograph of the learning model in the detailed description of the invention) obtained by taking each of the first N objects is stored, and the control thereof is performed. The part is a photograph of the trained model and a second object different from the N objects (corresponding to a room photograph of an estimation target in the detailed description of the invention) and a plane of the second object. A photograph of the second object in the plan view of the second object is taken based on the figure (corresponding to the floor view corresponding to the room photograph of the place to be estimated in the detailed description of the invention). An information processing device is provided that estimates the shooting position and shooting direction.

An information processing device having the above configuration can estimate the shooting position and shooting direction in which a photograph of an object is taken in a plan view of the object by using a trained model.

In one embodiment of the present invention, an information processing apparatus (invention) having a storage unit (corresponding to the storage unit 33 in the detailed description of the invention) and a control unit (corresponding to the control unit 32 in the detailed description of the invention). In the information processing method in the information processing apparatus 3 in the detailed description of the above, the storage unit provides information on the first N objects (information on the room of the learning model in the detailed description of the invention). Corresponds to) and one or more photographs of each of the first N objects (corresponding to the room photograph of the learning model in the detailed description of the invention) and stores the trained model based on the correspondence relationship. Then, the control unit performs the trained model, a photograph of a second object different from the N objects (corresponding to a room photograph of an estimation target in the detailed description of the invention), and the second object. The second object in the plan view of the second object based on the plan view of the object (corresponding to the floor view corresponding to the room photograph of the place to be estimated in the detailed description of the invention). An information processing method is provided that estimates the shooting position and shooting direction in which the photo was taken.

The information processing method provided with the above configuration makes it possible to estimate the shooting position and shooting direction in which a photograph of an object is taken in a plan view of the object by using a trained model by a computer.

In one embodiment of the invention, the computer is an information processing unit having a storage unit (corresponding to the storage unit 33 in the detailed description of the invention) and a control unit (corresponding to the control unit 32 in the detailed description of the invention). A device (corresponding to the information processing device 3 in the detailed description of the invention), wherein the storage unit corresponds to information on the first N objects (corresponding to information about the room of the learning model in the detailed description of the invention). ) And one or more photographs taken of each of the first N objects (corresponding to the room photograph of the learning model in the detailed description of the invention) and memorize the trained model based on the correspondence relationship. , The control unit has a photograph of the trained model and a second object different from the N objects (corresponding to a room photograph of an estimation target in the detailed description of the invention) and the second object. Photograph of the second object in the plan view of the second object based on the plan view of the object (corresponding to the floor view corresponding to the room photograph of the place to be estimated in the detailed description of the invention). An information processing program is provided that functions as an information processing device that estimates the shooting position and shooting direction in which the image was taken.

An information processing program having the above configuration makes it possible to use a computer as a device for estimating a shooting position and a shooting direction in which a photograph of an object is taken in a plan view of the object by using a trained model.

In one embodiment of the present invention, the information processing system includes an information processing device (corresponding to the information processing device 3 in the detailed description of the invention), and the information processing device is a storage unit (corresponding to the detailed description of the invention). It has a storage unit 33) and a control unit (corresponding to the control unit 32 in the detailed description of the invention), and the storage unit has information on the first N objects (corresponding to the detailed description of the invention). , Corresponds to information about the room of the learning model) and one or more photographs (corresponding to the room photograph of the learning model in the detailed description of the invention) of each of the first N objects. A photograph of the trained model and a second object different from the N objects (a room photograph where the control unit is an estimation target in the detailed description of the invention). (Corresponding to) and the plan view of the second object (corresponding to the floor plan corresponding to the room photograph of the place to be estimated in the detailed description of the invention). An information processing system is provided that estimates the shooting position and shooting direction in which the photograph of the second object in the figure is taken. The information processing system may further include a terminal accessible to the information processing apparatus via an internet line or the like.

An information processing system having the above configuration makes it possible to estimate the shooting position and shooting direction in which a photograph of an object is taken in a plan view of the object by using a trained model in a computer, when the system includes a terminal. Then, it is possible to grasp the result of estimation of the shooting position and the shooting direction on the terminal.

In one embodiment of the present invention, in estimating the photographing position and the photographing direction, the control unit compresses the photograph of the second object based on the trained model and photographs the second object. It may be characterized in that it recognizes a feature consisting of the content and position of the object in the above-mentioned second object and contrasts it with the feature consisting of the content and position of the object in the plan view of the second object.

By providing the above configuration, it is possible to reduce the amount of calculation for estimating the shooting position and shooting direction, and to estimate the shooting position and shooting direction with sufficient accuracy even in a normal server device at the time of filing the present invention. do.

In one embodiment of the present invention, recognizing a feature consisting of the content and position of an object in the photograph of the second object is information compressed by the control unit into a vector in the channel direction in the photograph of the second object. It may be characterized by extracting information compressed into a vector in the width direction.

By providing the above configuration, it is possible to improve the accuracy of estimation of the shooting position and the shooting direction.

In one embodiment of the present invention, the contrast is that the control unit compresses the information compressed into the vector in the channel direction and the information compressed into the vector in the width direction in the photograph of the second object, respectively. It may be characterized by compressing the plan view of the second object with the compressed information more than once.

By providing the above configuration, it is possible to improve the accuracy of estimation of the shooting position and the shooting direction.

In one embodiment of the present invention, the control further comprises a heat map (for example, a two-dimensional heat map) for the control unit to estimate the imaging position and the imaging. It may be characterized by including outputting and evaluating a heat map for estimating a direction (for example, a one-dimensional heat map).

By providing the above configuration, it is possible to improve the accuracy of estimation of the shooting position and the shooting direction.

In one embodiment of the present invention, the control unit compresses the information compressed into the vector in the channel direction and the information compressed into the vector in the width direction in the photograph of the second object, respectively. Then, one or more copies are made based on the width direction and the height direction of the plan view of the second object, and the plan view of the second object is compressed with the compressed information based on the duplicated information. It may be a feature.

By providing the above configuration, the information compressed in the vector in the channel direction of the photograph and the information compressed in the vector in the width direction can be separately associated with the plan view, and the accuracy may be improved. ..

In one embodiment of the present invention, the information on the first N objects is information on N real estate properties (for example, room contents, room information, information on objects in a room, information on floor plans). There may be one or more photographs of each of the first N objects, which may be one or more first room photographs of the real estate property, and the second object may be present. The plan view of the above may be a second floor plan, and the photograph of the second object may be a second room photograph.

By providing the above configuration, it is possible to estimate the shooting position and shooting direction in which a room photograph (for example, a panoramic image) in a floor plan of a predetermined room is taken.

In one embodiment of the present invention, the information processing device has a control unit and associates a plan view of the object with a photograph of the object, and the control unit is a three-dimensional feature amount of the plan view. And a one-dimensional feature amount compressed into a vector in the channel direction extracted from the photograph and a one-dimensional feature amount compressed into a vector in the width direction, and each one-dimensional feature amount in the width direction of the three-dimensional feature amount and An information processing apparatus is provided that duplicates and concuts one or more in the height direction. In one embodiment of the present invention, the plan view of the object may be a floor plan of a real estate property, and the photograph of the object may be a room photograph.

With the information processing apparatus provided with the above configuration, when associating a three-dimensional feature amount with a one-dimensional feature amount, it is possible to associate the three-dimensional feature amount without losing the information possessed by the three-dimensional feature amount.

In one embodiment of the present invention, there is an information processing method for associating a plan view of an object with a photograph of the object in an information processing apparatus having a control unit, wherein the control unit is the 3 in the plan view. The dimensional feature amount, the one-dimensional feature amount compressed into the channel direction vector extracted from the photograph, and the one-dimensional feature amount compressed into the width direction vector, and each one-dimensional feature amount of the three-dimensional feature amount An information processing method is provided in which one or more copies are duplicated and combined in the width direction and the height direction. In one embodiment of the present invention, the plan view of the object may be a floor plan of a real estate property, and the photograph of the object may be a room photograph.

With the information processing method provided with the above configuration, when associating a three-dimensional feature amount with a one-dimensional feature amount, it is possible to associate the three-dimensional feature amount without losing the information possessed by the three-dimensional feature amount.

In one embodiment of the present invention, the computer is an information processing device having a control unit for associating a plan view of the object with a photograph of the object, and the control unit is a three-dimensional feature of the plan view. The quantity, the one-dimensional feature quantity compressed into the channel direction vector extracted from the photograph, and the one-dimensional feature quantity compressed into the width direction vector, and each one-dimensional feature quantity in the width direction of the three-dimensional feature quantity. And a computer program that operates as an information processing device that duplicates one or more in the height direction and concuts is provided. In one embodiment of the present invention, the plan view of the object may be a floor plan of a real estate property, and the photograph of the object may be a room photograph.

With the information processing program having the above configuration, when the three-dimensional feature amount and the one-dimensional feature amount are associated with each other in the information processing apparatus, the information possessed by the three-dimensional feature amount can be associated without being lost.

In one embodiment of the present invention, the information processing system includes an information processing apparatus including a control unit for associating a plan view of an object with a photograph of the object, wherein the control unit is 3 in the plan view. The dimensional feature amount, the one-dimensional feature amount compressed into the channel direction vector extracted from the photograph, and the one-dimensional feature amount compressed into the width direction vector, and each one-dimensional feature amount of the three-dimensional feature amount An information processing system is provided that duplicates and concats one or more in the width direction and the height direction. The information processing system may further include a terminal accessible to the information processing apparatus via an internet line or the like. In one embodiment of the present invention, the plan view of the object may be a floor plan of a real estate property, and the photograph of the object may be a room photograph.

With the information processing system provided with the above configuration, when associating a three-dimensional feature amount with a one-dimensional feature amount in an information processing apparatus, it is possible to associate the three-dimensional feature amount without losing the information possessed by the three-dimensional feature amount.

In one embodiment of the present invention, an information processing device having a control unit and associating a plan view of an object with a photograph of the object, wherein the control unit has a feature amount of the plan view and a photograph of the object. An information processing apparatus is provided which controls and controls the feature amount extracted from the photograph, outputs the control result as a one-dimensional or two-dimensional heat map, and performs the association. In one embodiment of the present invention, the plan view of the object may be a floor plan of a real estate property, and the photograph of the object may be a room photograph. In another embodiment of the present invention, the control unit is provided, and the surrounding information of an object (corresponding to the "terminal device 5" in the modified example 2) and the surrounding information (corresponding to the "surrounding image" in the modified example 2). An information processing device that associates with a photograph taken from the object (corresponding to "calculating the current orientation of the terminal device 5" in the second modification), and the control unit controls the surrounding information. An information processing apparatus is provided which controls and controls the feature amount extracted from the photograph and outputs the control result as a one-dimensional heat map to perform the correspondence.

The information processing apparatus provided with the above configuration makes it possible to improve the accuracy when associating a plan view (or surrounding information or surrounding image) with a photograph.

In one embodiment of the present invention, there is an information processing method in an information processing apparatus having a control unit and associating a plan view of an object with a photograph of the object, wherein the control unit is the plan view of the object. Provided is an information processing method in which a feature amount and a feature amount extracted from the photograph are concut and compared, and the control result is output as a one-dimensional or two-dimensional heat map to perform the correspondence. .. In one embodiment of the present invention, the plan view of the object may be a floor plan of a real estate property, and the photograph of the object may be a room photograph. In another embodiment of the present invention, the control unit is provided, and the surrounding information of an object (corresponding to the "terminal device 5" in the modified example 2) and the surrounding information (corresponding to the "surrounding image" in the modified example 2). An information processing method in an information processing device that associates with a photograph taken from the object (corresponding to "calculating the current orientation of the terminal device 5" in the second modification), wherein the control unit Provided is an information processing method in which a feature amount of the surrounding information and a feature amount extracted from the photograph are concatenated and compared, and the control result is output as a one-dimensional heat map to perform the correspondence. Will be done.

The information processing method provided with the above configuration makes it possible to improve the accuracy when associating a plan view (or surrounding information or a surrounding image) with a photograph.

In one embodiment of the present invention, the computer is an information processing device having a control unit and associating a plan view of an object with a photograph of the object, wherein the control unit is a feature of the plan view. A computer program that operates as an information processing device that controls the amount and the feature amount extracted from the photograph, controls the control amount, outputs the control result as a one-dimensional or two-dimensional heat map, and performs the correspondence. Is provided. In one embodiment of the present invention, the plan view of the object may be a floor plan of a real estate property, and the photograph of the object may be a room photograph. In another embodiment of the present invention, the computer has a control unit and corresponds to the surrounding information (corresponding to the "peripheral image" in the modified example 2) of an object (corresponding to the "terminal device 5" in the modified example 2). An information processing device that associates (.) With a photograph taken from the object (corresponding to "calculating the current orientation of the terminal device 5" in the second modification), wherein the control unit: The feature amount of the surrounding information and the feature amount extracted from the photograph are computerized and compared, and the control result is output as a one-dimensional heat map to operate as an information processing device that performs the correspondence. A computer program is provided.

An information processing program having the above configuration makes it possible to improve the accuracy when associating a plan view (or surrounding information or surrounding image) with a photograph in an information processing apparatus.

In one embodiment of the present invention, the information processing system includes an information processing apparatus including a control unit for associating a plan view of an object with a photograph of the object, wherein the control unit is a feature of the plan view. An information processing system is provided in which an amount and a feature amount extracted from the photograph are concut and compared, and the control result is output as a one-dimensional or two-dimensional heat map to perform the association. The information processing system may further include a terminal accessible to the information processing apparatus via an internet line or the like. In one embodiment of the present invention, the plan view of the object may be a floor plan of a real estate property, and the photograph of the object may be a room photograph. In another embodiment of the present invention, the terminal provided with the photographing means (corresponding to the "terminal device 5" in the modified example 2) and the surrounding information of the terminal (corresponding to the "surrounding image" in the modified example 2). An information processing system including an information processing device having a control unit for associating a photograph taken from the terminal (corresponding to "calculating the current orientation of the terminal device 5" in the modification 2). Then, the control unit concuts and controls the feature amount of the surrounding information and the feature amount extracted from the photograph, outputs the control result as a one-dimensional heat map, and performs the correspondence. , An information processing system is provided.

The information processing system provided with the above configuration makes it possible to improve the accuracy when associating a plan view (or surrounding information or surrounding image) with a photograph in an information processing apparatus.

In one embodiment of the present invention, the estimated shooting position and shooting direction may be displayed by using a mark such as an arrow on the floor plan, and an operation of an input unit in the user terminal device (for example, selecting the mark). By doing so), the corresponding room photo may be displayed.

In one embodiment of the invention, the information processing device provides a virtual tour that provides an automatic and / or interactive tour of a real estate property to a user terminal device using an estimated shooting position and shooting direction. .. In a virtual tour, the shooting direction is provided by a corresponding panoramic image in a specific room (position) in the floor plan visited in the virtual tour, and the shooting direction is changed by receiving a signal from the user to change the direction (shooting direction). It is also possible to provide different panoramic images.

In one embodiment of the present invention, an information processing apparatus having a storage unit and associating the first information with the second information, the three-dimensional feature amount obtained from the first information and the second information. First one-dimensional feature amount (for example, one-dimensional feature amount compressed into a vector in the channel direction) and second one-dimensional feature amount (for example, one-dimensional feature amount compressed into a vector in the width direction) obtained from the above information. And one or more of each one-dimensional feature amount are duplicated in the width direction and the height direction of the three-dimensional feature amount and combined to perform the correspondence between the first information and the second information. An information processing device is provided.

According to the information processing apparatus having the above configuration, it is possible to provide an information processing apparatus that performs information processing (concut) when the dimensions of the feature quantities of the two inputs are different.

In one embodiment of the present invention, the computer has a control unit, and the control unit has a three-dimensional feature amount of the first information and a first one-dimensional feature amount obtained from the second information (for example). , One-dimensional feature amount compressed into a vector in the channel direction) and a second one-dimensional feature amount (for example, one-dimensional feature amount compressed into a vector in the width direction). Provided is a computer program that duplicates one or more in the width direction and the height direction of the above, converts the first information, and operates as an information processing apparatus for associating the first information with the second information.

According to the computer program having the above configuration, it is possible to provide a computer program capable of performing information processing (concut) when the dimensions of the feature quantities of the two inputs are different.

In one embodiment of the present invention, there is an information processing method having a control unit, wherein the control unit has a three-dimensional feature amount of the first information and a first one-dimensional feature amount obtained from the second information. For example, the one-dimensional feature amount compressed into a vector in the channel direction) and the second one-dimensional feature amount (for example, the one-dimensional feature amount compressed into a vector in the width direction) are combined with the three-dimensional feature amount. An information processing method is provided in which one or more copies are duplicated in the width direction and the height direction of a quantity, and the first information and the second information are associated with each other.

According to the information processing method having the above configuration, it is possible to provide an information processing method in an information processing apparatus capable of performing information processing (concut) when the dimensions of the feature quantities of the two inputs are different.

In one embodiment of the present invention, an information processing system including an information processing apparatus having a control unit, wherein the control unit has a three-dimensional feature amount of the first information and a first information obtained from the second information. A one-dimensional feature amount (for example, a one-dimensional feature amount compressed into a vector in the channel direction) and a second one-dimensional feature amount (for example, a one-dimensional feature amount compressed into a vector in the width direction) are each described as one-dimensional feature amount. An information processing system is provided that duplicates one or more of the three-dimensional feature quantities in the width direction and the height direction and concuts them to associate the first information with the second information.

According to the information processing system having the above configuration, it is possible to provide an information processing system having an information processing device as a component capable of performing information processing (concut) when the dimensions of the feature quantities of the two inputs are different. can.

According to one aspect of the present invention, it is possible to estimate the shooting position and the shooting direction of a photograph of the object in a plan view of the object. For example, it is possible to estimate the shooting position and shooting direction by inputting a panoramic image and a floor plan of a real estate (property).

FIG. 1 is a diagram showing an example of a floor plan image. FIG. 2 is a diagram showing an example of a room photograph (panoramic photograph) of the room shown in FIG. 1 taken by a digital camera. FIG. 3 is a diagram showing an example of the configuration of the information processing device 3 (server). FIG. 4 is a diagram showing an example of the configuration of an information processing system 1 including an information processing device 3 (server), an Internet 4, and a user terminal device 5. FIG. 5 is a diagram showing an example of the configuration of the user terminal device. FIG. 6 is a diagram for showing a preliminary step of an information processing step according to an embodiment of the present invention. Specifically, an example of the steps until the storage unit 33 stores the trained model G is shown. FIG. 7 is a diagram for showing an information processing step according to an embodiment of the present invention. FIG. 8 is a diagram for supplementarily explaining an information processing step according to an embodiment of the present invention. Specifically, it is a description about a step in the case of extracting information in the width direction from a room photograph (for example, a panoramic image). FIG. 9 is a diagram for supplementarily explaining an information processing step according to an embodiment of the present invention. Specifically, it is a description about a step in the case of extracting information in the width direction from a room photograph (for example, a panoramic image). In FIG. 8, the position of the object to be the target in the room photograph is shaded and displayed. FIG. 10 is a diagram for supplementarily explaining an information processing step according to an embodiment of the present invention. Specifically, the tensor information on the 3rd floor is possessed when the tensor information on the 1st floor (information in the channel direction and / or the information in the width direction) and the tensor information on the 3rd floor (information on the floor plan) are combined. It relates to a method that makes it possible to perform a concut without losing the spatial information to be performed.

The flowchart illustrating one embodiment of the invention means a process step and allows for alternative embodiments without limitation to the particular example. It is also permissible to carry out the process steps in a different order than the particular description herein, as long as it does not contradict the express statements herein. As described above, various changes can be made to the present embodiment without departing from the spirit and scope of the present invention. Hereinafter, in the present specification, the information processing content in the information processing device or the information processing device will be described by taking as an example a “floor plan (image)” of a predetermined room and a “room photograph” of the room.

[Object recognition]
Recently, regarding object recognition, convolutional neural networks (CNN) not only classify images (image recognition), but also detect the position and size of objects (object detection) and extract shapes (areas). It is also used for (extraction), for example, region CNN R-CNN (further, Faster R-CNN), Yoro, etc. for object detection, SegNet for region extraction, and U-Net (U-Net for estimating image segmentation). Neural networks such as U-shaped convolutional networks) have come to be used.

For example, in R-CNN, it learns whether a certain rectangle is an object or a background, and learns what is specifically reflected in the detected place. In U-Net (U-shaped convolutional network), both "local features" and "overall position information" of an object are integrated and learned. In the present invention, when these existing techniques and the techniques described in Non-Patent Documents 1 and 2 are used, detailed description of these existing techniques itself shall be omitted, and the technical contents thereof shall be incorporated in the present specification by reference. ..

[hardware]
The basic configuration of the hardware according to the embodiment of the present invention will be described. First, the information processing device 3 has a server in mind, but may be, for example, an electronic device such as a mobile terminal such as a smartphone or a tablet terminal, a notebook computer, or a desktop computer. The information processing device 3 is indispensable as an information processing device such as a physical arithmetic unit (CPU and / or GPU, not shown), a working memory for storing temporary work contents (RAM, not shown), and the like. It has a physical configuration.

As shown in FIG. 3, the information processing apparatus 3 has a communication unit 31. The communication unit 31 is connected so as to be able to communicate with the outside via a network such as the Internet. As the network, a known network connection method such as the Internet by a connection method regardless of a wired line or a wireless line, or a local area network using Ethernet (registered trademark) or the like can be used.

The information processing device 3 further includes a control unit 32 that processes image data and the like. As used herein as a "control unit" is meant a function in which the corresponding program is read by the CPU and / or GPU and executed by the CPU and / or GPU. The control unit 32 may be realized by the processor in the information processing apparatus 3 executing a predetermined program, or may be implemented by hardware.

The information processing device further includes a storage unit 33. Various data handled by the control unit 32 are stored in the storage unit 33. The storage unit 33 is data storage such as an internal hard disk drive, an internal solid state drive, and an external data storage (for example, an external hard disk drive, a tape drive for a server, etc.), and may be a data storage such as cloud storage. May be good. In that sense, the present invention allows an aspect in which an information processing device different from the control unit 32 and the storage unit 33 is used.

The information processing apparatus 3 may further include an input unit and a display unit (not shown), but these are not essential configurations. For example, the input unit is an interface for inputting information, such as a touch panel or microphone in a mobile terminal, a touch pad in a notebook computer or a desktop computer, a keyboard or a mouse. The display unit is an interface for displaying various information to the user of the information processing apparatus 3, and is a video display means such as a liquid crystal display or a head-mounted display.

[Terminal device 5]
The information processing system may further include a user terminal device 5 (FIG. 4). The user terminal device 5 (terminal 5a and terminal 5b) means a client terminal for input / output by the user. For example, it may be a mobile terminal such as a smartphone or a tablet terminal, or an electronic device such as a notebook computer, a desktop computer, or a workstation. The user terminal device 5 and the information processing device 3 are connected to each other via the network 4. In the second modification, the terminal device 5 is a robot equipped with a camera or an autonomous driving vehicle equipped with a camera, but this is also an example of a client terminal capable of accessing the information processing device 3 through the network 4. There is no change.

FIG. 5 is a diagram showing an example of the configuration of the user terminal device 5. As shown in FIG. 5, the terminal device 5 includes a (terminal) communication unit 51, a (terminal) control unit 52, a (terminal) storage unit 53, a (terminal) input unit 54, and a (terminal) display unit 55. And each part is connected to each other so as to be able to communicate with each other via a bus. Hereinafter, each part of the user terminal device 5 will be described with a terminal communication unit 51 or the like and a “terminal” at the beginning in order to distinguish each part from the information processing device 3.

The terminal communication unit 51 is a communication interface between the user terminal device 5 and the network 4. The terminal communication unit 51 can transmit and receive information between the terminal device 5 and the information processing device 3 via the network 4.

The terminal control unit 52 is a control means for performing various processes of the terminal device 5. The terminal control unit 52 may be realized by the processor in the terminal device 5 executing a predetermined program, or may be implemented by hardware.

The terminal storage unit 53 is, for example, a data storage such as an internal memory or an external memory (SD memory card or the like). Various data handled by the terminal control unit 52 are stored in the terminal storage unit 53.

The terminal input unit 54 is an interface for a user to input information to the terminal device 5, for example, a camera (CCD camera or CMOS camera) in a mobile terminal, a touch panel or microphone, a touch pad in a notebook computer, a keyboard, a mouse, or the like. Is. In the second modification in which the terminal device 5 is a robot or an autonomous vehicle, a camera is indispensable as the terminal input unit 54.

The terminal display unit 55 is an interface for displaying various information from the terminal device 5 to the user, and is a video display means such as a liquid crystal display. Specifically, for example, the terminal display unit 55 may display a GUI (Graphical User Interface) for receiving an operation from the user. In the second modification in which the terminal device 5 is a robot or an autonomous vehicle, the terminal display unit 55 is unnecessary.

[Floor plan]
A floor plan image (hereinafter, also simply referred to as a “floor plan”) is a diagram (for example, a plan view) of the arrangement of rooms (there may be more than one) in a building, and is usually drawn at a uniform scale. It has been. There are various types or styles of floor plans used by real estate retailers, architects, real estate agents, real estate information agents, etc. Normally, not only the flat shape of the room, but also the position and type of certain objects in the room (for example, doors, door opening / closing directions, kitchen, toilet, bathtub, floor heating position, etc.) are determined by certain rules. Are listed. The floor plan image can be of any image format including JPEG, TIFF, GIF, BMP, PNG and the like.

[Room photo]
In the present invention, the "room photograph" is intended to mean a color, grayscale, monochrome, or the like photograph of the space inside a room or a building. Room photographs can be of any image format, including JPEG, TIFF, GIF, BMP, PNG and the like.

In one embodiment of the present invention, it is assumed that the room photograph is a panoramic image, and in the present invention, the "panoramic image" can capture one or more objects at an internal location in a real estate property. It shall mean any image having a wide field of view.

In the present embodiment, it is assumed that the image (room photograph 10) of the space inside the room or the inside of the building is a panoramic image with an angle of view of 360 degrees, and such an angle of view is most suitable for the present invention. .. However, the panoramic image can also have a field of view of less than 360 degrees in the horizontal direction. Also, theoretically, as the room image 10 of the present invention, an image having a narrower field of view, for example, 270 degrees, 180 degrees or less, can be used. Hereinafter, the terms panoramic image and room photograph may be used without distinction in the present specification. That is, an image having an angle of view of 360 degrees, which is a photograph of a room or a space in a building, is also simply referred to as a "room photograph".

[Trained model]
In the present invention, data (“learned model”) that has been trained (automatically adjusting weights) related to an image or the like to be estimated in advance is stored in the storage unit 33. For example, in one embodiment of the present invention, a task of estimating a room type using a panoramic image as an input is trained by a convolutional neural network in advance, and a trained model obtained as a result is stored in a storage unit 33. (Fig. 6: "Preliminary step"). The learning may be performed in the control unit 32 of the information processing device 3, but may be performed in another information processing device and stored in the storage unit 33 of the information processing device 3.

In one example, the learning set is a set of room photographs (there may be multiple room photographs for one floor plan) and information on the room (room type), and the learning method or convolutional neural network thereof. The configuration is not particularly limited except that it is limited by other descriptions of the present specification, and standard deep learning (for example, standard distance learning) including the method described in Non-Patent Document 1 is used. Can be used. Here, in the present invention, the distance learning means a general learning method of a neural network that maps items that are known to correspond to each other close to each other and items that are known to not correspond to each other to a distance. For example, even if the weight parameter is automatically adjusted (learning) so that the vector of the feature amount of the corresponding pair becomes small, and the weight parameter is automatically adjusted so that the distance of the feature amount vector of the uncorresponding pair becomes large. good. As an example, a contrastive loss can be used. Contrastive loss is one of distance learning, which minimizes the output value (L) of the loss function based on paired data (for example, two image data) and learns a neural network (automatic adjustment of weights). I do. In addition, Triplet Loss can also be used. Triplet loss is one of the distance learning and is a method of automatically adjusting the weight based on 3 data. Specifically, first, the central one (“central data”) among the three data is determined. Next, for the remaining two data, data that is relatively similar to the center data and data that are not similar to the center data are designated. The neural network calculates the distance between the center data and the remaining two data, and updates the distance between similar ones to be larger than the distance between dissimilar ones.

The learned content is stored in the storage unit 33 as a trained model, which means that the storage unit stores information on the first N objects (for example, room content, room information, objects in the room). It is expressed that it remembers the trained model G based on the correspondence (that is, the learning set) between the information (information about the contents of the layout) and one or more photographs taken of each of the first N objects. You can also do it.

[Information processing of embodiment 1]
Hereinafter, the details of the information processing in the first embodiment will be described. The outline of the information processing of the first embodiment is described by the control unit 32 of the information processing apparatus 3 based on the trained model, the floor plan 20 of a predetermined room, and one or more room photographs 10 of the room. In the floor plan 20, the room photograph 10 estimates at which position the room photograph 10 was taken and in which direction the room photograph 10 was taken. The above-mentioned "floor plan 20 (image)" of a predetermined room and "room photograph 10" of the room are merely specific examples for explaining the present invention. The present invention itself is not limited to estimation of a "floor plan (image)" of a predetermined room and a "room photograph" of the room, but is not limited to a "plan view" of a predetermined object (s) and the subject. It also includes the form of estimating and judging the relationship between "photographs" of objects (s). Further, when the first information and the second information exist and the dimensions of the feature quantities of the two information are different, the form of estimating or determining the relationship between the two information is also included.

FIG. 7 shows an information processing step according to an embodiment of the present invention. Each step (step 1 to step 7) described in FIG. 7 will be described below.

[Details of information processing steps]
[Step 1]
In the first embodiment, first, the room photograph 10 to be estimated and the floor plan corresponding to the room photograph 20 are input to the information processing apparatus 3 (“step 1”). There are no particular restrictions on the input method. For example, the room photograph 10 to be estimated and the floor plan 20 corresponding to the room photograph 10 are stored in an external storage device such as a memory card or a USB memory, and the external storage device is directly connected to the information processing device 3. May be done. In this case, the received room photograph 10 and the floor plan 20 may be stored in the storage unit 33 or may be stored as they are in the external storage device.

In addition, for example, the information processing apparatus 3 may receive the room photograph 10 and the floor plan 20 from the external network 4 via the communication unit 31. In this case, the received room photograph 10 and the floor plan 20 are stored in the storage unit 33. Considering the information processing system 1 as a whole, not only the form in which the floor diagram 20 stored in advance in the storage unit 33 of the information processing device 3 is selected in the user terminal device 5 (for example, a smartphone), but also the layout diagram in the terminal device 5. It means that 20 may be photographed (scanned) or input and uploaded to the information processing apparatus 3, and then the room photograph 10 is photographed by the camera of the smartphone and the room photograph 10 is photographed. Can also be realized in the form of uploading to the information processing apparatus 3. In the following, a case where a panoramic image is received from the external network 4 as a room photograph 10 will be described.

[Step 2]
In step 2, the control unit 32 of the information processing apparatus 3 reads the room photograph 10 (panoramic image). Here, in one example, if the room photograph 10 (panoramic image) input to the information processing device 3 (that is, received by the information processing device 3) is a grayscale image, it is read as it is and information processing is performed later. , If it is a color image, read it in grayscale (“Step 2”).

[Step 3]
From the read room photograph (grayscale image), the control unit 32 extracts the object information (information in the channel direction, that is, the tensor information on the first floor) in the room photograph (panoramic image) based on the trained model. At the same time, a calculation is performed to extract the position information (information in the width direction, that is, the first-order tensor information) of each object in the panoramic image (“step 3”). The behavior at this time is further supplemented.

As an example, assuming that the resolution of the read grayscale panoramic image is H (height) = 256 and W (width) = 512, the following information in the width direction (that is, position information) The extraction will be supplementarily explained. As a matter of course, the resolution of the loaded grayscale panoramic image is not limited to such a resolution. FIG. 8 shows a diagram of a supplementary explanation for extracting information in the width direction. First, the control unit 32 convolves the grayscale panoramic image. By the convolution, for example, data of H (height) = 8, W (width) = 16, C (number of channels) = 1080 is converted. The control unit 32 may drop the converted data as information by averaging the information in the height direction. Then, by averaging the information in the height direction, it can be converted into data of W (width) = 16 and C (number of channels) = 1080. Here, the fully connected layer has W (width) = 16 and C (number of channels) = 64, and by vectorizing these, C (number of channels) = 1024 can be set. FIG. 9 shows, for example, the steps when it is assumed that the object exists on the left side of the room photograph 10 in FIG. 8, and the object positions are shown in shading. The part corresponding to the object in the data after each of the above processes is also shown in shading.

[Step 4]
The information in the channel direction corresponding to the object in the room photograph 10 and the information in the width direction corresponding to each position of the object in the panoramic image are stored in a predetermined file 100, and are stored in the room photograph 10 (panoramic image). It is associated and stored in the storage unit 33.

[Step 5]
The layout diagram 20 input to the information processing apparatus 3 is folded and encoded. In one example, the floor plan 20 is encoded using a MobileNet-based architecture. As a result, a considerable amount of calculation can be reduced. In addition, while the normal convolution performs the convolution in the spatial direction and the convolution in the channel direction at the same time, in one embodiment of the present invention, the convolution is performed in the width direction (position information of each object in the panoramic image) by using MobileNet or the like. After that, the channel direction (that is, the information of the object in the image) is performed. That is, the convolution in the spatial direction and the channel direction is not performed at the same time, but is performed in the order described above. The architecture used for convolution is not limited to MobileNetb, but if MobileNet is used, there is no limitation on the version. Preferably, MobileNetV2 is used instead of MobileNetV1. Generally, in convolution, the number of channels increases with each layer and the amount of parameters is squeezed, but MobileNet V1 finally has a shape of 7 × 7 × 1024, whereas MobileNet V2 has 7 × 7 ×. Since it is only up to 320, the number of parameters can be made smaller than that of MobileNet V1. Further, in relation to the estimation accuracy provided in the present invention, even if the floor plan is compressed to the extent that it is compressed by MobileNet V2, no particular big problem occurs.

[Step 6]
The information obtained in step 5 and the feature amount (information in the channel direction corresponding to the object in the image) in the room photograph 10 (panorama image) stored in the file 100, and the position of the object in the panorama image. (Information in the corresponding width direction) is associated with each other by concating (vertically connecting) each other. In one example, the concut is performed multiple times (preferably twice).

In this respect, while the floor plan is a tensor on the third floor, the information in the channel direction corresponding to the object in the image and the information in the width direction corresponding to each position of the object in the panoramic image are both on the first floor. It is a tensor. Therefore, these cannot be directly concut. By dropping the tensor on the 3rd floor to the tensor on the 1st floor (that is, the information on the tensor on the 3rd floor that does not correspond to the tensor on the 1st floor will not be converted). Although it is not unthinkable to use a method of concutting the image, spatial information will be lost, and there will be a problem in specifying the shooting position and shooting direction. Therefore, as described below, the present inventors concatenate the tensor information on the first floor (information in the channel direction and / or information in the width direction) and the tensor information on the third floor (information on the floor plan). At that time, we came up with a method to perform concat without losing the spatial information held by the tensor information on the 3rd floor. The method will be described below with reference to FIG.

The above method is a method in which a tensor on the first floor is roughly duplicated according to the shape of the tensor in the floor plan, and concuts are performed in the channel direction and the width direction, respectively. As described above, the information in the channel direction and the information in the width direction are both first-order tensors and are only C (number of channels), but this is converted into the amount of information (1, 1, C1). Then, assuming that the floor plan image is the amount of information of (H2, W2, C2), a plurality of (× H2) duplications (1, 1, C) of (1, 1, C) are performed in the H (height) direction with the floor plan image as a reference. Then, it is further duplicated (× W2) in the W (width) direction and converted into an information amount (H2, W2, C1). As a result, the information in the channel direction and the information in the width direction are converted into information of the same dimension as the third-order tensor called (H2, W2, C1), respectively. Finally, (H2, W2, C2) and (H2, W2, C1) are concated assuming that they are all tensors on the third floor.

Here, in Non-Patent Document 1, the concut was performed based only on the channel information. However, the present inventors have found that when the concat is performed based only on the channel information, the information on the positional relationship is insufficient, and as a result, the accuracy of the estimation according to the present invention cannot be guaranteed. Therefore, the present inventors have come up with a technique that the accuracy can be remarkably improved by using the information in the width direction for the comcut.

[Step 7]
The combined information is decoded, output as a heat map of the same size as the floor plan, and evaluated. These can be performed by existing techniques, for example, by an architecture based on a U-shaped convolutional network (U-Net). However, in one example, at this time, the position information (two-dimensional) and the direction information (one-dimensional) are separately output as a heat map. This makes it possible to evaluate the shooting position information (two-dimensional) and the shooting direction information (one-dimensional) from the heat map (that is, compare by the difference between the integrated values of the probability distributions of the controls).

In one example, Jensen-Shannon (JS) diversity is used as the Loss in the estimation of the shooting direction and the estimation of the shooting position, that is, the Logarithmic Loss which is the cross entropy in machine learning. That is, when there are two probability distributions P (x) and Q (x) for the same probability variable x (ie, angle information), the distance (ie, difference) of these probability distributions is Kullback-Leibler (ie, difference). Although it can be evaluated using (“KL”) diversity, KL diversity does not meet the distance apocalypse due to its lack of symmetry (ie, it is not equivalent when two probability distributions are exchanged), but Jensen-Shannon (“KL”). JS ") divergence has symmetry and is suitable for the present invention.

In Non-Patent Document 2, which is a document related to machine learning, the absolute value of sin and the absolute value of cos are more important than learning the two signed sin and cos as outputs for (angle information) estimation. It is said that it was more accurate to learn to reduce the difference by using the sign of sin and the sign of cos as outputs. However, the present inventors have found that the accuracy can be significantly improved by using a one-dimensional heat map as an output for estimating the shooting direction.

[Additional information processing]
As a result, a set of one or more images, preferably a set of panoramic images showing a wide field of view, and a corresponding floor plan image are stored in the storage unit 33 of the information processing apparatus 3. Next, these one or more panoramic images are matched with a specific shooting position and shooting direction in the floor plan in which the panoramic image is taken. Based on the matching, a mark such as an arrow can be displayed on the floor plan (see the arrow in FIG. 1), and the user can display the set of panoramic images and the corresponding floor plan image.

[Additional configuration]
Once the shooting position and shooting direction of each of the panoramic images taken in the floor plan are specified, it is possible to create a virtual tour that provides an automatic and / or interactive tour of the real estate property. In the virtual tour, by receiving a signal from the user terminal device 5 that changes the direction (shooting direction) provided by the corresponding panoramic image in a specific room (position) in the floor plan visited in the virtual tour. It is also possible to provide panoramic images in different shooting directions. Further, in the information processing apparatus 3, it is also possible to edit a panoramic image, modify a photograph from a predetermined direction, and combine and display a photograph from a different direction. It is also possible to provide a standard field of view according to the viewing angle of the user by cropping or the like instead of the wide field of view image of the panoramic image.

As an example, the virtual tour may be an interactive tour, and for example, the user may be allowed to tour the inside of the room by using a head-mounted display or the like as the display unit 55 of the user terminal device 5. At that time, according to one embodiment of the present invention, since the shooting position and shooting direction of each panoramic image are accurately calculated and associated with the floor plan, the user virtually visits each room in a form corresponding to the floor plan. can do. In addition, the input unit 54 of the head-mounted display is configured to detect the line of sight of the user at a predetermined place so that the virtual tour can be paused, progressed, or switched to another viewpoint. It may be.

At this time, when entering the room B from the room A, the image of the room B seen from the room A is provided so that the user's experience can match the actual experience. The room photograph 10 in the most suitable shooting direction can be selected and displayed on the display unit 55 of the user terminal device 5.

[Embodiment 2]
As yet another embodiment, a method of specifying an object in both the room photograph 10B and the floor plan 20B and specifying the shooting position and the direction based on the positional relationship of the specified object is proposed. For example, in FIGS. 1 and 2, assuming that the shooting position, the position of the kitchen on the floor plan, and the kitchen are shown in front in the room photograph, the control unit 32 takes a picture based on these information. It is possible to estimate the direction.

Since the position (that is, the shooting position) on the floor plan on which the room photograph was taken can be specified by the method of the first embodiment, detailed description thereof will be omitted.

The position of the object on the floor plan can be specified not only by the method of the first embodiment but also by another method. For example, an object is identified using a trained model by a neural network, and the identified object corresponds to the trained model and is predetermined. Specific examples include, but are limited to, kitchen islands, doors and their doorknobs, walls, stairs, railings, toilets, sinks, bathtubs, showers, stoves, refrigerators, washing machines, shelves, etc. It's not a thing. In one example, by detecting the position of the doorknob, the right door and the left door are distinguished and specified.

[Modification 1]
The present invention is not limited to the above embodiment, and can be appropriately modified without departing from the gist. That is, according to one embodiment of the present invention, even if the dimensions of the feature quantities of the two inputs are different, both are controlled as described in FIG. 10 and the corresponding detailed description (for example, "step 6" above). Since it is possible to cut and evaluate, for example, an image and an explanatory text of the image (for example, "a dog is running on the left and a balloon is flying on the right") are input as inputs. In some cases, it is not a judgment of "whether the dog is running" or "whether the balloon is flying", but whether the image is as described in the explanation (that is, in the above example, "the dog is running to the left". It is possible to accurately determine (whether or not the image is a "balloon flying to the right"). In that sense, the present invention is not limited to information processing between a "floor plan" and a "room photograph", but is also limited to information processing between a "plan view" and a "photograph of an object". However, it relates to information processing when the dimensions of the feature quantities of the two inputs are different.

[Modification 2]
Further, the method of outputting and evaluating the angle information as a one-dimensional heat map described in the above "step 7" can be widely applied in other fields where accurate angle information is required. Specifically, as a modification, in the fields of robotics and automatic driving, a terminal device 5 equipped with a camera as a (terminal) input unit 54 (the "terminal device 5" in this modification is, for example, a robot equipped with a camera). And an automatic driving vehicle equipped with a camera), the image information around the terminal is given to the information processing device 3, and the image taken by the camera mounted on the terminal in the control unit 32 of the information processing device 3 In the form of calculating the current orientation of the terminal 5 based on the image information around the terminal, appropriately changing the orientation of the terminal for moving to the destination from there, and controlling the orientation of the terminal. Also contributes to improving the accuracy of the calculation of the angle information (that is, the orientation of the terminal). In this case, what is described as the floor plan 20 in the above embodiment corresponds to "image information around the terminal", and what is described as the room photograph 10 (or panoramic image) in the present embodiment is "mounted on the terminal". Corresponds to "images taken by the camera". In addition, the heatmap output is also useful in calculating which direction a person or object in a particular image is facing.

As described above, the present invention not only specifies the shooting position and shooting direction of the room photograph in the floor plan and associates the plan view of the object with the photograph of the object, but also associates the image with the description of the image. It is useful because it can be applied to correct / incorrect judgment and calculation of angle information (direction) related to robots and autonomous vehicles.

Information processing system 1
Room photo 10
Floor plan (image) 20
Information processing device 3
Communication unit 31
Control unit 32
Memory 33
Network 4
User terminal device 5
Communication unit 51
Control unit 52
Memory 53
Input unit 54
Display 55
File 100
Learning model G

Claims (6)

An information processing device having a storage unit and a control unit.
The storage unit performs machine learning by a convolutional neural network based on the correspondence between the plan view of the first N spaces and one or more photographs taken of each of the first N spaces. Remember the finished model,
The control unit extracts information obtained by compressing the photograph of the second space based on the trained model, the photograph of the second space different from the N spaces, and the plan view of the second space. Then, by combining the plan view of the second space with the compressed information, the shooting position and the shooting direction in which the photograph of the second space in the plan view of the second space is taken are estimated.
Information processing equipment. The estimation includes the control unit performing a Loss calculation using the Jensen-Shannon diversity.
The information processing apparatus according to claim 1. The estimation includes evaluating the result of the concut by a U-shaped convolutional network.
The information processing apparatus according to claim 1 or 2. Computer,
It has a storage unit and a control unit.
Learning that the storage unit performed machine learning by a convolutional neural network based on the correspondence between the plan view of the first N spaces and one or more photographs taken of each of the first N spaces. Remember the finished model,
The control unit extracts information obtained by compressing the photograph of the second space based on the trained model, the photograph of the second space different from the N spaces, and the plan view of the second space. Information processing that estimates the shooting position and shooting direction in which the photograph of the second space in the plan view of the second space is taken by combining the plan view of the second space with the compressed information. Operate as a device,
Computer program. The estimation includes the control unit performing a Loss calculation using the Jensen-Shannon diversity.
The computer program according to claim 4. The estimation includes evaluating the result of the concut by a U-shaped convolutional network.
The computer program according to claim 4 or 5.

Images