JP7383470B2 - Information processing device, information processing method and program - Google Patents

Description

One aspect of the present invention relates to an information processing device, an information processing method, and a program that present information representing the progress of work on a work target.

At construction sites and construction sites, it is common practice to take photos and videos of the site in order to manage the progress of work. For example, the administrator visually evaluates the progress of the work by looking at photos and videos.

Here, a technique is known in which images taken at a construction site are sent to a dedicated homepage and managed for progress management, troubleshooting, etc. (see, for example, Patent Document 1).
It is also known to take photographs of construction records using a 360° camera at construction sites, real estate properties, etc. (see, for example, Non-Patent Document 1).

Japanese Patent Application Publication No. 2003-87486

Regolith Co., Ltd., “Collaborating with “RICOH THETA” to release construction records in 360° photos/“Compatible with electronic delivery of maintenance work by the Ministry of Land, Infrastructure, Transport and Tourism””, [online], searched December 13, 2019, Internet <URL: https://spider-plus.com/news/update/231/>

With conventional technology, managers evaluate work progress in their heads based on their experience from images taken on-site.

This invention has been made in view of the above circumstances, and its purpose is to provide a technique that allows the progress of work to be easily and accurately grasped without relying on experience.

In order to solve the above-mentioned problems, a first aspect of the present invention provides an information processing device that presents information representing the progress of work on a work object, which is based on image data taken of a real space including the work object. a first image acquisition unit that acquires a first image representing the real space corresponding to a first photographing date and time, which is generated in a second image acquisition unit that acquires a second image representing the image; and an evaluation unit that calculates the progress rate of the work at the first photographing date and time by comparing the first image and the second image. and an output unit that generates and outputs information representing the progress rate , and the evaluation unit includes a plurality of first objects comprised of edges and surfaces included in the first image, A plurality of second objects comprised of edges and surfaces, which are included in the second image and correspond to each of the plurality of first objects, are identified, and the plurality of first objects and the plurality of The similarity between the objects is calculated by comparing corresponding second objects, and the weighted average of the similarities between the objects is calculated using weights set in advance for each type of object. The progress rate of the above-mentioned work is determined by calculation .

According to the first aspect of the invention, an image generated based on image data of a real space taken at a specific date and time and an image serving as a reference for work progress are acquired by the information processing device, By comparing them, the progress rate of the work at a specific date and time is determined and presented.
This makes it possible to grasp the progress of work and perform objective evaluations without relying on the experience of the administrator.

That is, according to each aspect of the present invention, it is possible to provide a technique that allows the progress of work to be easily and accurately grasped.

FIG. 1 is a diagram showing the overall configuration of a system including an information processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram showing the hardware configuration of an information processing apparatus according to an embodiment of the present invention. FIG. 3 is a block diagram showing the software configuration of an information processing apparatus according to an embodiment of the present invention. FIG. 4 is a diagram showing a processing sequence of an entire system including an information processing apparatus according to an embodiment of the present invention. FIG. 5 is a diagram showing an image of photographing a real space with a camera. FIG. 6 is a diagram showing an example of a location selection screen displayed on a user terminal. FIG. 7 is a diagram showing an example of a date and time selection screen displayed on a user terminal. FIG. 8 is a diagram showing an example of a captured image display screen displayed on a user terminal. FIG. 9 is a diagram showing an example of a comparison screen displayed on a user terminal. FIG. 10 is a diagram showing another example of the comparison screen displayed on the user terminal. FIG. 11 is a diagram showing an example of progress evaluation results displayed on a user terminal. FIG. 12 is a diagram showing an example of a table used for progress evaluation.

Embodiments of the present invention will be described below with reference to the drawings.
[One embodiment]
(composition)
(1) System FIG. 1 is a diagram showing the configuration of an entire system including an information processing device 1 according to an embodiment of the present invention.
This system connects an information processing device 1 and cameras CD1, CD2, . ．． ．． , CDm (hereinafter collectively referred to as "camera CD"), and user terminals UT1, UT2, . ．． ．． , UTn (hereinafter collectively referred to as "user terminals UT") are connected to each other so that they can communicate with each other. Note that an arbitrary number of cameras CD and an arbitrary number of user terminals UT may be connected to the information processing device 1.

The user terminal UT is, for example, a client terminal such as a personal computer, a mobile phone, a smartphone, or a tablet terminal, which is equipped with a browser or an application in its place. The user terminal UT is equipped with a display device using liquid crystal, organic EL (Electro Luminescence), or the like, or is used in combination with such a display device. According to one embodiment, the user terminal UT is used by, for example, a builder, a real estate sales agent, a housing business, etc., to manage and evaluate the progress of work such as construction work or renovation work. The user terminal UT may also be used by the owner of the house to be worked on to check the progress of the work.

The camera CD uses, for example, a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The camera CD may be any camera, such as a digital camera, panoramic camera, or 360-degree camera, which has a still image or video shooting function, as long as it can take pictures of the real space including the work object. The camera CD may also be equipped with a communication function. According to one embodiment, a still image or a moving image (hereinafter simply referred to as an "image") taken by a camera CD is provided with information indicating the date and time of the photograph (hereinafter "date and time information") and identifying the real space in which the photograph was taken. information (hereinafter referred to as "location information") is added and transmitted to the information processing device 1 as image data. The location information may be added manually by the person taking the image, or may be added automatically as location information such as GPS information. The image data may further include information identifying the camera CD that took the image.

The image data may be directly transmitted from the camera CD to the information processing device 1 via the network NW, or may be transmitted via the user terminal UT. Note that the camera CD and the user terminal UT may be integrated, such as a smartphone with a camera function. The camera CD may transmit image data each time a photograph is taken, or the image data may be stored in the memory of the camera CD and transmitted at fixed intervals or in fixed amounts.

The network NW is composed of, for example, an IP network represented by the Internet, and a plurality of access networks for accessing this IP network.

(2) Information processing device The information processing device 1 is, for example, a server device placed on a cloud, and evaluates the progress of work on the work object based on image data taken of a real space including the work object, Output information showing progress. According to one embodiment, the work is work related to a building, such as construction work, interior work, and equipment work. However, the information processing device 1 is not limited to these, and can also be applied to other tasks that are executed according to preset deadlines and schedules. According to one embodiment, the work object is an object of construction in a building, such as columns, girders, beams, ceilings, walls, floors, electrical equipment, gas pipes, water pipes, optical fiber cables, air conditioning/air supply/exhaust equipment. etc. However, the information processing device 1 is not limited to these, and can be applied to any work target. According to one embodiment, real space refers to any space that includes these work objects.

According to one embodiment, the information processing device 1 has the function of a web server, and allows users to view photographs of construction work objects and information representing the progress of the work through a dedicated website. Provide services that make it possible.

(2-1) Hardware Configuration FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing device 1.
The information processing device 1 includes, for example, a hardware processor 20A such as a CPU (Central Processing Unit). A program memory 20B, a data memory 30, and a communication interface (I/F) 10 are connected to this hardware processor 20A via a bus 40.

Communication interface 10 includes one or more wired or wireless communication interfaces. The communication interface 10 enables the transmission and reception of information between the camera CD and other external devices such as the user terminal UT via the network NW. As the wired interface, for example, a wired LAN is used, and as the wireless interface, for example, an interface that adopts a low power wireless data communication standard such as a wireless LAN or Bluetooth (registered trademark) is used.

The program memory 20B is composed of a non-volatile memory such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive) that can be written to and read from at any time as a storage medium, and a non-volatile memory such as a ROM. Stores programs necessary to execute various control processes related to the configuration.

The data memory 30 is composed of a nonvolatile memory such as an HDD or SSD that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory) as a storage medium. It is used to store various data acquired and created.

(2-2) Software Configuration FIG. 3 is a functional block diagram showing the software configuration of the information processing device 1 according to an embodiment of the present invention in association with the hardware configuration shown in FIG. 2. The control unit 20 includes the hardware processor 20A and the program memory 20B.
The communication interface 10 includes a receiving section 11 that receives various data from an external device via the network, and a transmitting section 12 that transmits various data to the external device via the network. For example, the receiving unit 11 receives image data from a camera CD and passes it to the control unit 20. The receiving unit 11 also receives a user request from the user terminal UT and passes it to the control unit 20. This user request may be, for example, an HTTP (Hyper Text Transport Protocol) request. The receiving unit that receives image data and the receiving unit that receives user requests may be provided separately. The transmitter 12, as an output unit, receives a response message output from the control unit 20 and transmits it to the requesting user terminal UT. This response may be, for example, an HTTP response.

The storage area of the data memory 30 is provided with an image data storage section 31 and a table storage section 32.

The image data storage unit 31 is used to store image data taken by the camera CD or various image data generated based on the image data. Image data is stored in association with date and time information and location information.

The table storage unit 32 is used to store an evaluation table used to determine the progress rate.

Note that the storage units 31 and 32 are not essential components, and may be provided in a storage device such as a database server located in the cloud.

The control unit 20 includes an image processing section 21, a user request reception section 22, a display image acquisition section 23, a progress evaluation section 24, and a display data generation section 25 as processing function sections using software. These processing functional units are realized by causing the hardware processor 20A to execute programs stored in the program memory 20B. Control unit 20 may also be implemented in a variety of other formats, including integrated circuits such as application specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs).

The image processing section 21 receives image data from the receiving section 11 and writes them into the data memory 30. According to one embodiment, the image processing unit 21 extracts date and time information and location information from image data, performs predetermined processing, and then associates the image data with the date and time information and location information and stores the image data. It is stored in the section 31.

The user request receiving unit 22 receives a user request from the receiving unit 11, analyzes it, and performs processing according to the request. According to one embodiment, the user request includes a request to obtain data necessary for viewing a particular web page, a request to obtain a particular image, and a request to evaluate the progress of the task. The user request reception unit 22 receives the user request, extracts necessary identification information, and passes it to the display image acquisition unit 23.

The display image acquisition section 23 acquires or generates a necessary image based on the information received from the user request reception section 22, and performs a process of passing the image to the progress evaluation section 24 or the display data generation section 25. According to one embodiment, the display image acquisition unit 23 receives a request to acquire an image corresponding to a specific location and date, reads out the corresponding image data from the image data storage unit 31, and sends the image data to the display data generation unit 25. hand over. Alternatively, the display image acquisition section 23 receives a request for evaluating the progress of the work, reads out image data necessary for evaluating the progress from the image data storage section 31, and passes it to the progress evaluation section 24.

The progress evaluation section 24 performs a process of evaluating the progress of the work based on the image data received from the display image acquisition section 23 and passing the evaluation result to the display data generation section 25. According to one embodiment, the progress evaluation unit 24 compares two pieces of image data received from the display image acquisition unit 23 and calculates their similarity. Then, the progress evaluation unit 24 determines the progress rate of the work as information representing the progress of the work based on the calculated similarity, and passes the evaluation result including the progress rate to the display data generation unit 25. The progress rate expresses the degree of progress of work as a percentage, and may be determined based on the state in which all work has been completed, or may be determined based on the expected state of work at each evaluation point. good.

The display data generation section 25 generates a response message including display data in accordance with the user's request based on the image data received from the display image acquisition section 23 and passes the response message to the transmission section 12 . The display data generation section 25 also receives the evaluation results from the progress evaluation section 24 , generates a response message including display data to be presented to the user, and performs a process of passing the response message to the transmission section 12 . Display data may include character strings and images.

(motion)
Next, an overview of the operation of the entire system including the information processing device 1 configured as described above will be explained.
FIG. 4 is a diagram showing an example of the processing sequence of the entire system.

(1) Collection and accumulation of images First, the information processing device 1 collects and accumulates image data obtained by photographing real space from a camera CD. Although only one camera CD is shown in FIG. 4 for convenience, the information processing device 1 can collect image data from a plurality of camera CDs.

In the arrow C1, the camera CD photographs the real space including the work object by the operation of a worker at the construction site, and transmits the generated image data to the information processing device 1.

According to one embodiment, the camera CD is a 360 degree camera operable in conjunction with a smartphone. A 360-degree camera, for example, is equipped with an ultra-wide-angle fisheye lens that extends over 180 degrees in the front-rear direction, and can simultaneously capture images or videos in the front-rear direction. The 360-degree camera also performs pre-processing including tilt correction, distortion correction, color correction, etc. based on the captured front and back images, and then stitches the images together using so-called "image stitching technology". Together, it is possible to generate 360-degree spherical image data.

FIG. 5 shows an image in which a camera CD, which is a 360-degree camera, photographs a space TG inside a building under construction, as a real space including a work object, and generates image data. The camera CD is fixed at an arbitrary position in the space TG, takes images in the front and back directions by remote control via a smartphone, and stitches the images in the front and back directions using the image stitching described above to generate 360-degree image data. do. The generated 360-degree image data is transferred from the camera CD to the smartphone using the WiFi function, for example, along with information indicating the date and time of the image. Location information for identifying the shooting location is added to the image data by a smartphone user (for example, the above-mentioned worker). The location information may include, for example, address, building name, room number, latitude and longitude, height, and the like. Also, some of this information may be automatically added to the image data by the camera CD itself. For a specific space TG, the image may be taken at one location, such as near the center of the space TG, or may be taken at a plurality of positions while moving at predetermined intervals.

The smartphone transmits image data to the information processing device 1 by, for example, uploading the image data to a dedicated web page in response to a user's operation. Each piece of image data to be transmitted includes at least location information and date and time information, as described above. The transmitted image data may also include information identifying the camera CD or smartphone that took the image, as well as GPS information regarding the location of the image.

When the information processing device 1 receives image data from the camera CD (or smartphone), in step S101, under the control of the image processing unit 21, the information processing device 1 associates the received image data with location information and date and time information, and stores the image data in the image data storage unit. 31 to be memorized. For example, the image data is stored in association with "A city, A prefecture, apartment X, room 102" as location information and "June X, 2019" as date and time information.

Here, according to one embodiment, the image processing unit 21 extracts features from a plurality of image data photographed at close photographing positions on the same day and associates them with each other, thereby creating a virtual 3D space representing the real space. You can also generate models. The image processing unit 21 can also cause the image data storage unit 31 to store the generated 3D spatial model in addition to or instead of the 360-degree image data.

In this manner, each time the information processing device 1 receives image data from the camera CD, it performs necessary processing and then stores the image data in the image data storage unit 31. Note that the information processing device 1 may store the received two-dimensional (2D) image (photo) data as 2D data. Further, the information processing device 1 may perform the stitching process for joining a plurality of images as described above.

Furthermore, according to one embodiment, the image data storage unit 31 also stores layout data representing the layout (floor plan) of a building or a room in association with location information. Image data is associated with the layout based on the generated three-dimensional (3D) spatial model. The layout data may be, for example, created by the information processing device 1 using commercially available floor plan creation software or the like based on image data or a 3D spatial model. Alternatively, the information processing device 1 may associate image data with pre-stored layout data based on features extracted from image data or a 3D spatial model. Alternatively, the operator of the information processing device 1 may read out the image data stored in the image data storage section 31 at any timing and manually associate it with the layout data.

The subsequent operations will be described below on the premise that a certain amount of image data has already been collected and stored in the information processing device 1 through the above-described processing.

(2) Provision of initial screen The information processing device 1 can provide a service for viewing stored images in response to a request from the user terminal UT. The user terminal UT used to view images may be the same or different from the user terminal used to upload image data. In the following example, the user terminal UT is a personal computer equipped with a display installed in a construction company and operated by a progress manager, and the user US is the progress manager.

For example, the user US attempts to check the progress of work at a certain work site. For the purpose of such confirmation, the user US operates the personal computer UT and starts the browser at the dashed line arrow U1 in FIG. 4 in order to view the web page provided by the information processing device 1. Then, the browser of the personal computer UT transmits a request to obtain the Web page to the information processing device 1.

In step S102, the information processing device 1 receives a web page acquisition request from the browser of the personal computer UT through the receiving unit 11. The receiving unit 11 passes this acquisition request to the user request accepting unit 22. The user request reception unit 22 extracts necessary information from the acquisition request, for example, information on the web page related to the request and browser information of the personal computer UT, and passes the extracted information to the display image acquisition unit 23. The display image acquisition section 23 reads the HTML file of the requested Web page from a storage section (not shown) in the data memory 30 and passes it to the display data generation section 25 . If there is image data necessary for displaying the requested Web page, the display image acquisition section 23 also reads it from the data memory 30 and passes it to the display data generation section 25 . The display data generation section 25 generates a response message including the above-mentioned HTML file and passes it to the transmission section 12.

In step S103, the transmitter 12 transmits the received response message to the browser of the personal computer UT.

The browser of the personal computer UT that received the message displays a corresponding web page based on the received HTML file in D1.

FIG. 6 shows an example of a display image of a web page provided by the information processing device 1. In the example of FIG. 6, on page 60, the name of the building "X Mansion" and the address "A City, Prefecture A..." are displayed in a box 61, and a photo of the exterior of the building is displayed in a box 62. Similarly, box 63 displays the name "YY Building" and address "B City, Prefecture B..." of another building, and box 64 displays a photo of the exterior of the building. According to one embodiment, links to other web pages are set in boxes 62 and 64, respectively, and user US operates mouse pointer 80 using an input device such as a mouse to select one of the By clicking inside the box, you can select the building you wish to view. Links may be set in the building name boxes 61 and 63. Furthermore, when displaying a Web page, the objects to be viewed may be restricted for each user according to his/her authority. Note that the mouse pointer 80 is only an example, and if the user terminal UT is a smartphone, it may be selected via a touch panel, or other functions such as keyboard operation, voice input, line of sight input, gesture input, etc. can be used in addition to mouse operation. It may also be input by operation.

(3) Selection of evaluation target The dashed arrow U2 in FIG. 4 indicates the target selection operation by the user US as described above. For example, as shown in FIG. 6, when the user US selects "X Mansion" displayed in the box 62, the browser of the personal computer UT accepts the user's selection operation and browses the linked page. The information processing device 1 is again requested to provide information for the processing.

In step S104, the receiving unit 11 of the information processing device 1 receives this acquisition request from the browser of the personal computer UT. The receiving unit 11 passes this acquisition request to the user request accepting unit 22.

Next, in step S<b>105 , the information processing device 1 extracts necessary information, for example, information on a linked page, from the acquisition request under the control of the user request reception unit 22 and passes it to the display image acquisition unit 23 .

In step S<b>106 , the information processing device 1 reads the HTML file of the linked page and necessary image data from the data memory 30 under the control of the display image acquisition unit 23 and passes them to the display data generation unit 25 . According to one embodiment, the display image acquisition unit 23 acquires, as necessary image data, image data for causing the user US to specify a room number to be viewed and date and time information to be viewed. The display data generation section 25 generates a response message based on the information received from the display image acquisition section 23 and passes it to the transmission section 12 .

In step S107, the transmitter 12 transmits the received response message to the browser of the personal computer UT.

The browser of the personal computer UT that received the response message displays the corresponding web page based on the received HTML file in D2.

FIG. 7 shows an example of a display image of such a web page. In the example of FIG. 7, on the web page 65, the name and address of the building currently being displayed are displayed in the box 66, "X apartment, A city, A prefecture...", and the viewable room number "Room 101" is displayed in the selection box 67. , Room 102,...'' is displayed. It may be possible to browse not only by room number but also by floor. Also, viewing authority may be set for each user.

As shown in FIG. 7, when "Room 102" is selected as a viewing target by the user US, the presence or absence of images that can be viewed for "Room 102 of Apartment X" is displayed in chronological order in a box 68. In this example, a black triangle pointing downward indicates that there is an image taken at that date and time. For example, when the user US places the mouse pointer 80 on one of the triangles, the corresponding date and time information is displayed as a pop-up, and by further clicking on the triangle, the user US can select the date and time to be viewed.

More specifically, in FIG. 7, when "Room 102" is selected by the user US through the browser of the personal computer UT, the process is again similar to that described with respect to arrow U2, steps S104 to S107, and D2 in FIG. processing is performed. For example, the user request reception unit 22 of the information processing device 1 extracts the location information “X Mansion, Room 102” from the acquisition request, and the display image acquisition unit 23 extracts the location information “X Mansion, Room 102” from the image data storage unit 31. Reads the date and time information of all the images that can be viewed and are linked to "Room 102". Then, the display data generating section 25 generates a display that allows the user US to view this date and time information on the time axis, and transmits it to the browser of the personal computer UT via the transmitting section 12. The browser of the personal computer UT displays a time axis display as shown in box 68 in FIG. 7 at D2.

Similarly, if the user US further selects "June Thereafter, at D2, a web page including an image corresponding to "X Mansion Room 102" on "June X" is displayed on the browser of the personal computer UT.

FIG. 8 shows an example of a display image of such a web page. In the example of FIG. 8, on the web page 70, box 71 displays the location information of the building being viewed, “X Mansion No. 102” and the address “A city, prefecture A...”, and box 72 displays the location information of the image being viewed. Date and time information “June X, 2019” is displayed. Box 73 displays image data (hereinafter referred to as "photographed image") corresponding to the location information and date and time information. The photographed image may be any image created based on image data of a real space including a work target at a specific place and at a specific date and time. In this example, the photographed image is an image created by cutting an arbitrary angle from 360-degree image data.

Also, at this time, the layout of "X Mansion Room 102" may be displayed based on the layout data created and stored in advance as described above. For example, links to image data taken in each room may be set in the layout display.

(4) Acquisition and Display of Comparison Images Next, if the user US wants to visually know the progress of work at a specified location and date and time, for example, click the “Compare” button 701 displayed as a GUI component on the web page 70. You can choose. For example, a link to a web page for displaying images in a comparative manner is set in the comparison button 701.

The dashed arrow U3 in FIG. 4 indicates that such a user request has been entered via the browser of the personal computer UT. Upon receiving this input, the browser of the personal computer UT transmits the above request to the information processing apparatus 1 along with date and time information and location information regarding the image being displayed.

In step S108, the information processing device 1 receives a user request from the personal computer UT using the receiving unit 11.

In step S<b>109 , the information processing device 1 receives this request under the control of the user request receiving unit 22 , extracts necessary information, and passes it to the display image obtaining unit 23 . According to one embodiment, the user request reception unit 22 includes, as necessary information, information specifying a link destination page and a captured image displayed on the browser at the time the user request is input to the personal computer UT. Contains information regarding an image to be used for comparison (hereinafter referred to as a "comparison image"). The comparison image is an image that serves as a reference for the progress of work on the work target. For example, the comparison image is an image representing a state in which the work on the work target has been completed, and may be an image created based on an estimated completion drawing or an estimated completion 3D model. The completed concept drawing may be BIM (Building Information Modeling) data or CAD (Computer-Aided Design) data.

Alternatively, the comparison image may be an image corresponding to the date and time created based on the work schedule, for example, an image expressing the expected progress expected at the date and time in virtual space. Such a comparison image may be created and stored in advance as an image representing work progress at multiple points in time using BIM data or CAD data, or may be created using machine learning based on the work schedule. It may be automatically created by, for example, Similarly, any image that represents the progress that the work should have made or the state that the work target should be in at the relevant date and time may be used. The comparison image may also be an image representing the completed state of each work step.

Alternatively, the comparison image may be an image taken of the same real space (for example, the same room in the same building) as the photographed image at a different date and time than the photographed image. The comparison image may also be an image taken of another room in the same building or a similar building at any shooting date and time.

The comparison image is stored in advance in the image data storage unit 31 in association with location information or location information and date and time information. When the comparison image is a completed rendering, one comparison image may be stored for specific location information (e.g., apartment 102), regardless of the date and time information of the photographed image, or multiple location information (e.g. A common comparison image may be stored for a plurality of rooms in the X apartment. What kind of comparison image to use may be arbitrarily set by the administrator of the information processing device 1 or the like. In the following, it is assumed that a computer graphics rendering of the completed room is set as a comparison image for each room.

The display image acquisition unit 23 reads the completed image data as a comparison image from the image data storage unit 31 based on the location information of the photographed image being displayed received from the user request reception unit 22.

In step S110, the information processing device 1 generates a response message necessary for displaying the web page, including the HTML file of the linked page, the photographed image being displayed, and the read comparison image.

Here, the information processing device 1 may adjust the camera angle and camera position in the 3D space of the captured image and the comparison image under the control of the display data generation unit 25. An example of such an adjustment process is a process in which features are extracted from both images and adjusted so that the angles and sizes of the features match in the displayed 2D image. An example of a method for extracting such features is a method of detecting edges or color changes. Detection of edges and color changes may be performed using any method. For example, it is possible to detect edges and color changes by converting a color image into a grayscale image and applying a certain bias to obtain a binary image (for example, see Japanese Patent Application Laid-Open No. 2014-219781). ).

Alternatively, objects in the image are recognized using a model trained in advance to recognize objects that may exist in the target image, such as windows, walls, floors, ceilings, etc., and image processing is performed based on the correspondence between them. It is also possible. Alternatively, when photographing with the camera CD, the photographing conditions may be adjusted according to a comparison image prepared in advance.

After making the above adjustments as necessary, the display data generation section 25 generates a response message and passes it to the transmission section 12.

In step S111, the transmitter 12 of the information processing device 1 transmits the received response message to the browser of the personal computer UT.

The browser of the personal computer UT that received the message displays the corresponding web page in which the image is embedded in D3.

FIG. 9 shows an example of a display image of such a web page. On page 75, the photographed image 100 is displayed side by side on the left side, and the comparison image 200 is displayed side by side on the right side. In this example, the photographed image 100 is an image in which a part of the 3D space is generated based on the 360-degree image photographed on June . The photographed image 100 also shows a large front window 102, a small window 103 on the left side, and a gypsum board 104 to which a wall cloth has not yet been attached, which exist in the real space. In this example, the comparison image 200 is an image based on a completed rendering of the room whose size and angle have been adjusted so that it can be compared with the photographed image 100, and information 201 indicating that it is a completed rendering is overlaid. be done. The comparison image 200 also shows a large front window 202 and a small window 203 on the left side. In the comparison image 200, the gypsum board 104 that appears in the photographed image 100 is not displayed, but a wall cloth is pasted thereon.

The image displayed as the photographed image 100 may be a photograph (2D image) taken with a normal camera rather than a 360-degree camera. The photographed image 100 may be created by adjusting (enlarging, reducing, trimming, etc.) photographed image data in order to correspond to the completed rendering. The comparison image 200 may be an image based on computer graphics, as described above, or may be based on still images or videos taken in other rooms of the same building or other similar buildings. good.

According to one embodiment, the information processing device 1 can redisplay the photographed image 100 and the comparison image 200 by changing the camera angle. For example, assume that the user US clicks near the left side of the photographed image 100, for example, around the small window 103, using the mouse pointer 80 in FIG. In this case, the browser of the personal computer UT determines that it has received an input instruction to change the angle to the left, generates a signal requesting a change of angle to the left, and transmits it to the information processing device 1 . Upon receiving this signal, the information processing device 1 changes the camera angle 90 degrees to the left in the 3D space for each of the captured image 100 and the comparison image 200 under the control of the user request reception unit 22 and the display image acquisition unit 23. Performs processing to create a rotated and cropped image. These images may be prepared in advance as images from a plurality of angles. Then, the information processing device 1 uses the display data generation unit 25 to generate a message including the created image and sends it again to the browser of the personal computer UT. The browser of the personal computer UT generates a display based on the received message. The above processing can be performed in the same manner as described with respect to U3, S108 to S111, and D3 in FIG. 4, for example.

FIG. 10 shows an example of an image that is redisplayed on the browser of the personal computer UT after the angle change instruction is given. On page 75 of FIG. 10, the photographed image 100 and the comparison image 200 are displayed side by side, similarly to FIG. Both the photographed image 100 and the comparison image 200 represent images cut out by rotating the angle 90 degrees to the left with respect to the 3D space shown in FIG. The small windows 103 and 203 that were displayed on the left in FIG. 9 are displayed on the front in FIG. In the photographed image 100, a gypsum board 104 with no wall cloth installed is displayed near the center thereof.

The web page 75 shown in FIG. 10 further includes a "progress evaluation" button 81, which is a GUI component for instructing progress evaluation. When the user US clicks the progress evaluation button 81 via the mouse pointer 80, the browser of the personal computer UT transmits a signal requesting the information processing device 1 to start evaluation of progress. The dashed arrow U4 in FIG. 4 indicates that the user US has entered this request into the browser of the personal computer UT. In step S112, the information processing device 1 receives this request under the control of the user request receiving unit 22, and starts evaluating the progress as follows.

(5) Evaluation of progress First, in step S113, the information processing device 1, under the control of the progress evaluation unit 24, specifies two images to be used for evaluating progress and performs a process of comparing them. According to one embodiment, the progress evaluation unit 24 acquires the identification information of the photographed image 100 and the comparison image 200 that were displayed by the browser of the personal computer UT at the time when the progress evaluation button 81 was pressed, and compares them. , and perform comparison processing.

For example, the progress evaluation unit 24 calculates the degree of similarity between two images as an example of comparison processing. For calculating the similarity, for example, SIFT (Scale-Invariant Feature Transform), AKAZE (Accelerated-KAZE), SURF (Speed-Up Robust Features), ORB (Oriented BRIEF), BRISK (Binary Robust Invariant Scalable Keypoints), etc. algorithm is used. In this case, the progress evaluation unit 24 extracts feature points from each of the two images using the above algorithm, and calculates the degree of similarity by determining the distance between each of the extracted feature points.

The progress evaluation unit 24 can also detect differences between two images using a background subtraction method. At that time, the background subtraction method may be used after feature point matching is performed using algorithms such as SIFT, AKAZE, SURF, ORB, and BRISK. Thereby, for example, as a difference between the photographed image 100 and the comparison image 200 in FIG. 10, it is possible to easily detect a portion where the wall cloth is not installed where the gypsum board 104 is exposed.

As another example of the comparison process, the following various methods can also be adopted. The first method is to extract features from a hidden layer of a trained CNN (Convolutional Neural Network) that performs object recognition on an input image, and calculate a degree of similarity. The second method is to create a histogram of pixel values of each image and calculate the degree of similarity by comparing the histograms. The third method is to calculate the degree of similarity between images by obtaining the correlation coefficient of each pixel value. The fourth method is to calculate the average value of the RGB values of each pixel for each image, and calculate the degree of similarity based on the distance between them.

Alternatively, the progress evaluation unit 24 may perform edge detection as described above. That is, the progress evaluation unit 24 can calculate the degree of similarity by binarizing both images, performing edge detection, and comparing the length, angle, number of pixels, etc. of the detected edges. Alternatively, the similarity may be calculated using a trained model that is trained in advance to output the similarity using a large amount of image data. Alternatively, each image may be divided into a plurality of segments, and the degree of similarity may be calculated for each divided segment using any of the various methods described above.

Next, in step S114, the information processing device 1 performs a process of evaluating the progress of the work based on the calculated similarity under the control of the progress evaluation unit 24. According to one embodiment, the progress evaluation unit 24 obtains a progress rate as information representing the progress of the work based on the calculated similarity. The progress rate may be determined using any method. For example, the progress evaluation unit 24 may use the calculated degree of similarity (for example, calculated as a percentage) as the value of the progress rate. Alternatively, the progress evaluation unit 24 may first perform object recognition such as a ceiling, floor, wall, window, etc., and then calculate the degree of similarity for each recognized object using the above-described degree of similarity calculation method. Then, the overall progress rate may be determined by averaging the degrees of similarity calculated for each object, or by performing a weighted average using preset weighting.

In step S115, the information processing device 1 generates a response message including display data for presenting the progress rate to the user as an evaluation result under the control of the display data generation unit 25.

Then, in step S116, the information processing device 1 transmits the response message to the browser of the personal computer UT via the transmitter 12.

The browser of the personal computer UT can display a web page based on the received message in D4.

FIG. 11 is an example of an image displaying evaluation results displayed by the browser of the personal computer UT. Box 82 displays "Progress rate is approximately 70%" as information representing the progress of the work.

Information indicating the progress of work is not limited to text only. For example, particularly different parts (parts with low similarity) may be highlighted. In the example of FIG. 11, in order to indicate areas where the degree of similarity is particularly low, parts of the photographed image 100 and comparison image 200 corresponding to the plasterboard 104 to which the cross has not yet been pasted are surrounded by broken lines 105 and 205, respectively. Highlighted. This can be done, for example, by first performing object recognition, calculating the similarity for each recognized object, and comparing the similarity with a predetermined threshold to identify objects whose similarity is below the predetermined threshold. It is carried out by Then, the display data generation unit 25 generates a display for emphasizing an object whose degree of similarity is less than a threshold value, and causes the display to be displayed superimposed on the image.

FIG. 12 shows an example of an evaluation table that can be used by the progress evaluation unit 24 to determine the progress rate. The evaluation table specifies a preset weighting coefficient for each object that can be recognized from the image data, and is stored in the table storage unit 32 by, for example, the administrator of the information processing device 1. For example, the progress evaluation unit 24 can calculate the degree of similarity for each recognized object, and multiply the calculated degree of similarity by the weight specified in this table to obtain the progress rate for each object. In the example of the evaluation table in FIG. 12, the "object" refers to a recognizable object, and includes, for example, a window, ceiling, floor, wall, etc. The "weighting coefficient" is arbitrarily set in advance for each object (object) in consideration of its importance, etc., and is set, for example, to a value of 0 or more and 1 or less. However, the present invention is not limited to this, and the weighting coefficient may be a value larger than 1, and the sum of all weighting coefficients may be 1 or may not be 1. “Similarity” is the degree of similarity for each object calculated by the progress evaluation unit 24 as described above. The "progress rate" here represents the progress rate for each object, and is determined by, for example, multiplying the "similarity" by the "weighting coefficient." In the example shown in Figure 12, the degree of similarity for the window is calculated as 83%, and the weighting coefficient for the window is set to 0.8 in advance, so the progress rate of the work on the window is 83 x 0.8 = 66.4%. It is calculated as follows. Note that the progress rate for each object does not necessarily need to be determined.

Further, the progress evaluation unit 24 can determine the overall progress rate using the evaluation table. For example, the progress evaluation unit 24 obtains the overall progress rate as a weighted average of the degrees of similarity calculated for each object. In the example of FIG. 12, the progress rate for each object (66.4%, 94.0%, 85.5%, 15.0%, ...) is calculated by multiplying the similarity of each object by a weighting coefficient. , by dividing the sum of these progress rates by the sum of weighting factors ((66.4+94.0+85.5+15.0+...)/(0.8+1+0.95+0.75+...)), the overall progress is calculated. A rate of ``70.2%'' is obtained. Alternatively, the sum of the progress rates of each object may be divided by the sum of the number of objects. Alternatively, the overall progress rate may be determined by simply calculating the average of the similarities of each object. Furthermore, the progress rate may be determined using another table or other formula.

(effect)
As described in detail above, in one embodiment of the present invention, an information processing device that presents information representing the progress of work on a work object is provided, and the information processing apparatus presents information representing the progress of work on a work object, based on image data taken of a real space including the work object. A generated first image representing the real space corresponding to the first photographing date and time is obtained, and a second image representing the real space is obtained as a reference for the progress of the work on the work object. , by comparing the first image and the second image, determine the progress rate of the work at the first photographing date and time, and obtain information representing the progress rate as information representing the progress of the work on the work target. is now output.

This makes it possible to perform an objective progress evaluation that does not depend on the experience or subjectivity of the evaluator, based on images of the object. Since the images are stored in association with the location and date/time information specified by the user, the captured images can be easily extracted based on the location and date/time specified by the user, and the progress rate of the work at that point can be determined.

Furthermore, in determining the progress rate, features are extracted from the first image and the second image, and the degree of similarity between the first image and the second image is calculated by comparing the extracted features. The progress rate of the work can also be determined based on the degree of similarity. Thereby, stable and objective evaluation results can be obtained by calculating the degree of image similarity.

Furthermore, in determining the progress rate, a first object included in the first image and a second object corresponding to the first object included in the second image are identified, and a second object corresponding to the first object included in the second image is identified. It is also possible to calculate the degree of similarity between the first object and the second object, and determine the progress rate of the work based on the calculated degree of similarity. In this way, when objects in images can be recognized, the similarity between comparable objects recognized in two images is calculated, and the progress rate is determined based on the calculated similarity. This makes it possible to determine the progress rate using clearer criteria.

Furthermore, if object recognition is possible, objects for which the degree of similarity between two images is less than a predetermined threshold can be highlighted. This makes it possible to clearly present objects with a low degree of similarity to the user, which can be useful for evaluating the progress of a task, such as identifying an object whose progress is delayed.

Further, in order to evaluate the progress rate, an image in a virtual space such as a BIM image or a CAD image may be used as an image to be compared with the photographed image. The comparison image may correspond to a conceptual drawing, or may represent a state in which each process included in the work plan or work schedule has been completed. Alternatively, the contrast image may represent the expected progress of the work at a particular point in time. Alternatively, the images may be images taken of the same real space at different dates and times, such as images taken at the start of construction or every month. The comparison image may be any image that represents what the progress of the work should be at a specific location and date and time. The photographed image and the comparison image may be displayed in a two-dimensional or three-dimensional manner.

Since a wide variety of images can be used in this way, it is possible to evaluate progress using an optimal comparison image depending on the object, content, purpose, etc. of the work or evaluation. Furthermore, by displaying an appropriate comparison image alongside the photographed image, it is possible to visually confirm the progress of the work schedule more accurately.

[Other embodiments]
Note that this invention is not limited to the above embodiments. For example, the information processing device 1 does not necessarily need to use 360-degree image data, nor does it necessarily need to create a 3D spatial model. The information processing device 1 can realize the above embodiment by collecting still images that can be compared with a comparison image.

Furthermore, when using an evaluation table such as the one described above to calculate the progress rate, different tables may be prepared for each type of construction work or construction method. Alternatively, the degree of similarity may not be calculated, and the two-choice determination of "work unfinished/work completed" may be simply performed for each recognizable object, and the progress rate may be determined according to the determination result.

Alternatively, instead of an evaluation table, a regression formula may be prepared, and coefficients may be calculated and used in advance by multiple regression analysis for each construction content and construction method. For example, it is possible to use the following regression equation.
Y=α ₁ X ₁ + α ₂ X ₂ + α ₃ X ₃ +...
In the formula, Y is the calculated progress rate, and X ₁ , X ₂ , X ₃ , . ．． ．． is the image similarity of each object. Coefficients α ₁ , α ₂ , α ₃ , . ．． ．． may be calculated in advance by multiple regression analysis, or may be arbitrarily set by the administrator of the information processing device 1 based on materials provided by building material/material manufacturers and construction companies.

Further, the display images shown in FIGS. 6 to 11 are merely examples, and various variations are possible. Some displays may be combined or some displays may be omitted. For example, the progress rate may be displayed immediately by pressing a comparison button 701 as shown in FIG. In addition, instead of reading out images and determining the progress rate each time the user US specifies, the progress rate is determined in advance by comparing each photographed image with a rendering of the completed image, and when the user US specifies the location and date, The calculated progress rate may be displayed.

In the above embodiment, the information processing device 1 has been described as including several functional units, but these are only examples of implementation of each functional unit. For example, multiple functional units described as being implemented in one device may be implemented across multiple separate devices, and conversely, multiple functional units described as being implemented across multiple separate devices may be implemented across multiple separate devices. It is also possible that the functional units described above are implemented in one device. Furthermore, each of the functional units described in the above embodiments may be realized using a circuit. The circuit may be a dedicated circuit that implements a specific function, or may be a general-purpose circuit such as a processor.

Furthermore, the flow of each process explained above is not limited to the procedures described, and the order of some steps may be changed, or some steps may be executed in parallel. . Further, the series of processes described above does not need to be executed continuously in time, and each step may be executed at any timing.

At least part of the processing in each of the embodiments described above can also be realized by using, for example, a processor installed in a general-purpose computer as the basic hardware. A program that implements the above processing may be provided while being stored in a computer-readable recording medium. The program is stored on a recording medium as an installable file or an executable file. Examples of recording media include magnetic disks, optical disks (CD-ROM, CD-R, DVD, etc.), magneto-optical disks (MO, etc.), semiconductor memories, and the like. The recording medium may be any medium as long as it can store the program and is readable by a computer. Further, a program for realizing the above processing may be stored on a computer (server) connected to a network such as the Internet, and may be downloaded to a computer (client) via the network.

In short, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the invention at the implementation stage. Moreover, each embodiment may be implemented in combination as appropriate, and in that case, the combined effect can be obtained. Furthermore, the embodiments described above include various inventions, and various inventions can be extracted by combinations selected from the plurality of constituent features disclosed. For example, if a problem can be solved and an effect can be obtained even if some constituent features are deleted from all the constituent features shown in the embodiment, the configuration from which these constituent features are deleted can be extracted as an invention.

1... Information processing device 10... Communication interface (I/F)
DESCRIPTION OF SYMBOLS 11...Reception part 12...Transmission part 20...Control unit 20A...Hardware processor 20B...Program memory 21...Image processing part 22...User request reception part 23...Display image acquisition part 24...Progress evaluation part 25...Display data generation part 30...Data memory 31...Image data storage unit 32...Table storage unit 40...Bus 60, 65, 70, 75...Web page 701...Comparison button 80...Mouse pointer 81...Progress evaluation button 100...Photographed image 200...Comparative image CD , CD1, CD2, CDm...Camera UT, UT1, UT2, UTn...User terminal TG...Space

Claims (9)

An information processing device that presents information representing the progress of work on a work target,
a first image acquisition unit that acquires a first image representing the real space corresponding to a first shooting date and time, which is generated based on image data taken of the real space including the work target;
a second image acquisition unit that acquires a second image representing the real space, which serves as a reference for the progress of the work on the work target;
an evaluation unit that calculates the progress rate of the work at the first photographing date and time by comparing the first image and the second image;
an output unit that generates and outputs information representing the progress rate ;
Equipped with
The evaluation unit includes a plurality of first objects composed of edges and surfaces included in the first image, and a plurality of first objects included in the second image corresponding to each of the plurality of first objects. , identify a plurality of second objects composed of edges and surfaces, and compare the corresponding ones of the plurality of first objects and the plurality of second objects to determine the similarity between the objects. The information processing device calculates the progress rate of the work by calculating the degree of similarity between the objects and calculating a weighted average of the degrees of similarity between the objects using weights set in advance for each type of object. further comprising an evaluation table storing preset weights for each type of object,
The information processing device according to claim 1, wherein the evaluation unit determines the weight used to calculate a weighted average of similarities between the objects. The evaluation unit extracts features from the first image and the second image, and compares the extracted features to evaluate the degree of similarity between the first object and the second object . The information processing device according to claim 1, wherein the information processing device calculates the progress rate of the work based on the calculated similarity. The output unit is configured to highlight the first or second object in at least one of the first image and the second image when the similarity is less than a predetermined threshold. The information processing device according to claim 1 , which generates and outputs information. the second image is an image representing the expected progress of the work object in the virtual space at the first photographing date and time;
The information processing device according to claim 1. The first image is an image that is created based on the image data and represents the real space two-dimensionally or three-dimensionally,
The second image is an image two-dimensionally or three-dimensionally representing the real space corresponding to a second photographing date and time different from the first photographing date and time, which is created based on the image data. be,
The information processing device according to claim 1. the second image is an image representing the work object in a virtual space corresponding to a state in which the work is completed;
The information processing device according to claim 1. An information processing method executed by an information processing device that presents information representing the progress of work on a work target, the method comprising:
a step of obtaining a first image representing the real space corresponding to a first shooting date and time, which is generated based on image data taken of the real space including the work object;
acquiring a second image representing the real space, which serves as a reference for the progress of the work on the work object;
determining the progress rate of the work at the first photographing date and time by comparing the first image and the second image;
a step of generating and outputting information representing the progress rate ;
Equipped with
The process of determining the progress rate includes determining a plurality of first objects comprised of edges and surfaces included in the first image, and a second image corresponding to each of the plurality of first objects. by identifying a plurality of second objects comprised of edges and surfaces included in the object, and comparing corresponding ones of the plurality of first objects and the plurality of second objects. The information processing method calculates the degree of progress of the work by calculating the degree of similarity between objects, and calculating the weighted average of the degrees of similarity between the objects using weights set in advance for each type of object. A program that causes a processor to execute processing by each part of the information processing apparatus according to any one of claims 1 to 7 .