JP7145602B2 - Information processing system, information processing method, and program - Google Patents

Description

The present invention relates to an information processing system, an information processing method, and a program.

Cracks may occur in the walls of buildings, etc., and water may leak inside the buildings due to rain or the like. In order to take countermeasures against such water leakage, it is desirable to be able to identify the degree of water leakage at the location where water leakage occurs.

In addition, in various inspection work, there is known an operation in which an image of an inspection point to be inspected is left, and aging of the inspection point is detected by visually comparing the remaining images (see, for example, Patent Document 1). ).

JP 2016-143983 A

However, although it is easy to identify the degree of water leakage if you are present when the water is leaking, it is difficult to identify the degree of water leakage when the water leakage has stopped at the location where the water leakage occurred. is.

The problem to be solved by the present invention is to provide an information processing system, an information processing method, and a program for identifying the degree of water leakage from an image of a water leakage location.

(1) An information processing system according to one aspect of the present invention makes it possible to identify the degree of leakage of rainwater or tap water that has occurred in a building. The information processing system finds stain-like traces of water leakage caused by the wetness of the interior material on the surface of the interior material at the water leakage location in the building, and detects the water leakage location photographed using a visible camera. a display control unit that acquires a plurality of images including a first image and a second image that are images showing the state of water leakage in the water leakage trace at different shooting times, wherein the feature points of the first image and Control is performed so that the first image and the second image are superimposed and displayed so that the feature points of the second image are associated with each other and the water leakage range common to the first image and the second image overlaps. and a display control unit, based on the leak marks shown in both the first image and the second image, quantifying a change in the size of the area identified by the leak marks in both the images. a quantification unit that quantifies the change in the size of the range identified by the quantified leak traces, a situation change determination unit that quantifies the rate of change based on the leak traces shown in the two images; A request processing unit that receives a request from a terminal device and performs processing according to the request, wherein the quantification result is sent to the request source terminal device that requests the quantification result. a request processing unit that provides the quantification result to the request source terminal device so as to be displayed on a display unit of the source terminal device . The display control unit provides the first image, which is a past image, to the terminal device in response to a request, and causes the monitor of the terminal device to transmit and display the first image, thereby capturing the second image. The second image captured by the imaging unit of the terminal device is obtained by enabling the position adjustment of the time, and the quantification unit determines a range common to the first image and the second image. using a plurality of reference points determined outside the water leakage trace, detecting changes over time in the brightness of the plurality of reference points at the plurality of reference points, and calculating changes in the water leakage trace. Then, the situation change determination unit detects the internal area of the trace of water leakage reflected in the first image and the second image, and quantifies the change in the detected area.

(2) In the information processing system described above, the quantification unit determines a change in the size of the range identified by the trace of water leakage as Calculated from the ratio of the inside and outside areas of the trace .

(3) In the above information processing system, the quantification unit determines a plurality of reference positions within a range common to both of the images, and determines the reference positions inside and outside of the water leakage trace related to the water leakage range. A change in size of the area identified by the leak mark is calculated based on the distribution of brightness at the position of .

(4) In the above information processing system, the quantification unit may measure the water leakage based on the brightness ratio of the reference position inside and outside the water leakage trace within a range common to both of the images. Calculate the change in size of the area identified by the trace of .

(5) In the information processing system described above, the situation change determination unit performs the quantification based on the size or area of the range identified by the water leak trace that has dried after getting wet once due to the water leak . The change in size of the area identified by the traced leak is shown as a percentage change in both images.

(6) Further, in the above information processing system, the water leakage point in the building is located on the frame side of the building that is hidden by the interior material of the building, and the image showing the water leakage situation in the building 2 is an image of a surface of an interior member of the building without an image of a water leak in the building;

(7) In the information processing system described above, the situation change determination unit may associate each of the plurality of images with the water leakage location in the building, and set information of each of the plurality of images, their feature points, and conditions at the photographing stage. is stored in the storage unit.

(8) In the above information processing system, the first image and the second image include an image of a subject that does not change depending on the water leakage situation in addition to the image showing the water leakage situation, and the display control unit and adjusting the first image and the second image so as to overlap each other based on the image of the subject that does not change depending on the water leakage state.

(9) Further, in the information processing method of one aspect of the present invention, when rainwater or tap water leakage occurs in a building, a computer of an information processing system that enables identification of the degree of the leakage is provided in the building. A stain-like water leakage trace caused by the wetness of the interior material is formed on the surface of the interior material at the water leakage location, and the image showing the water leakage situation at the water leakage location photographed using a visible camera. obtaining a plurality of images including a first image and a second image in which the traces of water leakage are photographed at different times, and making the feature points of the first image correspond to the feature points of the second image, controlling to overlap and display the first image and the second image so that the water leakage range common to the first image and the second image overlaps; quantifying a change in the size of the area identified by the leak marks in both of the images based on the leak marks shown in the images; and determining the size of the area identified by the quantified leak marks. a step of quantifying the rate of change based on the traces of water leakage shown in the two images ; receiving a request from the terminal device, performing processing according to the request, and obtaining the result of the quantification providing the quantification result to the requesting terminal device in response to a request from the requesting terminal device so as to display the quantification result on a display unit of the requesting terminal device; Accordingly, the first image, which is a past image, is provided to the terminal device, and the first image is transparently displayed on the monitor of the terminal device, thereby making it possible to adjust the position when the second image is captured. a step of acquiring the second image captured by an imaging unit of the terminal device; using the reference points to detect temporal changes in brightness of the plurality of reference points at the plurality of reference points to calculate changes in the traces of water leakage; and detecting an area inside the reflected trace of water leakage, and quantifying a change in the detected area.

(10) In addition, the program of one aspect of the present invention is a program that, when rainwater or tap water leakage occurs in a building, causes a computer of an information processing system capable of identifying the degree of the water leakage to occur in the building. A spot-like water leakage trace caused by the interior material getting wet is formed on the surface of the interior material at the location, and an image showing the water leakage situation at the water leakage location photographed using a visible camera. A plurality of images including a first image and a second image taken at different times of photographing the trace of the water leakage are acquired, and the feature points of the first image and the feature points of the second image are associated with each other to obtain the first image. Control is performed so that the first image and the second image are superimposed and displayed so that the water leakage range common to the first image and the second image overlap, and both the first image and the second image are displayed. quantify the change in the size of the area identified by the leak mark in both images, based on the leak mark shown in a step of quantifying a rate of change based on the traces of water leakage shown in the two images ; a step of providing the quantification result to the requesting terminal device so as to display the quantification result on a display unit of the requesting terminal device in response to a request from the requesting terminal device requesting The first image, which is a past image, is provided to the terminal device according to the above, and the first image is displayed on the monitor of the terminal device so that the position can be adjusted when the second image is captured. a step of acquiring the second image captured by an imaging unit of the terminal device; using the reference points to detect temporal changes in the brightness of the plurality of reference points at the plurality of reference points to calculate changes in the traces of water leakage; the first image and the second image a step of detecting the internal area of the trace of water leakage reflected in the image, and quantifying the change in the detected area.

ADVANTAGE OF THE INVENTION According to this invention, the information processing system, information processing method, and program which identify the grade of water leakage which occurred from the image of the location where water leakage occurred can be provided.

1 is a diagram for explaining an overview of an information processing system according to a first embodiment; FIG. 2 is a diagram for explaining a process of acquiring an image by applying the information processing system of FIG. 1; FIG. 4 is a diagram for explaining a history management DB of the embodiment; FIG. FIG. 10 is a diagram for explaining a process of recording an image by operating the mobile terminal device 10 of the embodiment; 4 is a flowchart showing a procedure of processing for recording an image by operating the mobile terminal device 10 of the embodiment; FIG. 4 is a diagram for explaining a process of recording an image by operating the information terminal device 30 of the embodiment; 4 is a flow chart showing a procedure of processing for recording an image by operating the information terminal device 30 of the embodiment. FIG. 5 is a diagram for explaining a contour identification function of the embodiment;

Hereinafter, an information processing system and an information processing method according to embodiments of the present invention will be described with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a diagram for explaining an outline of an information processing system according to an embodiment of the present invention.

The information processing system 1 shown in FIG. 1 includes a mobile terminal device 10 , a management device 20 and an operation terminal device 30 . The mobile terminal device 10, the management device 20, and the operation terminal device 30 are connected via a network NW by wireless communication.

If cracks occur in the outer wall or the like that forms the building BL, the waterproofness of the building BL will deteriorate. As a result, rainwater may flow through the cracks and enter the interior of the building BL, causing water leakage. If the location where the crack occurs is located on the padded wall covered with the interior of the building BL, it is difficult to directly visually recognize whether or not water leakage has occurred, or the extent of the water leakage.

By the way, the interior material of the building BL may get wet due to water leakage. In such a case, even if the rain stops and the water leakage stops and the interior material of the building BL dries, the area that was wet at the time of the water leakage remains as a stain. In addition, when the water leakage condition at the cracked portion worsens, the area where the interior material gets wet becomes wider, and accordingly the area where the interior material is stained becomes larger.

Therefore, the information processing system 1 of the embodiment indirectly identifies the occurrence of water leakage by detecting the range of stains on the interior material. For example, by adopting such an identification method, even if cracks in the outer wall of the building BL occur in places covered with the interior materials of the building BL, the information processing system 1 can detect the presence of water leakage. identifiable to what they do and to what extent. An example of the information processing system 1 will be described below.

FIG. 2 is a diagram for explaining a process of acquiring an image by applying the information processing system of FIG.

(mobile terminal device)
A mobile terminal device 10 shown in FIG. In the mobile terminal device 10, the communication unit 11, the storage unit 12, the control unit 13, the display operation unit 14, and the imaging unit 15 may be provided in one housing. It may be provided on the body. For example, the communication unit 11, the storage unit 12, and the control unit 13 of the mobile terminal device 10 may include a CPU (Central Processing Unit), a RAM (Random Access Memory), a nonvolatile storage device, a portable recording medium drive device, an imaging It may be configured as a computer including the unit 15 and the like.

The communication unit 11 is connected to the network NW by wireless communication and communicates with the management device 20 under the control of the control unit 13 .

The storage unit 12 stores image data captured by the imaging unit 15, information received from the management device 20, information to be displayed on the display/operation unit 14, information on operations detected by the display/operation unit 14, and the like. Programs (OS (operating system), application programs), etc. for functioning are stored.

The control unit 13 controls the communication unit 11 to communicate with the management device 20 . In this case, the control unit 13 communicates with the management device 20 using predetermined communication partner (destination) information including the URI of the management device 20 and the like.

For example, the mobile terminal device 10 communicates with the management device 20 by executing an OS or an application program. The mobile terminal device 10 can use a protocol such as HTTP (Hypertext Transfer Protocol) to browse content stored in the management device 20 or the like. For example, the content is described in HTML (HyperText Markup Language) or the like, image data, or the like. In addition to the method described above, a general method may be applied as the method for the mobile terminal device 10 to acquire the content or the like.

In addition, the control unit 13 acquires an image captured by the imaging unit 15 by controlling the imaging unit 15 according to user's operation. The control unit 13 controls so that information such as content and image data obtained from the management device 20 is displayed on the display operation unit 14 . The control unit 13 may control the image data obtained from the management device 20 to be displayed alone on the display operation unit 14 or to be superimposed on the image captured by the imaging unit 15 and displayed.

The display/operation unit 14 includes a liquid crystal display, an electroluminescence display, and the like as a display unit, and is also formed as a touch panel for accepting operations.

The imaging unit 15 includes an optical system and an imaging device (not shown), and generates image data based on a signal from the imaging device.

(Management device)
Next, the management device 20 will be explained.
The management device 20 shown in FIG. 2 includes a storage unit 21, a request processing unit 22, a data acquisition unit 23, a display control unit 24, a quantification unit 25, a situation change determination unit 26, a communication unit 27, A display unit 28 is provided. The management device 20 includes a storage unit 21, a request processing unit 22, a data acquisition unit 23, a display control unit 24, a quantification unit 25, a situation change determination unit 26, a communication unit 27, and a display unit 28. , may be provided in one housing, or a part thereof may be provided in a different housing. For example, the storage unit 21, the request processing unit 22, the data acquisition unit 23, the display control unit 24, the quantification unit 25, the situation change determination unit 26, and the communication unit 27 of the management device 20 are configured by the CPU, It may be configured as a computer including a RAM, a nonvolatile storage device, a portable recording medium drive device, and the like.

The storage unit 21 stores a history information DB 211 and the like.
FIG. 3 is a diagram for explaining a history management DB according to the embodiment;

The history management DB 211 stores building IDs, floors, floor plan IDs, room IDs, interior surface IDs, position data, original image IDs, shooting date data, shooting condition data, and range specific image IDs. , range data, and data such as change data.

The building ID stores data (such as BL01) for identifying the target building BL (target building). The floor stores data (F1, F2, etc.) for specifying each floor in the building BL. The floor plan ID stores floor plan data for each floor of the target building. For example, a drawing with a plan view ID of BL011 indicates a drawing (plan view) of the F1 floor of the building BL identified by the building ID of BL01, and a drawing with BL011 indicates a drawing of the F2 floor of the building BL. The room ID stores information for specifying the target room in the building BL. RM01 is an example of a room ID.

The interior surface ID stores data for specifying the ceiling (C), wall, floor (F), and the like. For example, a wall may be specified by applying an orientation (N/E/W/S) looking at the wall from the approximate center of the room.

The position data stores data indicating a position on a plan view. P01, P02, P03, etc. are examples of position data.

The original image ID stores the file name of the image data shot in the building. For example, an image data file is uniquely identified from this file name. Note that the image data of the image taken in the building may be stored together with the file name in the original image ID, or may be stored in another storage area. The photographing date data includes data specifying the year and month when the image was photographed in the building. The photographing condition data stores data indicating the surrounding environment of the target building on the day or the day before the image was photographed. For example, the shooting condition data includes weather information on the day or the day before the image was shot, lighting conditions of indoor lighting, and the like.

The range identification image ID stores the file name of the image data to which the result of identifying the leak range based on the above image data is added. For example, a file of image data to which the result of identifying the leak area is added is uniquely identified from the file name. Note that the image data to which the result of specifying the leak range may be stored together with the file name in the range specifying image ID, or may be stored in another storage area.

The range data stores data relating to the water leakage range specified based on the image data. For example, the data regarding the identified leak range is data indicating the position and range of the leak location in the image based on the image data.

The change data stores information about a change in the leak range detected based on the current image data with respect to the leak range detected based on the past image data.

For example, it is assumed that there are positions identified by P01, P02, P03, etc. in a room located on the F1 floor of the building BL identified by the building ID BL01 and identified by the room ID RM01. .
Until now, the file names of the images (IM1701 and IM1702) at the positions identified by P01 and P02 are stored in the original image ID. It can be identified from the items of the shooting condition data that the environment on the day these images were taken was "clear".
Note that the history management DB 211 has not yet stored the file name of the image at the position identified by P03 for the current year (NA).

Although the above describes an example of image data for the current year, image data for previous years are also stored in the history management DB 211 in the same manner. For example, image data with shooting date data of 2016M1, 2016M2, etc. are those of past years.

Returning to FIG. 2 , the request processing unit 22 and the data acquisition unit 23 control the communication unit 27 to communicate with the mobile terminal device 10 or the operation terminal device 30 . In this case, the request processing unit 22 and the data acquisition unit 23 communicate using predetermined partner (destination) information of the mobile terminal device 10 or the operation terminal device 30 .

For example, the management device 20 communicates with the mobile terminal device 10 or the operation terminal device 30 by executing an OS or an application program. The management device 20 uses a protocol such as HTTP (Hypertext Transfer Protocol) to provide content and image data stored in the management device 20 or the like so that they can be browsed from the mobile terminal device 10 or the operation terminal device 30. can be done. For example, the content is described in HTML (HyperText Markup Language) or the like, image data, or the like. In addition to the method described above, a general method may be applied as the method by which the management device 20 provides content and the like.

The request processing unit 22 receives various requests from the mobile terminal device 10 or the operation terminal device 30, causes each unit to execute processing according to the request, and returns a response to the request to each terminal device.

The data acquisition unit 23 acquires the image data of the leak location from the mobile terminal device 10, adds various related information, and writes and stores it in the history information DB 211. FIG.

In response to various requests from the mobile terminal device 10 or the operation terminal device 30, the display control unit 24 reads, from the history information DB 211 or the like, a drawing, an image, or the like of a position specified by a building, floor, room, position, or the like. , to each requesting terminal device.

The quantification unit 25 quantifies the leak range identified from the trace of the leak based on the image of the leak location. The details will be described later.

The situation change determination unit 26 determines a change in situation based on the rate of change (rate of change) of the leakage range.

The communication unit 27 is connected to the network NW by communication, and communicates with the mobile terminal device 10 or the operation terminal device 30 under the control of the request processing unit 22 and the data acquisition unit 23 .

The display unit 28 includes a liquid crystal display, an electroluminescence display, and the like, and is also formed as a touch panel for accepting operations.

(Operation terminal device)
The operation terminal device 30 shown in FIG. 2 includes a communication section 31, a storage section 32, a control section 33, and a display section . The configuration of each unit of the operation terminal device 30 conforms to the configuration of the mobile terminal device 10 .

FIG. 4 is a diagram for explaining processing for recording an image by operating the mobile terminal device 10 of the embodiment. FIG. 5 is a flowchart showing a procedure of processing for recording an image by operating the mobile terminal device 10 of the embodiment.

The mobile terminal device 10 accepts the user's operation, notifies the management device 20 of the information on the building selected by the user's operation, and requests the drawing information on the building (step SA11). The management device 20 receives the request from the mobile terminal device 10 (step SA21), and in response to the request, provides the mobile terminal device 10 with the drawing information related to the building selected by the user's operation (step SA22). .

The mobile terminal device 10 receives the drawing information related to the building selected by the user's operation from the management device 20 and displays it on the display operation unit 14 (step SA12). For example, FIG. 4A shows an example of a drawing (plan view BL01F1) displayed on the display unit. In this plan view BL01F1, markers (P01, P02, P03, etc.) are shown at positions where water leakage was detected in the past.

The user refers to the plan view BL01F1 displayed on the display unit and selects markers (P01, P02, P03, etc.) corresponding to desired positions marked on the plan view BL01F1. For example, it is assumed that the user has performed an operation to select the marker P03.

The mobile terminal device 10 receives the user's operation and requests the past image data of the selected desired position from the management device 20 (step SA13). The management device 20 receives the request from the mobile terminal device 10 (step SA23), and in response to the request, provides the mobile terminal device 10 with the past image data of the desired position selected by the user (step SA24). ).

The mobile terminal device 10 receives the past image data of the desired position selected by the user from the management device 20 and displays it on the display unit (step SA14). For example, FIG. 4B shows an example of an image (IM1601, etc.) displayed on the display operation unit 14 . This image IM1601 shows an image (IM1601PA) of the water leakage situation captured at a position where water leakage was detected in the past.

As shown in FIG. 4(c), the mobile terminal device 10 displays the past image data on the display/operation unit 14, executes the image capturing process, and captures an image with the image capturing unit 15 (step SA15). .
The mobile terminal device 10 uploads the image data of the image captured by the imaging unit 15 (step SA16). The management device 20 acquires the uploaded image data and stores it in the history information DB 211 (step SA26).

Note that when executing the photographing process in step SA15 described above, the mobile terminal device 10 sets either the past image or the current image IMNOW captured by the imaging unit 15 so that one of them is transparent. After adjustment, one image in which two images are superimposed is displayed on the display operation unit 14. - 特許庁Reference numeral SI1601 indicates the water leakage location in the past image, and reference numeral SI1701 indicates the water leakage location in the image of the current year. In FIG. 4(c), the current image IMNOW is indicated by a solid line, and the past image is indicated by a dotted line. As a result, the images of the subject are displayed in an overlapping state because the conditions such as the image pickup position and the image pickup system are the same. An image captured in such a state includes an image of the condition of the leakage location to be captured.

However, the mobile terminal device 10 is not installed at a fixed point, but is carried by the user. Therefore, even if the user attempts to align the position where the image is captured with the position where the image was captured in the past, it may be difficult to perfectly match the two images.

Therefore, in the present embodiment, as described above, correction will be described in the case where the images of the subject do not overlap completely due to a mismatch in the positions at which the images are captured.

(Detection of water leakage range)
FIG. 6 is a diagram for explaining processing for recording an image by operating the information terminal device 30 of the embodiment. FIG. 7 is a flow chart showing a procedure of processing for recording an image by operating the information terminal device 30 of the embodiment.

First, the operation terminal device 30 notifies the management device 20 of information about the building selected by the user's operation, and requests drawing information about the building (step SB31). The management device 20 receives the request from the operation terminal device 30 (step SB21), and in response to the request, provides the operation terminal device 30 with the drawing information related to the building selected by the user's operation (step SB22). .

The operation terminal device 30 receives the drawing information related to the building selected by the user's operation from the management device 20 and displays it on the display unit 34 (step SB32). For example, FIG. 6A shows an example of a drawing (plan view BL01F1) displayed on the display unit. In this plan view BL01F1, markers (P01, P02, P03, etc.) are shown at positions where water leakage was detected in the past.

The user refers to the plan view BL01F1 displayed on the display unit and selects markers (P01, P02, P03, etc.) corresponding to desired positions marked on the plan view BL01F1. For example, it is assumed that the user has performed an operation to select the marker P03.

The operation terminal device 30 receives the user's operation and requests the image data of the selected desired position from the management device 20 (step SB33). The management device 20 receives a request from the operation terminal device 30 (step SB23), and in response to the request, at least two or more image data shot at different times at desired positions selected by the user, It is provided to the operation terminal device 30 (step SB24).

The operation terminal device 30 receives from the management device 20 at least two pieces of image data photographed at different times at desired positions selected by the user, and displays them on the display unit 34 (step SB34). .

For example, FIG. 6B shows an example of images (IM1601, IM1701, etc.) displayed on the display unit . The above two or more images include an image IM1601 (hereinafter referred to as the first image) indicating the occurrence of water leakage detected in the past, and an image IM1601 detected after the past (when the image IM1601 was captured). and an image IM1701 (hereinafter referred to as a second image) showing the state of occurrence of water leakage. The management device 20 generates and transmits data for arranging the first image and the second image in separate areas on the screen displayed on the operation terminal device 30 . As a result, the operation terminal device 30 causes the image data of the first image and the second image to be displayed by dividing the image data into areas and arranging them side by side. It becomes possible to refer to the image and the second image.

Here, the user selects common feature points of the images included in both images, and designates the position of the selected feature points. For example, the user touches the position of the selected feature point on the screen of the display unit 34 so as to register the position of the selected feature point on each image. The feature points specified by the user may be selected from points on the housing of the lighting equipment, points on the boundary between panel materials stretched on the ceiling and walls, points on the frame provided at the inspection door, etc. . For example, the code TP indicates a feature point specified by the user. Note that the number of feature points TP may be determined as follows. The management device 20 equalizes the number of feature points TP received in response to user requests for each image. The number is, for example, at least two or more. In the illustrated example, four feature points TP are registered.

In this case, the operation terminal device 30 detects the position designated by the user's operation, and sends information regarding the registration request requesting that the designated position be associated with the first image and the second image. (Step SB35). Note that the above is performed after the operation of the icon EC31 (“alignment”) is detected.

In response to the request for registration, the management device 20 performs registration processing for associating the feature points TP of each image with each image (step SB25). Note that the management device 20 may perform the alignment process as follows. The management device 20 stores the position coordinates indicating the specific position in the history information DB 211 in association with the image data and the identification information of the feature point TP so that the feature point TP is associated with the specific position on the image. . Further, the operation terminal device 30 may request the management device 20 to adjust the two images (the first image and the second image) displayed side by side so that they can be seen on the same scale. The management device 20 receives the request from the operation terminal device 30 and adjusts the scale of at least one of the first image and the second image using the plurality of feature points TP. In this adjustment, the management device 20 may perform rotation of the image based on an arbitrary point, trapezoidal transformation, etc., in addition to adjusting the scale. The image denoted by reference IM1601C is the image IM1601 (first image) that has been adjusted as described above. The two images adjusted as described above have the same scale, direction, etc., and become images such that corresponding feature points TP are overlapped by parallel movement on the display unit.

The operation terminal device 30 requests the management device 20 whether to display the two images side by side or to display them in an overlapping manner. Receives a response with the requested image data. The operation terminal device 30 can appropriately request the switching of the display mode.

Next, after detecting the operation of the icon EC32 (“water leakage trace detection”) shown in FIG. A request is made to display an image with a clarified range (step SB36).

Upon receipt of the above request, the management device 20 generates image data that displays a display indicating the range where the stain occurs, as shown in FIG. 6(d), added to each of the above two images. . The management device 20 transmits the generated image data to the operation terminal device 30 . The management device 20 carries out the water leakage trace detection process (step SB26).

Next, the operation terminal device 30 receives a response from the management device 20 and displays an image that clearly indicates the range where the stain occurs in each of the above two images.

Next, operation terminal device 30 requests management device 20 to quantify the range in which stains occur in each of the above two images (step SB37).

Upon receiving the above request, the management device 20 counts the areas in which stains occur in each of the above two images, and performs a quantitative evaluation process of transmitting the result to the operation terminal device 30 (SB27).

For example, the operation terminal device 30 receives the result of the quantitative evaluation process and displays the result. For example, as shown in FIG. 6E, the results displayed on the operation terminal device 30 may include the area enlargement ratio of the area where the stain occurs in each image and the brightness change rate of the stain. good.

(Method for quantifying water leakage range)
Here, a method for quantifying the water leakage range will be described. The quantification unit 25 identifies the water leakage range in the image based on the trace of water leakage shown in the image, and quantifies the size of the identified range.

Furthermore, the quantification unit 25 quantifies the change in the size of the water leakage range in both images based on the common water leakage traces shown in the two images. In this way, by subjecting the common water leakage traces shown in the two images to the evaluation target, it becomes easy to distinguish between the states shown in the two images.

In addition, the situation change determination unit 26 quantifies the degree of change by indicating the calculated change in the leakage range by the rate of change in both images. For example, the quantification unit 25 calculates the ratio of the areas inside and outside the leakage range within the range common to both images, and compares the area ratios to calculate the degree of change in the leakage range. In this case, a plurality of reference positions are defined within a range common to both images, and the quantification unit 25 calculates changes in the leak range based on the distribution of brightness at the reference positions inside and outside the leak range. can be calculated.

For example, in an image having a range of a predetermined size, a plurality of reference positions are determined in a grid pattern. The grid spacing may be determined according to the required detection accuracy. The lattice may be associated with the imaging elements arranged at predetermined intervals in the imaging device of the imaging unit.

In the following, an example will be described in which the processing of the quantification unit 25 is performed using a deep learning technique for the identification function of estimating the outline of the subject. FIG. 8 is a diagram for explaining the contour identification function of the embodiment. The above identification function is known as semantic segmentation. Encoder/decoder networks, full convolution networks (FCN), and the like are known as techniques for realizing semantic segmentation.

The encoder/decoder network shown in FIG. 8(a) has an input layer and an output layer determined according to the required resolution. An encoder provided after the input layer includes a plurality of convolution layers and a plurality of pooling layers, acquires an input image by the input layer, and reduces the resolution of the input image. A decoder that precedes the output layer includes a plurality of depooling layers and a plurality of transposed convolution layers. The encoder and decoder are connected in a layer with reduced resolution, and combine information (pooling position information) etc. reduced from the input image by the encoder to generate a high-definition image up to the resolution of the output layer. . In this process, the encoder-decoder network estimates the region that satisfies the desired conditions as the desired region.

Also, the full convolution network shown in FIG. 8(b) has an input layer and an output layer determined according to the required resolution. A full convolution network includes a plurality of convolutional layers in the intermediate layer after the input layer. An input layer acquires an input image, and an intermediate layer down-resizes the input image. A full convolution network generates output layer data of the required resolution by deconvolution based on the data from each convolution layer. In this process, the full convolution network estimates a region that satisfies the desired conditions as the desired region.

Each of the encoder/decoder network and the full convolution network described above can be trained in advance using teacher data to improve their discriminating ability. As the training data, an image recording water leakage that occurred in the past may be used. For example, the user may identify the water leakage location in the image, create model data for the case of correct identification, and use it as teacher data. In this way, by using past image data as teacher data, it is possible to improve estimation accuracy.

The quantification unit 25 of the embodiment has an identification function of estimating the contour of the subject, so that even if the shape of the trace (contour) of the water leak to be detected is different, the characteristics different from the shape of the contour can be determined. Extract the range using
It should be noted that the quantification unit 25, which has learned using an image in which water leakage is recorded as training data, can extract even water leakage traces (contours) of various shapes. In this respect, it is possible to obtain an estimation accuracy different from that of pattern matching or the like.

According to the above embodiment, the display control unit acquires a plurality of images including the first image and the second image, which are images showing the state of water leakage at the water leakage location in the building BL and which are taken at different times. , the feature points of the first image and the feature points of the second image are associated with each other, and the first image and the second image are superimposed and displayed so that the leakage range common to the first and second images overlaps. The quantification unit 25 quantifies the change in the water leakage range in both the first image and the second image based on the traces of water leakage shown in the images of both the first image and the second image. , the degree of leakage that has occurred can be identified.

Further, the quantification unit 25 calculates the degree of change in the water leakage range from the ratio of the areas inside and outside the water leakage range within the range common to both images. Accordingly, a change in the water leakage range can be detected from the ratio of the areas inside and outside the water leakage range in the range common to the two images.

Further, the quantification unit 25 calculates a change in the water leakage range based on the brightness distribution of the reference positions within and outside the water leakage range, with a plurality of reference positions defined within a range common to both images. do. Accordingly, the change in the water leakage range can be calculated based on the distribution of brightness at a plurality of reference positions defined within a range common to both images. Also, the detection accuracy can be determined by the density of the reference positions.

In addition, the situation change determination unit 26 may indicate the calculated change in the leakage range by the rate of change in both images. As a result, even for small-scale leaks that tend to be overlooked in identification that prioritizes the size of leaks corresponding to both images, by focusing on the rate of change, it is possible to identify places where the state is suddenly changing. , can be identified as a portion with little change.

In addition, even if the leak location in the building BL is located on the side of the frame of the building BL that is hidden by the interior materials of the building BL, the image showing the state of the leak in the building BL will not show the location of the leak in the building BL. By using an image of the surface of the interior member of the building BL that does not include the image, it is possible to determine the state of water leakage without removing the interior member at the relevant position.

Note that the first image and the second image include an image of a subject that does not change depending on the water leakage situation, in addition to the image showing the water leakage situation. The display control unit adjusts the first image and the second image based on the image of the subject that does not change depending on the water leakage condition, so that two images are displayed based on the image of the subject that does not change depending on the water leakage condition. Facilitates contrasting two images.

(Second embodiment)
The situation change determination unit 26 of the second embodiment stores each of a plurality of images, their characteristic points, and set information of conditions at the imaging stage in the storage unit in association with the water leakage locations in the building BL. For example, the weather on the day before or on the day of the shooting is an example of conditions in the shooting stage.

For example, the quantification unit 25 may use a determination condition that matches the weather on the day before or on the day of photography. If the weather is rainy the day before or on the day of the shooting, water leakage may occur. If the interior material is not dry at the photographing stage, the image may differ from the dry state.

Therefore, the quantification unit 25 may adjust the determination criteria by using the information of the conditions at the imaging stage, such as the weather, for the determination.

According to the second embodiment, in addition to the effects similar to those of the first embodiment, the quantification unit 25 and the situation change determination unit 26 determine the situation change including conditions at the imaging stage. made available to

(Third embodiment)
The quantification unit 25 of the third embodiment calculates the change in the leak range based on the ratio of the brightness of the reference positions inside and outside the leak range in the range common to both images.

The quantification unit 25 determines a reference point near the leak location in the first image, and calculates the brightness ratio of the reference point in the second image corresponding to the first image. For example, when the water leakage situation worsens and the water leakage range widens, the outline (mark) moves outward. At this time, the brightness of the reference point changes due to the outline overlapping the reference point. The quantification unit 25 may detect this change in brightness.

According to the third embodiment, in addition to the same effect as the first embodiment, the quantification unit 25 reduces the computational load of calculating the brightness ratio at the reference position. It is possible to detect the deterioration of the water leakage condition with a few methods.

A program for realizing the information processing system 1 is recorded in a computer-readable recording medium, and the program recorded in the recording medium is read into a computer system and executed. A processing operation such as adding shared information may be performed on the contents of the . It should be noted that the "computer system" referred to here includes hardware such as an OS and peripheral devices. Also, the "computer system" includes a WWW system provided with a home page providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible discs, magneto-optical discs, ROMs and CD-ROMs, and storage devices such as hard discs incorporated in computer systems. In addition, "computer-readable recording medium" means a volatile memory (RAM) inside a computer system that acts as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. , includes those that hold the program for a certain period of time.

Also, the program may be transmitted from a computer system storing this program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in a transmission medium. Here, the "transmission medium" for transmitting the program means a medium having a function of transmitting information, such as a network or a communication line. Further, the program may be for realizing part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the functions described above in combination with a program already recorded in the computer system.

While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, as well as the scope of the invention described in the claims and equivalents thereof.

Reference Signs List 1 information processing system, BL building, 10 mobile terminal device, 20 management device, 21 storage unit, 22 request processing unit, 23 data acquisition unit, 24 display control unit, 25 quantification unit, 26... Situation change determination unit, 27 communication unit, 28 display unit, 30... Operation terminal device

Claims (10)

An information processing system in which rainwater or tap water leakage occurs in a building and the degree of the leakage can be identified,
On the surface of the interior material at the water leakage location in the building, a stain-like water leakage trace caused by the interior material getting wet is formed, and the water leakage situation at the water leakage location photographed using a visible camera is shown. A display control unit that acquires a plurality of images including a first image and a second image that are images and are taken at different time points of the traces of water leakage, wherein the characteristic points of the first image and the second image are obtained. a display control unit that controls to overlap and display the first image and the second image so that the water leakage range common to the first image and the second image overlaps with each other in correspondence with the feature points;
a quantification unit that quantifies, based on the water leakage traces shown in both the first image and the second image, a change in the size of the range identified by the water leakage traces in both of the images; ,
a situation change determination unit that quantifies a rate of change in the size of the range identified by the quantified leak traces based on the leak traces shown in the two images;
A request processing unit that receives a request from a terminal device and performs processing according to the request, wherein the quantification result is sent to the request source terminal device that requests the quantification result. a request processing unit that provides the quantification result to the request source terminal device so as to be displayed on a display unit of the source terminal device;
with
The display control unit
The first image, which is a past image, is provided to the terminal device in response to a request, and the first image is transparently displayed on the monitor of the terminal device, thereby enabling position adjustment when the second image is captured. and acquiring the second image captured by the imaging unit of the terminal device,
The quantification unit
brightness of the plurality of reference points over time using a plurality of reference points defined outside the water leakage trace within a range common to the first image and the second image; Detecting a change and calculating a change in the trace of water leakage,
The situation change determination unit
Detecting the internal area of the water leakage trace reflected in the first image and the second image, and quantifying the change in the detected area;
Information processing system. The situation change determination unit
The information according to claim 1, wherein a change in the size of the area identified by the water leakage mark is calculated from the ratio of the inner and outer areas of the water leakage mark related to the water leakage area within the range common to both of the images. processing system. The quantification unit
A plurality of reference positions are defined within a range common to both images, and the leak marks are identified based on the brightness distribution of the reference positions inside and outside the leak marks related to the leak area. 3. The information processing system according to claim 1 or 2, wherein the change in the size of the range is calculated. The quantification unit
4. Within the range common to both images, a change in the size of the range identified by the water leakage trace is calculated based on the brightness ratio of the reference positions inside and outside the water leakage trace. The information processing system according to . The situation change determination unit
The change in the size of the range identified by the quantified leak mark based on the size or area of the range identified by the leak mark that has dried after being wetted by the leak water 5. The information processing system according to claim 3, wherein the rate of change in the image is indicated. The water leakage point in the building is on the frame side of the building that is hidden by the interior member of the building,
4. The information processing system according to claim 3, wherein the image indicating the state of water leakage in the building is an image of the surface of the interior member of the building that does not include the image of the water leakage location in the building. The situation change determination unit
6. The information processing system according to claim 5, wherein each of the plurality of images and paired information of the characteristic points and imaging stage conditions are stored in a storage unit in association with water leakage locations in the building. The first image and the second image include, in addition to the image showing the water leakage situation, an image of a subject that does not change depending on the water leakage situation,
The information according to any one of claims 1 to 7, wherein the display control unit adjusts the first image and the second image so as to overlap based on an image of the subject that does not change depending on the water leakage situation. processing system. A computer of an information processing system that makes it possible to identify the degree of leakage of rainwater or tap water that has occurred in the building,
On the surface of the interior material at the water leakage location in the building, a stain-like water leakage trace caused by the interior material getting wet is formed, and the water leakage situation at the water leakage location photographed using a visible camera is shown. Obtaining a plurality of images including a first image and a second image in which the water leakage mark is photographed at different times, and matching the feature points of the first image with the feature points of the second image. and controlling to overlap and display the first image and the second image so that the water leakage range common to the first image and the second image overlap,
quantifying a change in the size of the area identified by the leak marks in both the first image and the second image based on the leak marks shown in both the first image and the second image;
quantifying a rate of change in the size of the area identified by the quantified leak signature based on the leak signature shown in both images;
receiving a request from a terminal device, performing processing according to the request, and displaying the quantification result on the requesting terminal device in response to a request from the requesting terminal device requesting the quantification result; providing results of the quantification to the requesting terminal for display on a unit;
The first image, which is a past image, is provided to the terminal device in response to a request, and the first image is transparently displayed on the monitor of the terminal device, thereby enabling position adjustment when the second image is captured. and acquiring the second image captured by the imaging unit of the terminal device;
brightness of the plurality of reference points over time using a plurality of reference points defined outside the water leakage trace within a range common to the first image and the second image; detecting a change and calculating a change in the leak mark;
an information processing method comprising: detecting an internal area of the water leak trace reflected in the first image and the second image, and quantifying a change in the detected area. In the computer of the information processing system that makes it possible to identify the degree of leakage of rainwater or tap water that has occurred in the building,
On the surface of the interior material at the water leakage location in the building, a stain-like water leakage trace caused by the interior material getting wet is formed, and the water leakage situation at the water leakage location photographed using a visible camera is shown. Acquiring a plurality of images including a first image and a second image in which the traces of water leakage are photographed at different times, and matching the feature points of the first image with the feature points of the second image. causing the first image and the second image to be superimposed and displayed so that the water leakage range common to the first image and the second image overlap,
Based on the water leak traces shown in both the first image and the second image, quantify the change in the size of the area identified by the water leak traces in the both images,
quantifying a change in the size of the area identified by the quantified leak signature by a rate of change based on the leak signature shown in both of the images;
receiving a request from a terminal device, performing processing according to the request, and displaying the quantification result on the requesting terminal device in response to a request from the requesting terminal device requesting the quantification result; causing the requesting terminal to provide results of the quantification for display on a unit;
calculating a rate of change in the size of the area identified by the quantified leak signature based on the leak signature shown in both images;
The first image, which is a past image, is provided to the terminal device in response to a request, and the first image is transparently displayed on the monitor of the terminal device, thereby enabling position adjustment when the second image is captured. and acquiring the second image captured by the imaging unit of the terminal device;
brightness of the plurality of reference points over time using a plurality of reference points defined outside the water leakage trace within a range common to the first image and the second image; detecting a change to calculate a change in the leak mark;
a step of detecting the internal area of the water leak trace reflected in the first image and the second image, and quantifying a change in the detected area.

Images