JP6937501B1 - Information processing method, information processing system, information processing device and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing method, an information processing system, an information processing device and a computer program for realizing a service for evaluating an air condition. SOLUTION: In the information processing method according to the present embodiment, a projection pattern corresponding to a projection destination in a building is projected by a projector, and light is blocked by a light blocking pattern corresponding to the projection pattern projected by the projector. An image taken with the projection destination on which the projection pattern is projected by the projector is performed by the photographing unit through the unit, and the state of air in the building is evaluated based on the image taken by the photographing unit. , The information processing device performs a process of outputting information on the evaluation result of the air condition. Further, the state of air in the building may be evaluated based on an image taken by the photographing unit with a predetermined gas flowing into the building. [Selection diagram] FIG. 7

Description

The present invention relates to an information processing method, an information processing system, an information processing device, and a computer program for evaluating the state of air in a building.

Outbreaks of the new coronavirus are having a major impact on the world economy. It is said that the situation of three-cs (sealed, dense, close) is likely to lead to outbreaks, and various measures are implemented in facilities such as restaurants or movie theaters to avoid the situation of three-cs. For example, in restaurants, measures are taken such as limiting the seats that can be used so as to keep a certain distance, or providing a threshold between the seats. In addition, for example, in trains or offices, measures are taken to circulate air, such as keeping windows open at all times. However, there is no way for the stores or facilities that have taken the measures to confirm whether these measures have really prevented outbreaks. For example, the spacing between seats in a restaurant, the position and size of the thresholds provided between the seats, etc. may not be sufficient to prevent splashing due to coughing or sneezing of those who use these seats. Further, for example, there is a possibility that air is not actually circulated even if the window is opened, and there is a possibility that air is carried from person to person by blowing air from a fan or an air conditioner (air conditioner).

In Patent Document 1, the solvent is evaporated on the upstream side of the airflow to include the solvent vapor in the airflow, and the concentration distribution of the solvent vapor in the airflow is optically visualized on the downstream side of the airflow to prevent the turbulence of the airflow. A method for visualizing airflow turbulence, which is visualized as the concentration distribution of solvent vapor, has been proposed. In this method, visualization is performed by the Schlieren method.

Japanese Unexamined Patent Publication No. 2008-64679

In order to confirm whether or not measures to prevent infectious diseases are sufficient, it is effective to observe, for example, the air flow or air circulation generated when a person coughs, sneezes, or talks. .. The Schlieren method, which is also used in the method for visualizing airflow turbulence described in Patent Document 1, is a method capable of visualizing changes in the refractive index of air or the like, for example, visualizing the flow of air in a room or the like. Can be expected. However, in order to perform visualization by the Schlieren method, it is necessary to construct a special optical system called the Schlieren optical system, and it is necessary to visualize the air so far and evaluate whether or not countermeasures against infectious diseases are sufficient. Was not done.

The present invention has been made in view of such circumstances, and an object of the present invention is an information processing method, an information processing system, an information processing device, and a computer program for realizing a service for evaluating an air condition. Is to provide.

The information processing method according to one embodiment is to obtain a projection pattern generated by adjusting the predetermined image so that the captured image obtained by capturing the projection pattern projected by the projector on the projection destination in the building matches the predetermined image. projected by the projector to the projection target through a light blocking section that blocks light by the light blocking pattern of the projector according to the projection pattern projected, the projector is the background of the projection target by projecting the projection pattern photographed Is performed by the photographing unit, the state of air in the building is evaluated based on the image taken by the photographing unit, and the information processing apparatus performs a process of outputting information regarding the evaluation result of the state of air.

In the case of one embodiment, it can be expected to realize a service for evaluating the state of air.

It is a schematic diagram for demonstrating the structure of the photographing apparatus used in the information processing system which concerns on this Embodiment. It is a block diagram which shows the structure of the information processing apparatus of the photographing apparatus which concerns on this embodiment. It is a schematic diagram which shows an example of a background image and a cut-off grid. It is a schematic diagram for demonstrating the modification of a projection image. It is a schematic diagram which shows an example of the image which visualized exhaled breath and the like. This is an image example for explaining the difference in the photographed image depending on the presence or absence of the polarizing plate. It is a flowchart for demonstrating the outline of the service provided by the information processing system which concerns on this Embodiment. It is a block diagram which shows the structure of the terminal apparatus which concerns on this embodiment. It is a block diagram which shows the structure of the server apparatus which concerns on this embodiment. It is a schematic diagram which shows one configuration example of an assessment DB. It is a schematic diagram which shows one configuration example of a monitor DB. It is a schematic diagram which shows one display example of the evaluation result by the terminal apparatus which concerns on this embodiment. It is a schematic diagram which shows one display example of the evaluation result by the terminal apparatus which concerns on this embodiment. It is a flowchart which shows the procedure of the process which the terminal apparatus which concerns on this embodiment performs in a ventilation assessment service. It is a flowchart which shows the procedure of the process which the terminal apparatus which concerns on this embodiment performs in a ventilation monitor service.

A specific example of the information processing system according to the embodiment of the present invention will be described below with reference to the drawings. It should be noted that the present invention is not limited to these examples, and is indicated by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

<Shooting device>
FIG. 1 is a schematic diagram for explaining a configuration of a photographing apparatus used in the information processing system according to the present embodiment. The information processing system according to the present embodiment uses a photographing device 1 that visualizes the air flow by the Schlieren method, and uses, for example, various types of restaurants, coffee shops, residential exhibition halls, showrooms, shopping centers, movie theaters, museums, and the like. The inside of the building is photographed, the state of the air inside the building is evaluated based on the image of the air flow in the photographed building, and the evaluation result is notified to the user such as the owner, user or builder of the building. .. Further, in the information processing system according to the present embodiment, dirt such as water droplets or leftover food adhering to the inside of the building is detected based on the image taken by the photographing device 1, and the detection result is notified to the user.

The photographing device 1 according to the present embodiment includes a projector 10, a first photographing unit (photographing unit) 11, a second photographing unit (photographing unit) 12, and an information processing device 20 that controls their operations. It is composed of. The projector 10 is a device that projects an image onto a screen or the like, and in the present embodiment, the projector 10 projects an image generated by the information processing device 20 onto a wall 101 or the like in a building to be evaluated. Further, the projector 10 according to the present embodiment projects an image by infrared rays as invisible light rays that cannot be seen by humans. Therefore, for example, even when the photographing device 1 is installed in a building such as a restaurant, the image projected by the projector 10 is not visually recognized by a customer or the like who is using the store.

The first photographing unit 11 is an imaging unit that photographs an image that visualizes the air flow such as exhalation by the Schlieren method. The first photographing unit 11 includes a lens 11a, a cut-off grid (light blocking unit) 11b, an infrared lens 11c, and the like. The lens 11a, the cut-off grid 11b, and the infrared lens 11c are housed side by side in, for example, a cylindrical case. The lens 11a is an optical element that refracts and converges light, and forms an image of an image to be photographed by the first photographing unit 11. The cut-off grid 11b cuts (blocks) a part of the light from the lens 11a to the infrared lens 11c. In the present embodiment, the cut-off grid 11b is configured by using, for example, a liquid crystal panel, and the pattern for cutting light can be changed. The information processing device 20 controls the cut pattern by the cutoff grid 11b. The infrared lens 11c transmits infrared rays (for example, light having a wavelength of 0.7 μm to 1000 micrometers) and does not transmit light other than infrared rays. The infrared lens 11c converges infrared rays that have passed through the cutoff grid 11b onto an image sensor (not shown). The first photographing unit 11 includes an image pickup element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and an infrared lens 11c captures a converged infrared image and obtains an image obtained by photographing. Data is output to the information processing device 20.

The second photographing unit 12 is an photographing unit that photographs a polarized state. The second photographing unit 12 includes a polarizing plate (polarizing unit) 12a and a visible light lens 12b. The polarizing plate 12a and the visible light lens 12b are housed side by side in, for example, a cylindrical case. The polarizing plate 12a is an optical element that allows light in a specific direction to pass through and blocks light other than that. The polarizing plate 12a according to the present embodiment has, for example, a rotation mechanism using a motor, and can be rotated according to the control of the information processing device 20. As a result, the second photographing unit 12 can rotate the polarizing plate 12a by an arbitrary angle and acquire a light-shielded image in an arbitrary direction. The visible light lens 12b transmits visible light (for example, light having a wavelength of 360 nanometers to 830 nanometers) and does not transmit other light. The visible light lens 12b converges visible light that has passed through the polarizing plate 12a onto an image pickup device (not shown). The second photographing unit 12 includes an image pickup element such as a CCD or CMOS, takes an image in which the visible light lens 12b converges, and outputs the image data obtained by the photographing to the information processing apparatus 20. The first imaging unit 11 and the second imaging unit 12 may be individually provided with an image sensor, or may share one image sensor.

The second imaging unit 12 rotates the polarizing plate 12a to perform imaging in a plurality of polarization directions, but the present invention is not limited to this. For example, by switching a plurality of polarizing plates 12a having different polarization directions, imaging may be performed in a plurality of polarization directions. Further, for example, an image sensor in which a polarizing element in a plurality of directions is arranged for each pixel may be used to perform imaging in a plurality of directions at a time.

Further, the photographing device 1 according to the present embodiment is a device that can be called by a name such as a so-called edge camera, an AI (Artificial Intelligence) edge camera, or an edge AI camera, and is equipped with a processor or the like that performs advanced arithmetic processing. The information processing device 20 is provided. As a result, the photographing device 1 can perform a process of photographing an image that visualizes the air flow, a process of changing the polarization state, and the like. The information processing device 20 controls the operation of each unit of the photographing device 1 and performs various processes using the images obtained by the images taken by the first photographing unit 11 and the second photographing unit 12. FIG. 2 is a block diagram showing a configuration of an information processing device 20 of the photographing device 1 according to the present embodiment. The information processing device 20 according to the present embodiment includes a processing unit 21, an input / output unit 22, a storage unit 23, a communication unit 24, and the like.

The processing unit 21 uses an arithmetic processing unit such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit) or a GPU (Graphics Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory) and the like. It is composed of. The processing unit 21 reads and executes the program 23a stored in the storage unit 23 to control the projector 10, the first photographing unit 11, the second photographing unit 12, and the like, and the image obtained by photographing. Performs various processes such as evaluation or detection based on.

The input / output unit 22 inputs / outputs data to / from the projector 10, the first photographing unit 11, and the second photographing unit 12. The input / output unit 22 is connected to the projector 10, the first photographing unit 11 and the second photographing unit 12 via a signal line, for example, and inputs / outputs data by serial communication or parallel communication via the signal line. .. The input / output unit 22 appropriately transmits the data given by the processing unit 21 to the projector 10, the first photographing unit 11, or the second photographing unit 12, and also appropriately transmits the data to the projector 10, the first photographing unit 11 or the second photographing unit 12. The data input from is given to the processing unit 21.

The storage unit 23 is configured by using a non-volatile memory element such as a flash memory or an EEPROM (Electrically Erasable Programmable Read Only Memory). The storage unit 23 stores various programs executed by the processing unit 21 and various data required for processing by the processing unit 21. In the present embodiment, the storage unit 23 stores the program 23a executed by the processing unit 21. Further, the storage unit 23 may store image data projected by the projector 10, image data captured by the first photographing unit 11 and the second photographing unit 12, and the like.

The communication unit 24 communicates with various devices via a network N such as a LAN (Local Area Network) or the Internet. In the present embodiment, the communication unit 24 communicates with the server device 3 or the terminal device 5, and transmits the captured image and information such as a processing result such as evaluation or detection based on the image. The communication unit 24 may perform either wired or wireless communication. The communication unit 24 transmits the data given by the processing unit 21 to another device, and gives the data received from the other device to the processing unit 21.

In the present embodiment, the program 23a is written to the storage unit 23, for example, at the manufacturing stage of the photographing apparatus 1. Further, for example, in the program 23a, the photographing device 1 may acquire what is distributed by the remote server device or the like by communication. Further, for example, the program 23a is provided in a mode of being recorded on a recording medium such as a memory card or an optical disk, and the photographing apparatus 1 may read the program 23a from the recording medium and store it in the storage unit 23. Further, for example, in the program 23a, the writing device may read out what has been recorded on the recording medium and write it in the storage unit 23 of the photographing device 1. The program 23a may be provided in a mode of distribution via a network, or may be provided in a mode recorded on a recording medium.

Further, in the processing unit 21 of the photographing apparatus 1 according to the present embodiment, the processing unit 21 reads out and executes the program 23a stored in the storage unit 23, so that the projection image generation unit 21a and the first photographing processing unit 21a are executed. The 21b, the second imaging processing unit 21c, and the like are realized as software-like functional units.

The projection image generation unit 21a performs a process of generating data of an image projected by the projector 10. The image projected by the projector 10 is used for the first imaging unit 11 to capture an image that visualizes the air flow such as exhalation by the Schlieren method. In the present embodiment, the pattern of the image projected by the projector 10 is, for example, a striped pattern (grid pattern) in black and white or monochrome, but the pattern is not limited to this, and various patterns (patterns) are adopted. Can be done. Further, in the present embodiment, the projection image generation unit 21a creates a striped pattern projected by the projector 10 according to the shape and pattern of the projection destination (wall 101 in the example shown in FIG. 1) on which the projector 10 projects an image. Generate a tuned image. The projection image generation unit 21a optimizes the shape and size of the striped pattern to be projected according to the shape and pattern of the projection destination, generates an optimized striped image, and gives it to the projector 10. , The generated image is projected onto the projector 10. In the present embodiment, the projector 10 projects a striped image with infrared rays.

The first photographing processing unit 21b performs a process of controlling photography by the first photographing unit 11. The first photographing processing unit 21b photographs an image in which the air flow is visualized by the first photographing unit 11. At this time, the first photographing processing unit 21b changes the pattern in which the cutoff grid 11b of the first photographing unit 11 blocks light according to the pattern (pattern) projected by the projector 10 on the projection destination wall 101 or the like. .. In the present embodiment, the first photographing processing unit 21b has a pattern in which the black and white of the striped pattern projected by the projector 10 is inverted, that is, a pattern that blocks a portion of the striped pattern projected by the projector 10 corresponding to white. The cutoff grid 11b controls the blocking pattern. By blocking the light by the cut-off grid 11b in this pattern, the first photographing unit 11 uses the Schlieren method to control the air flow and the like existing between the photographing device 1 and the wall 101 or the like on which the image of the projector 10 is projected. The image visualized by the above can be taken. In the present embodiment, the first photographing processing unit 21b photographs an image in which the air flow is visualized by the first photographing unit 11 as a moving image.

In the present embodiment, the cut-off grid 11b blocks light with a pattern obtained by reversing the pattern projected by the projector 10, but the present invention is not limited to this. The cut-off grid 11b may block light with an incomplete inversion pattern having some deviation or difference from the complete inversion pattern, instead of a perfect inversion pattern with respect to the projection pattern of the projector 10. Even in this case, the first imaging unit 11 can capture an image that visualizes the air flow. The cutoff pattern by the cut-off grid 11b may be any pattern as long as it can capture an image in which the air flow is visualized by the first imaging unit 11.

The second photographing processing unit 21c photographs the polarized state by the second photographing unit 12. At this time, the second imaging unit 21c controls the rotation angle of the rotation mechanism provided on the polarizing plate 12a of the second imaging unit 12 to perform imaging by the second imaging unit 12. The second photographing processing unit 21c rotates, for example, the rotation mechanism by a predetermined angle (for example, 1 ° or 10 °) to change the angle of polarization by the polarizing plate 12a by a predetermined angle, and the second photographing unit 12 takes an image. I do. For example, when the photographing device 1 is installed in a building such as a restaurant, the photographing device 1 photographs the seat in the building through the polarizing plate 12a of the second photographing unit 12, so that water droplets or food attached to the table of the seat or the like is eaten. It is possible to photograph the presence of dirt such as leftovers. Further, the second imaging processing unit 21c rotates the polarizing plate 12a to change the angle of polarization, so that stains that are difficult to find at one angle can be found by photographing from another angle. The second photographing processing unit 21c performs a plurality of times of photographing at a predetermined timing, for example, before and after the seat is used by the customer, by rotating the polarizing plate 12a once. The second imaging processing unit 21c determines whether or not there is a change in the polarization state by comparing two images captured at the same polarization angle and at different timings. By comparing with the polarization state after use based on the polarization state of the table without stains before use, the second photographing processing unit 21c can sufficiently clean the table due to use or the inside of the building. It is possible to accurately detect an object to be cleaned (object to be cleaned) such as a portion that is not.

Further, the second imaging processing unit 21c does not rotate the rotation mechanism and the angle of the polarizing plate 12a except when the polarizing plate 12a is rotated at a predetermined timing to perform imaging by the second imaging unit 12 as described above. Is continuously photographed by the second photographing unit 12 in a fixed state. The image taken at this time is used, for example, in a process of detecting a human face from an image, a process of detecting a human body, a process of recognizing a human behavior, or the like. That is, the second imaging processing unit 21c photographs the inside of a normal building by the second imaging unit 12, changes the angle of the polarizing plate 12a at a predetermined timing, and photographs the polarized state by the second imaging unit 12. There is. For example, the second imaging unit 21c determines the timing before and after use by the passenger in the seat from the image obtained by the normal imaging by the second imaging unit 12, and the second imaging processing unit 21c changes the angle of the polarizing plate 12a to photograph the polarized state. It can be performed.

<Air flow visualization process>
The photographing device 1 according to the present embodiment captures an image in which the air flow is visualized by using the Schlieren method by the projector 10 and the first photographing unit 11. The Schlieren method is a method of observing a change in the refractive index as a difference between light and dark by utilizing the phenomenon that light passing through is refracted by a density gradient in a transparent gas. In the conventional Schlieren method, an optical system including a knife edge has been used, but in recent years, a method called a digital focusing Schlieren method or a background-oriented Schlieren method has been proposed. In these Schlieren methods, a background image such as a striped pattern is placed on the background of the observation target, and the image is taken through the cutoff grid corresponding to the background image, so that the change in the refractive index of the observation target is the difference between light and dark. It is possible to obtain an image that visualizes the exhaled breath or the air flow of the observation target. In the example shown in FIG. 1, the image projected by the projector 10 on the wall 101 of a building such as a store is the background image.

FIG. 3 is a schematic view showing an example of a background image and a cutoff grid. The background image shown in FIG. 3 corresponds to an image projected by the projector 10 on the wall 101 and captured by the first photographing unit 11 without being blocked by the cutoff grid 11b. This background image does not necessarily match the image projected by the projector 10. In the illustrated background image, the hatched area is black (dark) and the non-hatched area is white (light). The cut-off grid shown in FIG. 3 is a pattern in which light is blocked by the cut-off grid 11b of the first photographing unit 11. In the illustrated cut-off grid, the hatched region is a region that blocks light, and the non-hatched region is a region that transmits light.

The background image shown in FIG. 3 is a monochrome horizontal striped pattern, and in this example, the black region is thick and the white region is thin. The cut-off grid corresponding to this is a pattern in which the black and white of the background image is inverted, and in this example, the cut-off area is a thin pattern and the transparent area is a thick horizontal stripe pattern. The first photographing processing unit 21b of the photographing apparatus 1 controls the blocking pattern by the cutoff grid 11b in order to block the light with the pattern in which the background image projected by the projector 10 is inverted. Further, the background image and the cutoff grid are not limited to a striped pattern, and may be, for example, a pattern in which circular figures are arranged, a so-called polka dot pattern, or, for example, a pattern in which characters or numbers are appropriately arranged. Often, it may be various patterns other than these. The photographing apparatus 1 according to the present embodiment optimizes the image projected by the projector 10 by appropriately adjusting and changing it according to the shape, color, pattern, and the like of the wall 101 to which the projector 10 is projected.

Therefore, in the information processing system according to the present embodiment, it is necessary to perform a process of determining an image projected by the projector 10 in advance before taking an image in which the air flow is visualized by the photographing device 1. The photographing device 1 projects an image stored as a default projection image in the storage unit 23 onto the wall 101 of the building by the projector 10 when, for example, an operation such as initial setting is performed by the user. The photographing device 1 sets the cut-off grid 11b of the first photographing unit 11 in a state in which light is not blocked in the entire area, and the first photographing unit 11 takes an image projected on the wall 101. The photographing device 1 modifies the default projection image based on the image photographed by the first photographing unit 11.

FIG. 4 is a schematic diagram for explaining the modification of the projection image. An image with a horizontal stripe pattern is shown on the upper left side of FIG. 4 as an example of a default projection image stored in the photographing device 1. It is assumed that the photographing device 1 projects this projection image onto the wall 101 with the projector 10, and the image obtained by taking the image with the first photographing unit 11 is the photographed image shown on the upper right side of FIG. For example, when the wall 101 has a portion where the shape changes such as unevenness or a step, the image projected by the projector 10 is displayed by the first photographing unit 11 in a state where distortion such as bending or bending occurs at this portion. Be photographed.

Based on the image captured by the first imaging unit 11, the photographing device 1 sets the projected image so as to bring the captured image closer to the default projection image (so that the captured image and the default projection image match). Fix it. An example of the corrected projection image is shown on the lower left side of FIG. For example, the photographing device 1 detects a portion where the horizontal stripe pattern is distorted from the image photographed by the first photographing unit 11, and distorts the corresponding portion of the default photographing image in a direction of removing the distortion for photographing. The image can be modified. The method of correcting the image for shooting by the photographing device 1 is an example, and the method is not limited to this, and various correction methods may be adopted. For the correction of the photographed image, for example, a technique such as projection mapping can be used. Further, for example, a projection image may be generated using a learning model that has been machine-learned in advance. In this case, the learning model is machine-learned in advance using, for example, teacher data in which an image obtained by capturing a projected distorted pattern and a projection image obtained by appropriately modifying this image are subjected to machine learning, and the first imaging is performed. It may be configured to output a projection image by inputting an image taken by the unit 11.

Further, in the example of modifying the projection image shown in FIG. 4, the shape of the projection image is modified. Although not shown, the photographing device 1 may correct the shading of the projection image. For example, when a pattern or the like is drawn on the wall 101, the pattern projected by the projector 10 may have shading. Therefore, the photographing device 1 corrects the shading of the projection image so that the shading of the striped pattern of the captured image taken by the first photographing unit 11 becomes uniform. The photographing device 1 corrects the intensity of light at the time of projection so as to output stronger light to a portion that absorbs light more, for example, in a pattern drawn on a wall 101.

After generating the projection image projected by the projector 10 and determining the blocking pattern by the cut-off grid 11b of the first photographing unit 11 by the above procedure, the photographing apparatus 1 obtains the corrected projection image. The projector 10 projects the image toward the wall 101, the cutoff grid 11b blocks the light according to the determined blocking pattern, and the first photographing unit 11 takes a picture. By this photographing, the photographing apparatus 1 can acquire an image which visualized the air flow in a building and the like. FIG. 5 is a schematic diagram showing an example of an image in which exhaled breath and the like are visualized. The illustrated image corresponds to an image extracted from the image captured by the photographing apparatus 1 by enlarging a portion where the exhaled air of a person is visualized as an example of the air flow.

<Detection of dirt, etc. based on polarized light>
In the photographing apparatus 1 according to the present embodiment, the polarization state at a plurality of angles can be photographed by the second photographing unit 12 by rotating the polarizing plate 12a to change the angle of polarization. FIG. 6 is an image example for explaining the difference in the captured image depending on the presence or absence of the polarizing plate 12a. The two images shown in FIG. 6 are taken of a desk with spilled water droplets. The image on the left side of FIG. 6 is an image taken by the polarizing plate 12a without polarization, and the image on the right side is an image taken by the polarizing plate 12a with polarization at a predetermined angle. It is difficult to confirm the presence of transparent water droplets in an unpolarized image. On the other hand, in the captured image with polarized light, the presence of water droplets spilled on the desk can be visually confirmed.

Further, the photographing device 1 stores an image at a plurality of polarization angles photographed by the second photographing unit 12 in a state where the inside of the building to be photographed is clean in the storage unit 23 as a reference image. The photographing device 1 performs photographing at a plurality of polarization angles by the second photographing unit 12 at an appropriate timing (for example, after the customer uses the seat). The photographing device 1 can compare a reference image with an image captured at an appropriate timing, calculate, for example, the difference between the pixel values of the two images, and detect a portion where the polarization state has changed. ..

<Overview of evaluation service>
The information processing system according to the present embodiment can provide a service of notifying the user of information on ventilation of the building by evaluating the state of air in the building by using the above-mentioned photographing device 1. can. The services provided by the information processing system according to the present embodiment include a spot-based ventilation assessment service and a resident ventilation monitor service.

(1) Ventilation assessment service The ventilation assessment service evaluates the ventilation performance of this building by evaluating the air condition inside the building, for example, when constructing or remodeling the building, and constructs the building. This is a service that notifies users such as vendors or owners of evaluation results. For example, in response to a request from a user, the service provider brings the above-mentioned photographing device 1 and the terminal device 5 that performs evaluation based on the images taken by the photographing device 1 into the building to be evaluated for which construction or remodeling has been completed. Then, these devices are used to evaluate the air condition in the building. As an evaluation of the air condition, for example, numerical values such as the direction, velocity and amount of airflow in the building can be calculated, and for example, numerical values such as the number and size of places where the air flow is stagnant in the building. Can be calculated. Based on these evaluation results, for example, the ventilation performance of this building can be ranked according to whether or not the calculated numerical value exceeds a predetermined threshold value. Further, for example, the air condition may be evaluated by determining the rank regarding the ventilation performance of the building from the image captured by the photographing device 1 by using a learning model in which machine learning has been performed in advance. In order to evaluate the air condition, the service provider installs the photographing device 1 in the room to be evaluated in the building, for example, and the projection pattern of the image projected by the projector 10 of the photographing device 1 on the wall or the like of the room and the first 1 Perform initial setting work for determining a light blocking pattern by the cutoff grid 11b of the photographing unit 11.

After the initial setting of the photographing device 1 is completed, in the information processing system according to the present embodiment, the service provider performs the work of pouring a predetermined gas into the room or the like to be evaluated. The predetermined gas is, for example, smoke, which is a gas having different properties (for example, color, temperature, humidity, etc.) from the existing air existing in this room, or a gas having a refractive index different from that of the existing air. By pouring such a gas, a difference in refractive index and the like is generated from the existing air, and it can be expected that the photographing apparatus 1 that performs the processing by the Schlieren method can visualize the state of the air more accurately. A device for injecting a predetermined air is prepared by the service provider separately from the photographing device 1 and the terminal device 5, and discharges gas according to the operation of the service provider, but is linked with the photographing device 1 and the terminal device 5. And expectation may be discharged. It is not essential to pour a predetermined gas, and in the information processing system according to the present embodiment, shooting may be performed by the photographing device 1 without pouring a predetermined gas.

The device for pouring a predetermined gas is not brought into the room to be evaluated by the service provider, but for example, equipment installed in the room to be evaluated, for example, an air conditioner, a fan, a circulator, or the like is used. You may. For example, an air conditioner can be used to inject a gas having a temperature or humidity different from that in the room. Further, for example, using a fan or a circulator, outdoor air having a temperature or humidity different from that of indoor air can be flowed into the room.

Imaging with the imaging device 1 under various conditions such as a state in which a predetermined gas is poured, a state in which a predetermined gas has been poured, and a state in which the room is ventilated (opening a window, operating a ventilation fan, etc.). To obtain an image that visualizes the air flow. In the present embodiment, the image acquired from the photographing device 1 is a moving image, and the photographing device 1 transmits the captured moving image to the terminal device 5 by wire or wireless communication. However, the terminal device 5 may acquire the still image taken by the photographing device 1. Whether the photographing device 1 captures a moving image or a still image can be appropriately determined depending on what kind of evaluation is performed based on the image captured by the photographing device 1.

The terminal device 5 is a device in which an application program of the information processing system according to the present embodiment is installed in a general-purpose information processing device such as a tablet-type terminal device or a notebook computer. The terminal device 5 acquires an image taken by the photographing device 1, and based on this image, the airflow velocity (airflow velocity), the airflow direction (airflow direction), and the amount of airflow (air) in the room to be evaluated. Evaluate the ventilation performance of the room by calculating numerical values such as (flow rate). Further, the terminal device 5 identifies a place where the air flow is poor, a place where the air stays, etc. in the room to be evaluated based on the image taken by the photographing device 1, and determines the number and scale of these places. Evaluate the ventilation performance of the room accordingly. The terminal device 5 displays these evaluation results on the display unit and presents them to the user, and at the same time, for example, by transmitting the evaluation results to the server device 3, the evaluation results are stored and stored in the database of the server device 3 or the like. be able to. In the present embodiment, the terminal device 5 shall perform the evaluation based on the image taken by the photographing device 1, but the terminal device 5 is not limited to this, and for example, the evaluation is performed based on the image taken by the photographing device 1 by itself. For example, the server device 3 may perform the evaluation and the evaluation result may be displayed on the terminal device 5.

Based on the evaluation results of the ventilation performance of the room by the photographing device 1 and the terminal device 5, the user can change the position of the furniture in the room, operate the ventilation equipment, change the position to open the window, and the like. It is possible to take measures and repeat the evaluation by the photographing device 1 and the terminal device 5. If a high evaluation is not obtained even after repeated evaluations, the user can consider introducing further ventilation measures such as adding ventilation equipment to this room. After introducing further ventilation measures, the user can request an evaluation from the service provider, re-evaluate the ventilation performance of this room, and confirm the effect of the introduced ventilation measures. At this time, the terminal device 5 may acquire information such as the previous evaluation result and the captured image from the database of the server device 3 and compare it with the current evaluation result.

If a high evaluation is obtained, the builder or the like can complete the construction or remodeling of this room or building and hand it over to the owner. If there is a public certification body for ventilation performance of the building, the user or service provider may submit the evaluation result for ventilation to the public certification body and obtain the certification for ventilation performance from the public certification body.

(2) Ventilation monitor service The ventilation monitor service is a service in which a photographing device 1 is installed in a store such as a restaurant and the air flow in the store is constantly monitored. The service provider is installed at an appropriate position in the store (for example, a position where the audience seats can be photographed) in response to a request from a user such as a store owner, and the projection pattern and cutoff of the projector 10 are provided. Perform the initial setting work for determining the cutoff pattern of the grid 11b. Further, the service provider places the terminal device 5 that acquires the image taken by the photographing device 1 and evaluates the air condition at an appropriate position in the store, for example, a position that cannot be seen by customers such as the backyard of the store. Install. The imaging device 1 and the terminal device 5 used in the ventilation monitor service may have the same performance or function as the imaging device 1 and the terminal device 5 used in the ventilation assessment service, and may be different. You may.

The photographing device 1 performs projection by the projector 10 and photography by the first photographing unit 11 during a predetermined period such as during business hours of a store, and transmits an image that visualizes the air obtained by the photographing to the terminal device 5. .. At this time, a predetermined gas may be poured into the store as in the case of the ventilation assessment service. The terminal device 5 calculates numerical values such as the speed of the air flow and the amount of the air flow in the store based on the image acquired from the photographing device 1, and also has a place where the air flow is poor or the air stays in the store. Identify the location, etc. The terminal device 5 displays the calculated numerical value and information such as the specified location as an evaluation result regarding the ventilation of the store and presents it to the user. Further, the terminal device 5 repeatedly evaluates the ventilation of the store, compares the previous evaluation result with the current evaluation result, and for example, the calculated numerical values such as the speed and amount of the air flow change beyond the threshold value. If this is the case, or if, for example, the number and scale of air retention points change beyond the threshold value, a warning is output to the user.

In the present embodiment, the terminal device 5 shall perform the evaluation based on the image captured by the photographing device 1, but the present invention is not limited to this. For example, the photographing device 1 may perform the evaluation and transmit the evaluation result to the terminal device 5. Further, for example, the photographed image is directly transmitted from the photographing device 1 to the server device 3, or the photographed image is transmitted from the photographing device 1 to the server device 3 via the terminal device 5, and the server device 3 evaluates based on the photographed image. May be performed and the evaluation result may be transmitted to the terminal device 5.

Further, for example, when a photographed image is transmitted from the photographing device 1 to the server device 3 and the server device 3 performs an evaluation based on the photographed image, the store may not be equipped with the terminal device 5. In this case, the ventilation monitor service may be provided as a so-called cloud service, and the user accesses the server device 3 by using a terminal device such as a smartphone or a personal computer owned by the user, and the evaluation is performed based on the captured image. You can get the result. The ventilation assessment service and the ventilation monitor service according to the present embodiment can be provided as a cloud service.

Further, the imaging device 1 according to the present embodiment can perform imaging by the second imaging unit 12 via the polarizing plate 12a as described above. The photographing device 1 installed in the store periodically repeats the photographing by the second photographing unit 12, and transmits the photographed image to the terminal device 5. The terminal device 5 acquires a captured image by the second photographing unit 12 and compares it with an image previously captured to detect a portion where dirt or the like is attached to a table or the like in the store, and the user A warning is output to. In the ventilation monitor service, it is not always necessary to detect stains and the like based on the captured image of the second photographing unit 12 having the polarizing plate 12a. The unit 12 may not be provided.

FIG. 7 is a flowchart for explaining an outline of the service provided by the information processing system according to the present embodiment. First, in the information processing system according to the present embodiment, for example, a service provider or the like installs a photographing device 1 in a building to be evaluated, and a terminal device 5 performs a ventilation assessment based on an image taken by the photographing device 1. The service evaluates the air condition (step S1). At this time, if the evaluation in this building is the second time or later, the terminal device 5 acquires the previous evaluation result, compares it with the current evaluation result, and outputs information on the change in the evaluation result. .. For example, the terminal device 5 can identify the retention points of air in the building and present changes in the number and scale of the retention points between the previous time and the present time.

For example, if the evaluation result in step S1 determines whether or not to clear the predetermined standard (step S2), and if the evaluation result does not clear the standard (S2: NO), the builder or the owner, etc. The ventilation equipment of the building is improved (step S3), and the air condition is re-evaluated by the ventilation assessment service. When the evaluation result clears the standard (S2: YES), for example, the builder completes the construction or remodeling of this building, and the owner, for example, starts a business such as restaurant management using this building. , Proceed to step S4.

In the ventilation monitor service, for example, the imaging device 1 is installed so that the evaluation portion in the building can be photographed by focusing on the portion where the evaluation result is low in the ventilation assessment service. The photographing device 1 continuously photographs the inside of the building, and the terminal device 5 evaluates the air condition by the ventilation monitor service based on the image captured by the photographing device 1 (step S4). The terminal device 5 determines whether or not an abnormality has been detected from the evaluation result (step S5), and if an abnormality is detected (S5: YES), outputs a warning to the user (step S6), and steps S4. Return the process to. If no abnormality is detected (S5: NO), the terminal device 5 returns the process to step S4. The photographing device 1 and the terminal device 5 continuously carry out the ventilation monitor service of steps S4 to S6.

<Device configuration>
FIG. 8 is a block diagram showing the configuration of the terminal device 5 according to the present embodiment. The terminal device 5 according to the present embodiment includes a processing unit 51, a storage unit (storage) 52, a communication unit (transceiver) 53, a display unit (display) 54, an operation unit 55, and the like. The terminal device 5 is a device used by a service provider who evaluates the air condition in the ventilation assessment service, or a user such as an owner of a store or the like to evaluate the air condition in the ventilation monitor service, for example. It can be configured using an information processing device such as a personal computer, a smartphone, or a tablet terminal device.

The processing unit 51 is configured by using an arithmetic processing unit such as a CPU or MPU, a ROM, and the like. The processing unit 51 acquires an image captured by the photographing device 1 by reading and executing the program 52a stored in the storage unit 52, a process of evaluating an air condition or the like based on the acquired image, and a process of evaluating the air condition or the like based on the acquired image. , Performs various processes such as notifying the user of the evaluation result.

The storage unit 52 is configured by using a non-volatile memory element such as a flash memory. The storage unit 52 stores various programs executed by the processing unit 51 and various data required for processing by the processing unit 51. In the present embodiment, the storage unit 52 stores the program 52a executed by the processing unit 51. In the present embodiment, the program 52a is distributed by a remote server device or the like, which is acquired by the terminal device 5 by communication and stored in the storage unit 52. However, the program 52a may be written in the storage unit 52, for example, at the manufacturing stage of the terminal device 5. For example, in the program 52a, the terminal device 5 may read the program 52a recorded on the recording medium 98 such as a memory card or an optical disk and store it in the storage unit 52. For example, in the program 52a, the writing device may read what was recorded on the recording medium 98 and write it in the storage unit 52 of the terminal device 5. The program 52a may be provided in a mode of distribution via a network, or may be provided in a mode recorded on a recording medium 98.

The communication unit 53 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, and the like. In the present embodiment, the communication unit 53 communicates with the photographing device 1 and the server device 3 and the like via the network N. The communication unit 53 transmits the data given by the processing unit 51 to another device, and gives the data received from the other device to the processing unit 51.

The display unit 54 is configured by using a liquid crystal display or the like, and displays various images, characters, and the like based on the processing of the processing unit 51. The operation unit 55 accepts the user's operation and notifies the processing unit 51 of the accepted operation. For example, the operation unit 55 accepts a user's operation by an input device such as a mechanical button or a touch panel provided on the surface of the display unit 54. Further, for example, the operation unit 55 may be an input device such as a mouse and a keyboard, and these input devices may be configured to be removable with respect to the terminal device 5.

Further, in the terminal device 5 according to the present embodiment, the processing unit 51 reads out and executes the program 52a stored in the storage unit 52, so that the evaluation processing unit (evaluation unit) 51a and the display processing unit (information output unit) 51b and the like are realized in the processing unit 51 as a software-like functional unit. The program 52a may be a program dedicated to the information processing system according to the present embodiment, or may be a general-purpose program such as an Internet browser or a web browser. Further, the program 52a may be a different program for the one used in the ventilation assessment service and the one used in the ventilation monitor service. However, in this embodiment, the same program shall be used.

The evaluation processing unit 51a acquires an image that visualizes the air flow photographed by the projector 10 and the first photographing unit 11 by the photographing device 1, and evaluates the state of the air based on the acquired image. The evaluation processing unit 51a performs image processing for determining, for example, the size, length, or direction of the airflow captured in the acquired image, thereby determining the direction of the airflow in the building, the velocity of the airflow, the amount of the airflow, and the like. Calculate the numerical value. Further, the evaluation processing unit 51a detects from the image, for example, a place where no air is flowing or a place where the air flow is small, or a place where the air flow is swirling, and such air is retained. Calculate the number, size, amount, etc. of the locations.

In the ventilation assessment service, the evaluation processing unit 51a calculates a comprehensive ventilation score based on these plurality of calculated numerical values. For example, the evaluation processing unit 51a calculates a score for each value by comparing with a plurality of threshold values determined for each calculated value, and totals the calculated scores for the plurality of values to provide comprehensive ventilation. The score can be calculated. The evaluation processing unit 51a determines whether or not the building to be evaluated clears the standard by comparing the ventilation score with a predetermined reference value.

Further, in the ventilation monitor service, the evaluation processing unit 51a periodically repeatedly calculates the above-mentioned numerical value based on the image captured by the photographing device 1, and stores the calculated numerical value in the storage unit 52. The evaluation processing unit 51a detects deterioration of the air condition related to ventilation by comparing each calculated numerical value with a predetermined threshold value for each numerical value. Further, the evaluation processing unit 51a calculates the time-series changes of these numerical values calculated periodically, that is, the difference between the previous numerical value and the current numerical value, and the change of the calculated numerical value is set to a predetermined threshold value. Determine if it exceeds.

The display processing unit 51b performs a process of displaying various characters, images, and the like on the display unit 54. In the present embodiment, the display processing unit 51b performs a process of displaying the evaluation result of the air condition by the evaluation processing unit 51a on the display unit 54 and presenting it to the user. Some display examples of the evaluation results will be described below.

FIG. 9 is a block diagram showing a configuration of the server device 3 according to the present embodiment. The server device 3 according to the present embodiment includes a processing unit 31, a storage unit (storage) 32, a communication unit (transceiver) 33, and the like. In the present embodiment, it is assumed that the processing is performed by one server device, but a plurality of server devices may be distributed to perform the processing.

The processing unit 31 is configured by using an arithmetic processing unit such as a CPU, MPU or GPU, a ROM, a RAM, and the like. The processing unit 31 reads and executes the server program 32a stored in the storage unit 32, and thereby various processes such as a process of storing and managing information such as evaluation results in the ventilation assessment service and the ventilation monitor service in a database. I do.

The storage unit 32 is configured by using a large-capacity storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for processing by the processing unit 31. In the present embodiment, the storage unit 32 stores the server program 32a executed by the processing unit 31, and also stores information such as evaluation results in the ventilation assessment service, an assessment DB (database), and evaluation results in the ventilation monitor service. A monitor DB for storing information is provided.

In the present embodiment, the server program 32a is provided in a mode of being recorded on a recording medium 99 such as a memory card or an optical disk, and the server device 3 reads the server program 32a from the recording medium 99 and stores it in the storage unit 32. However, the server program 32a may be written to the storage unit 32, for example, at the manufacturing stage of the server device 3. Further, for example, in the server program 32a, the server device 3 may acquire what is distributed by another remote server device or the like by communication. For example, in the server program 32a, the writing device may read what has been recorded on the recording medium 99 and write it in the storage unit 32 of the server device 3. The server program 32a may be provided in the form of distribution via the network, or may be provided in the form of being recorded on the recording medium 99.

The assessment DB 32b is a database that stores information about the ventilation assessment service provided by the information processing system according to the present embodiment. FIG. 10 is a schematic diagram showing a configuration example of the assessment DB 32b. The assessment DB 32b included in the server device 3 according to the present embodiment stores, for example, information such as "user information", "building information", "evaluation date and time", and "evaluation result" in association with each other.

The "user information" is information about a user who has used the ventilation assessment service, and may include information such as "user ID" and "authentication information". The "user ID" is identification information uniquely assigned to the user. The "authentication information" is information used in an authentication process performed when, for example, a user accesses the server device 3 and browses information such as an evaluation result, and is, for example, a password or biometric information.

The "building information" is information about the evaluated building, and may include information such as "building ID", "room ID", and "building position". The "building ID" is identification information uniquely attached to the building. The "room ID" is identification information for identifying each room when, for example, a plurality of rooms exist in a building. The "building position" is information for identifying the place where the building is built, and may be, for example, an address, or may be, for example, position information by GPS (Global Positioning System).

The "evaluation date and time" is information on the date and time when the ventilation assessment service was evaluated for the building, and may include information such as "date" and "measurement time".

The "evaluation result" is information on the evaluation result of the air condition in the building by the ventilation assessment service, for example, "airflow direction (airflow direction)", "airflow velocity (airflow velocity)", and "airflow amount (airflow amount). Information such as "air flow)", "retention point", "ventilation rank", and "photographed image" may be included. The "airflow direction", "airflow velocity", and "airflow amount" are numerical information regarding the airflow calculated from the image captured by the photographing device 1. The "retention point" is information that identifies a place where air does not flow or has little flow in the building, and information such as the position, scale, and number of the stagnant points may be included in the dish. The "ventilation rank" is an evaluation result of a rank related to the ventilation performance of a building determined by a comparison result between a numerical value such as "airflow direction" and one or more threshold values. The "captured image" is data of an image captured by the photographing apparatus 1 at the time of evaluation.

The monitor DB 32c is a database that stores information about the ventilation monitoring service provided by the information processing system according to the present embodiment. FIG. 11 is a schematic view showing a configuration example of the monitor DB 32c. The monitor DB 32c included in the server device 3 according to the present embodiment stores, for example, information such as "user information", "building information", "device ID", "evaluation date and time", and "evaluation result" in association with each other. The information stored in the monitor DB 32c, which is the same as the information stored in the assessment DB 32b, will not be described.

The "user information" is information about a user who is using the ventilation monitor service, and may include information such as "user ID", "authentication information", "terminal ID", and "email address". The "terminal ID" is identification information for identifying a terminal device such as a smartphone used by a user. The "email address" is information indicating the destination of the e-mail to the user. The "terminal ID" and "email address" are used, for example, as information for determining a destination when transmitting a warning message to a user.

The "device ID" is identification information for identifying the photographing device 1 installed in the building in the ventilation monitor service. A plurality of photographing devices 1 may be installed in one building, and in this case, a plurality of device IDs may be stored in the monitor DB.

The configurations of the assessment DB 32b and the monitor DB 32c included in the server device 3 are not limited to those shown in the figure. In addition to the above information, the assessment DB 32b and the monitor DB 32c include, for example, the name of the user, the name of the building, environmental information (temperature, humidity, etc.) at the time of evaluation, information on the gas when a predetermined gas is poured, and the like. Various information can be stored.

The communication unit 33 of the server device 3 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, and the like. In the present embodiment, the communication unit 33 communicates with the terminal device 5 via the network N. The communication unit 33 transmits the data given by the processing unit 31 to another device, and gives the data received from the other device to the processing unit 31.

The storage unit 32 may be an external storage device connected to the server device 3. Further, the server device 3 may be a multi-computer including a plurality of computers, or may be a virtual machine virtually constructed by software. Further, the server device 3 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that accepts operation input, a display unit that displays an image, and the like. ..

Further, in the server device 3 according to the present embodiment, the processing unit 31 reads and executes the server program 32a stored in the storage unit 32, so that the information storage processing unit 31a, the information providing processing unit 31b, and the like are software. It is realized in the processing unit 31 as a functional unit. In this figure, as the functional unit of the processing unit 31, the functional unit that performs the processing related to the assessment DB 32b and the monitor DB 32c is shown, and the functional unit related to the other processing is not shown.

The information storage processing unit 31a receives the evaluation result transmitted from the terminal device 5 and performs a process of storing the received information in the assessment DB 32b or the monitor DB 32c of the storage unit 32.

The information providing processing unit 31b performs a process of providing information stored in the assessment DB 32b and the monitor DB 32c in response to a request from the terminal device 5 or a smartphone or the like used by the user. At this time, the information providing processing unit 31b requests the input of information such as the user ID and password in response to the access from the terminal device 5 or the like, and is associated with the user who is recognized as legitimate by the authentication process. Information such as the evaluation result is read from the database and transmitted to the requesting terminal device 5 or the like. The access from the terminal device 5 or the smartphone to the server device 3 may be performed, for example, via a specific application program, or for example, an access by inputting a specific URL (Uniform Resource Locator) or a two-dimensional code. It may be an access by reading.

<Display of evaluation results>
FIG. 12 is a schematic view showing an example of displaying an evaluation result by the terminal device 5 according to the present embodiment. The illustrated example is a three-dimensional display of the results of evaluating the air condition in the building in the ventilation assessment service. The terminal device 5 calculates numerical values such as the direction, velocity, and amount of the air flow at each point in the building based on the image that visualizes the air flow taken by the photographing device 1. Based on these calculated values, the terminal device 5 is reproduced in a three-dimensional virtual space with the direction of the airflow as the direction of the arrow, the speed of the airflow as the length of the arrow, and the amount of the airflow as the thickness of the arrow. An image in which a plurality of arrows indicating the air flow are arranged in the room to be evaluated is displayed on the display unit 54.

Further, the terminal device 5 is based on an image that visualizes the air flow taken by the photographing device 1, for example, a place where no air is flowing, a place where the air flow is small, or the air flow is swirling. The location etc. is detected from the image. In the terminal device, such a place is a place where air stays, and in the illustrated example, it stays in the room to be evaluated reproduced in a three-dimensional virtual space as an elliptical area containing a plurality of arrows. The part is displayed.

When displaying this example, the terminal device 5 stores in advance the information of the three-dimensional object that reproduces the room to be evaluated in the three-dimensional virtual space. The three-dimensional object of the room is generated in advance by a technique such as SFM (structure from motion) that generates the three-dimensional object based on the image of the room taken by the camera, for example. Although the photographing device 1 and the terminal device 5 may be used for photographing the room and generating the three-dimensional object, a device different from these devices may be used. Further, a three-dimensional object may be generated from the design data of the building and stored in the terminal device 5.

Further, in the illustrated display example, the evaluation result is shown in a plane by arranging an arrow or the like indicating an air flow on a plane at a point separated by a predetermined distance from the photographing point by the photographing apparatus 1. However, the terminal device 5 may spatially display the evaluation result by arranging an arrow or the like indicating the air flow in the depth direction further away from the photographing point and the front direction approaching the photographing point. Further, in the illustrated display example, the air flow is expressed by using arrows, but the present invention is not limited to this, and the air flow may be expressed and displayed by, for example, a color-coded display. Further, in the illustrated display example, the evaluation result is displayed three-dimensionally, but the present invention is not limited to this, and the evaluation result may be displayed two-dimensionally.

Further, in the information processing system according to the present embodiment, the evaluation by the ventilation assessment service may be performed a plurality of times. For example, if the building to be evaluated is evaluated as not having sufficient ventilation performance in the first evaluation, the builder of this building can improve the ventilation equipment and perform the second evaluation. .. The terminal device 5 stores the first evaluation result in its own storage unit 52 or the database of the server device 3, etc., and when the second evaluation is performed, the first evaluation result is acquired and the first evaluation result is obtained. The evaluation result of is compared with the evaluation result of the second time. The terminal device 5 identifies the difference between the two evaluation results, that is, the change in the evaluation result from the first time to the second time, and presents the information regarding the change in the evaluation result to the user.

The terminal device 5 displays, for example, two three-dimensional objects of a building side by side on the display unit 54, one of which shows an arrow of an air flow based on the first evaluation result and a retention point, and the other of which shows the second evaluation result. The arrow of the airflow based on it and the stagnant location may be indicated. In this case, the terminal device 5 can notify the user by highlighting, for example, a part where the evaluation result is improved and a part where the evaluation result is deteriorated in the second time rather than the first time by color-coding.

FIG. 13 is a schematic view showing an example of displaying an evaluation result by the terminal device 5 according to the present embodiment. In the illustrated example, the result of evaluating the air condition at a specific position in the building by the ventilation monitor service is displayed as a graph. In this example, the time-series changes in the airflow velocity, the airflow amount, and the retention amount evaluated by the terminal device 5 are displayed as a line graph. As the specific point, for example, a point determined to be a place where the air flow is small and the need for ventilation is high can be set based on the evaluation result of the ventilation assessment service.

In this example, the evaluation of the air flow by the photographing device 1 and the terminal device 5 is repeated once an hour. The terminal device 5 calculates numerical values such as the velocity and amount of the air flow at a specific point based on the image that visualizes the air flow photographed by the photographing device 1. Further, the terminal device 5 stores these calculated values in the storage unit 52, and based on the evaluation result of this time and the evaluation result stored in the storage unit 52 up to a predetermined time before, the air flow velocity and the air flow rate. And create a graph showing the time-series changes in the amount of retention. In the illustrated example, changes in air flow velocity, air flow rate, and retention amount from 9:00 to 17:00 in a certain day are shown graphically.

In the illustrated graph, the air flow velocity, the air flow rate, and the retention amount calculated based on the captured image are shown as an increase / decrease amount with respect to a predetermined reference value, a ratio thereof, a deviation value, or the like. The predetermined reference value includes, for example, a value measured in a ventilation assessment service, a value required by the national government or a certification body as ventilation performance, a value first measured in the day, or a time set in the day. Various values such as those measured in can be used.

The air flow velocity is calculated as, for example, the speed at which the gas flows at a certain point, and the average of the air flow velocities at a plurality of points may be used. The air flow rate is calculated as, for example, the total value of the air flow velocity in a certain range. The amount of retention can be obtained by, for example, calculating the vorticity of a place where the airflow is swirling, and calculating the ratio of the place where the vorticity exceeds the threshold value to the entire observation range. .. For example, if the total purchase of the observation range is 10 square meters and the area where the vorticity exceeds the threshold is 5 square meters, the retention amount can be calculated as 0.5.

Further, in this example, the air flow velocity, the air flow rate, and the retention amount are displayed on one graph, but these may be displayed on individual graphs. When the air flow velocity, the air flow rate, and the retention amount are individually displayed as a graph, each calculated value may be displayed as a graph instead of being displayed as an increase / decrease amount with respect to a predetermined reference value. In this case, the plurality of graphs are displayed side by side, for example, with the horizontal axis (time axis) aligned. Further, when the terminal device 5 receives a click operation or the like for one point on the graph, the terminal device 5 superimposes an arrow or the like indicating an air flow on the image taken by the photographing device 1 at the time corresponding to this one point (for example, the same as in FIG. 12). Image) may be displayed.

In addition, the terminal device 5 has "air circulation is poor", "check the ventilation fan, air conditioner, etc." and "open the window" as "hints for improving ventilation" at the bottom of the above graph. Display the character string of the message such as "Please." The terminal device 5 can acquire and display a message according to the evaluation result from a plurality of messages stored in the storage unit 52 in advance. The illustrated message is an example and is not limited to this.

Further, a learning model in which machine learning has been performed in advance, so-called AI (Artificial Intelligence), may be used for the output of these messages. For example, the learning model may be machine-learned in advance so as to accept an evaluation result regarding the air flow as an input and output a message according to the evaluation result. Further, for example, the learning model accepts the evaluation result regarding the air flow as an input, machine learning is performed in advance so as to estimate the factor leading to the evaluation result, and the terminal device 5 sends a message according to the factor estimated by the learning model. It may be output. Further, the terminal device 5 notifies the user when the evaluation result changes, receives input of information about the factor from the user at this time, collects the evaluation result and the correspondence of the factor as teacher data, and collects the evaluation result and the correspondence of the factor as teacher data. The learning model may be retrained. As a result, the learning model can learn the factors for each environment of the user, and can be expected to output a more appropriate message to the user.

Further, the terminal device 5 calculates the difference between the current evaluation result and the previous evaluation result for each value of the air flow velocity, the air flow rate, and the retention amount, and compares each value with the threshold value set for each value. conduct. The terminal device 5 calculates, for example, the difference between the current air flow rate and the previous air flow rate, and outputs a warning message to the user when the difference exceeds the threshold value, that is, when the air flow rate changes beyond the threshold value. I do. Although not shown, the terminal device 5 displays a warning message, for example, in any free area of the screen shown in FIG. 13, or by displaying a pop-up dialog or the like. At this time, the terminal device 5 may also output a warning sound or the like. Further, the terminal device 5 does not display the warning message on its own display unit 54, but may transmit the warning message to the smartphone or the like of the user registered in advance and display the warning message on the screen of the smartphone or the like. good.

For example, in the example shown in FIG. 13, the air flow rate changes (decreases) beyond the threshold value between 15:00 and 16:00. When calculating the evaluation result at 16:00, for example, the terminal device 5 can determine that the air flow rate has changed beyond the threshold value from the one at 15:00 and display a warning message to the user. The warning message may be displayed by the terminal device 5 when the evaluation result changes beyond the threshold value in the direction of deterioration, and the warning message is displayed for the change in the direction in which the evaluation result improves (improves). You do not have to do. For example, in the example shown in FIG. 13, the air flow rate increases between 9:00 and 10:00, but since this is a direction that improves as a result of the evaluation of ventilation, the terminal device 5 displays a warning message. You do not have to do. However, the terminal device 5 may display a message notifying that the evaluation result has improved.

Further, the terminal device 5 does not compare the amount of change in the numerical value such as the air flow rate calculated based on the image captured by the imaging device 1 with the threshold value, but uses, for example, a learning model that has been machine-learned in advance. It may be determined whether the time-series change of the numerical value of is worsening or improving. This learning model can be machine-learned in advance so as to accept, for example, a time-series change in the numerical value of the air flow rate as an input and output a certainty degree indicating that this change is worse.

Further, in the information processing system according to the present embodiment, the imaging device 1 can detect the adhesion of stains such as water droplets or leftover food based on the image captured by the second imaging unit 12. The photographing device 1 repeatedly performs photographing by the second photographing unit 12 at an appropriate timing, and transmits the image obtained by the photographing to the terminal device 5. The terminal device 5 performs a process of detecting the adhesion of dirt based on the captured image by the second photographing unit 12 received from the photographing device 1. The imaging device 1 may repeat imaging by the second imaging unit 12, for example, at a predetermined cycle, or may be performed at a timing such as when there are no more people (customers) in the store. When the photographing device 1 performs the photographing by the second photographing unit 12 at an aperiodic timing or a timing at which some event occurs, the terminal device 5 may instruct the photographing timing. The terminal device 5 compares the captured image of this time with the captured image of the previous time, detects a portion where the polarization state has changed as an adhered portion of dirt, and displays the detection result on the display unit 54. Although not shown, the terminal device 5 is a square surrounding the dirt detected on the image captured by the second imaging unit 12 of the imaging device 1, for example, as in the image shown as “polarized” on the right side of FIG. The image on which the frame is superimposed can be displayed on the display unit 54 to notify the user of the detection of dirt.

<Flowchart>
FIG. 14 is a flowchart showing a procedure of processing performed by the terminal device 5 according to the present embodiment in the ventilation assessment service. The evaluation processing unit 51a of the processing unit 51 of the terminal device 5 according to the present embodiment communicates with the photographing device 1 by the communication unit 53, so that the air photographed by the photographing device 1 to the first photographing unit 11 An image that visualizes the flow of the above is acquired (step S21). The evaluation processing unit 51a calculates, for example, the air flow velocity and the air flow rate, etc., regarding the state of the air in the building by performing image processing for determining the size, length, direction, etc. of the air flow reflected in the acquired image. (Step S22). Further, the evaluation processing unit 51a identifies from the image, for example, a place where no air is flowing or a place where the air flow is small, or a place where the air flow is swirling (step S23).

The display processing unit 51b of the processing unit 51 acquires the data of the three-dimensional object created in advance for the building to be evaluated by reading it from, for example, the storage unit 42 (step S24). The display processing unit 51b arranges a three-dimensional object of the building in the three-dimensional virtual space based on the data read in step S24, and an arrow corresponding to the air flow velocity, the air flow rate, etc. calculated in step S22 in the building. , And, by displaying an elliptical region or the like corresponding to the retention location, the evaluation result of the air condition in the building is displayed on the display unit 54 (step S25).

Further, the evaluation processing unit 51a stores the information on the air flow velocity, the air flow rate, etc. calculated in step S22, the information on the retention location specified in step S23, etc. as the evaluation result of the air condition related to the ventilation of this building. Store in unit 52 (step S26). The terminal device 5 may store the evaluation result in a database or the like of the server device 3 by transmitting the evaluation result to the server device 3.

The evaluation processing unit 51a determines whether or not there is a previous evaluation result for the evaluated building (step S27). If there is no previous evaluation result (S27: NO), the evaluation processing unit 51a ends the processing. If there is a previous evaluation result (S27: YES), the evaluation processing unit 51a acquires the previous evaluation result from the storage unit 52, the server device 3, or the like (step S28). The evaluation processing unit 51a compares the current evaluation result with the previous evaluation result acquired in step S28 (step S29). The evaluation processing unit 51a displays on the display unit 54 information on the comparison result of step S29, for example, points that have been improved in ventilation from before, points that have not been improved from before, points that have deteriorated from before, and the like (step). S30), the process is terminated.

FIG. 15 is a flowchart showing a procedure of processing performed by the terminal device 5 according to the present embodiment in the ventilation monitor service. The terminal device 5 according to the present embodiment repeatedly evaluates the air condition based on the image taken by the photographing device 1 installed in the building such as a store at a predetermined cycle, and the evaluation processing unit of the processing unit 51. 51a determines whether or not the timing for periodic evaluation has been reached (step S41). If the evaluation timing has not been reached (S41: NO), the evaluation processing unit 51a waits until the evaluation timing is reached. The evaluation cycle is appropriately determined according to the necessity of evaluation, such as once a day, once an hour, once every 10 minutes, once a minute, or once a second. Can be set to.

When the evaluation timing is reached (S41: YES), the evaluation processing unit 51a communicates with the photographing device 1 by the communication unit 53, so that the air flow photographed by the photographing device 1 to the first photographing unit 11 Is acquired (step S42). The evaluation processing unit 51a calculates, for example, the air flow velocity and the air flow rate, etc., regarding the state of the air in the building by performing image processing for determining the size, length, direction, etc. of the air flow reflected in the acquired image. (Step S43). Further, the evaluation processing unit 51a identifies from the image, for example, a place where no air is flowing or a place where the air flow is small, or a place where the air flow is swirling (step S44).

The evaluation processing unit 51a stores the information on the air flow velocity, the air flow rate, etc. calculated in step S43, the information on the retention location specified in step S44, etc. as the evaluation result of the air condition related to the ventilation of this building. Store in 52 (step S45). Next, the evaluation processing unit 51a acquires the previous evaluation results for a predetermined period (for example, one day's worth, one week's worth, one month's worth, etc.) for the evaluation that is periodically repeated (step S46). The display processing unit 51b of the processing unit 51 is a graph showing, for example, a change in the evaluation result shown in FIG. 13 based on the current evaluation result in steps S43 and S44 and the evaluation result for a predetermined period acquired in step S46. Is displayed on the display unit 54 (step S47).

Further, the evaluation processing unit 51a calculates the difference between the numerical values such as the air flow velocity and the air flow rate related to the current evaluation result and the numerical values such as the air flow velocity and the air flow rate related to the previous evaluation result (step S48). The evaluation processing unit 51a determines whether or not the difference calculated for each numerical value in step S48 exceeds a predetermined threshold value for each numerical value (step S49). When the difference does not exceed the threshold value (S49: NO), the evaluation processing unit 51a returns the processing to step S41. When the difference exceeds the threshold value (S49: YES), the display processing unit 51b warns the user by displaying, for example, a warning message notifying that the air condition has changed beyond the threshold value on the display unit 54. Output (step S50), and return the process to step S41.

<Summary>
In the information processing system according to the present embodiment having the above configuration, the imaging device 1 having the projector 10, the first imaging unit 11, and the second imaging unit 12 is used, and the projection pattern according to the projection destination in the building is projected on the projector 10. The first photographing unit 11 takes a picture with the projection destination on which the projection pattern is projected as a background through the cutoff grid 11b that blocks the light with a light-shielding pattern corresponding to the projection pattern. The terminal device 5 evaluates the state of air in the building based on the image taken by the first photographing unit 11 of the photographing device 1, displays the evaluation result on the display unit 54, and presents it to the user. As a result, it can be expected to realize a service for evaluating the state of air in the building using the photographing device 1 and the terminal device 5.

Further, in the information processing system according to the present embodiment, for example, the first photographing unit 11 of the photographing apparatus 1 takes a picture in a state where a predetermined gas having a color, temperature, humidity, etc. different from that of the air in the building is allowed to flow into the building. The terminal device 5 evaluates the air condition in the building based on the image. As a result, it can be expected that the photographing device 1 that captures an image that visualizes the air flow using the Schlieren optical system will perform photographing with higher accuracy.

Further, in the information processing system according to the present embodiment, the terminal device 5 evaluates the air flow rate, the air flow velocity, the air flow method, and the like in the building based on the image captured by the photographing device 1. As a result, the user can be expected to easily grasp the air condition in the building based on the presented evaluation result.

Further, in the information processing system according to the present embodiment, the terminal device 5 visualizes the evaluated air flow rate, air flow velocity, air flow method, etc. in the evaluated building on the image of the three-dimensional object of the building created in advance by using arrows and the like. To display. As a result, the user can expect to more easily grasp the evaluation result of the air condition.

Further, in the information processing system according to the present embodiment, the terminal device 5 identifies a place where air is accumulated in the building based on the image taken by the photographing device 1. As a result, the user can be expected to easily grasp the part of the building where the ventilation needs to be improved.

Further, in the information processing system according to the present embodiment, the terminal device 5 visualizes and displays the specified air retention location on the image of the three-dimensional object of the building created in advance. As a result, the user can expect to more easily grasp the location where the air stays in the building.

Further, in the information processing system according to the present embodiment, for example, in the ventilation assessment service, the evaluation result of the air condition based on the image taken by the photographing device 1 at the first time point is evaluated by the storage unit 52 of the terminal device 5 or the server device 3. The information on the change between the evaluation result at the first time point and the evaluation result at the second time point is output by the terminal device 5 after being stored in the database or the like and similarly evaluating the air condition at the second time point. As a result, the user can be expected to evaluate the air condition of the building to be constructed or remodeled a plurality of times and improve the ventilation equipment.

Further, in the information processing system according to the present embodiment, a location where air is retained in the building at the first time point is specified, and information regarding a change in air retention at this location at the second time point is output. As a result, the user can be expected to easily grasp the improvement effect on the place where the air stays in the building.

Further, in the information processing system according to the present embodiment, for example, in the ventilation monitor service, the photographing device 1 is installed in the building to periodically take pictures, and the terminal device 5 builds the building based on the periodically taken images. The state of the air inside is periodically evaluated, and for example, by displaying a graph of changes in the air flow velocity or the like during a predetermined period, information on changes in the evaluation results during a predetermined period is output. This allows the user to continuously monitor the air condition in the building.

Further, in the information processing system according to the present embodiment, for example, a difference between the previous evaluation result and the current evaluation result is calculated, and when this difference exceeds the threshold value, the terminal device 5 outputs a warning message. As a result, the user can quickly know that the air condition has deteriorated, and can expect to take measures such as ventilation.

Further, in the information processing system according to the present embodiment, the second photographing unit 12 of the photographing apparatus 1 photographs the inside of the building via the polarizing plate 12a capable of changing the direction of polarization, and the terminal device 5 is different. By comparing the shooting results at two time points, the change in the polarization state in the building is detected, the deposits adhering to the inside of the building are detected based on the change in the polarization state, and the information on the detected deposits is output. .. As a result, the user can continuously monitor the presence or absence of dirt and the like in the building.

In the present embodiment, the terminal device 5 evaluates based on the image captured by the photographing device 1, but the present invention is not limited to this. For example, the photographing device 1 may perform photographing and evaluation, transmit the evaluation result from the photographing device 1 to the terminal device 5, and the terminal device 5 may display the received evaluation result. Further, for example, even if the photographing device 1 transmits an image to the server device 3, the server device 3 performs an evaluation based on the image, transmits the evaluation result to the terminal device 5, and the terminal device 5 displays the received evaluation result. good.

Further, for example, based on an image that visualizes the air flow taken by the photographing device 1, it is determined whether or not the air conditioning equipment (for example, an air conditioner, a circulator, an electric fan, etc.) installed in the building by the terminal device 5 is operating. You may perform the processing to do. Further, for example, the terminal device 5 may perform a process of determining the open / closed state of a window, a door, or the like provided in the building based on an image that visualizes the air flow photographed by the photographing device 1.

The embodiments disclosed this time should be considered as exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, not the above-mentioned meaning, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 Shooting device 3 Server device 5 Terminal device 10 Projector 11 1st shooting section (shooting section)
11a Lens 11b Cut-off grid (light blocking part)
11c Infrared lens 12 2nd shooting section (shooting section)
12a Polarizing plate (polarizing part)
12b Visible light lens 20 Information processing device 21 Processing unit 21a Projection image generation unit 21b First imaging processing unit 21c Second imaging processing unit 22 Input / output unit 23 Storage unit 23a Program 24 Communication unit 51 Processing unit 51a Evaluation processing unit ( Evaluation department)
51b Display processing unit (information output unit)
52 Storage unit 52a Program 53 Communication unit 54 Display unit 55 Operation unit

Claims (14)

The projection pattern generated by adjusting the predetermined image so that the captured image obtained by capturing the projection pattern projected by the projector on the projection destination in the building matches the predetermined image is projected onto the projection destination by the projector.
The imaging unit takes a picture with the projection destination on which the projection pattern is projected by the projector via a light blocking unit that blocks light with a light blocking pattern corresponding to the projection pattern projected by the projector.
Based on the image taken by the photographing unit, the state of air in the building is evaluated.
Output information about the evaluation result of air condition,
An information processing method in which an information processing device performs processing. The state of air in the building is evaluated based on an image taken by the photographing unit with a predetermined gas flowing into the building.
The information processing method according to claim 1. Evaluate the air flow rate, air flow velocity, or air flow direction in the building.
The information processing method according to claim 1 or 2. The evaluated air flow rate, air flow velocity, or air flow direction is visualized and displayed on the image of the three-dimensional object of the building.
The information processing method according to claim 3. Based on the image taken by the photographing unit, a place where air is retained in the building is identified.
The information processing method according to any one of claims 1 to 4. The identified retention location is visualized and displayed on the image of the three-dimensional object of the building.
The information processing method according to claim 5. The evaluation result of the air condition based on the image taken by the photographing unit at the first time point is memorized.
At the second time point after the first time point, the air condition is evaluated based on the image taken by the photographing unit.
Outputs information on changes in the evaluation results at the first time point and the evaluation results at the second time point.
The information processing method according to any one of claims 1 to 6. Identify the location where air is stagnant in the building at the first time point,
Outputs information on changes in air retention at the location at the second time point.
The information processing method according to claim 7. Based on the images periodically taken by the photographing unit installed in the building, the state of air in the building is periodically evaluated.
Outputs information on changes in evaluation results over a specified period
The information processing method according to any one of claims 1 to 8. Compare the change with a predetermined threshold and
An alarm is output when the change exceeds the threshold value.
The information processing method according to claim 9. The photographing unit takes a picture of the inside of the building through the polarizing part capable of changing the direction of polarization.
By comparing the shooting results at two different time points, the change in the polarization state in the building was detected.
Based on the detected change in the polarization state, the deposits adhering to the inside of the building are detected.
Output information about detected deposits,
The information processing method according to any one of claims 1 to 10. A projector that projects a projection pattern generated by adjusting the predetermined image so that the captured image obtained by capturing the projection pattern projected on the projection destination in the building matches the predetermined image, and a projector that projects the projection pattern onto the projection destination.
An imaging unit that shoots with the projection destination on which the projection pattern is projected by the projector via a light blocking unit that blocks light with a light blocking pattern corresponding to the projection pattern projected by the projector.
An information processing device having an evaluation unit that evaluates the air condition in the building based on an image taken by the photographing unit, and an information output unit that outputs information regarding the evaluation result of the air condition by the evaluation unit. Information processing system equipped with. A light blocking unit that blocks light with a shading pattern according to the projection pattern generated by adjusting the predetermined image so that the captured image obtained by shooting the projection pattern projected by the projector on the projection destination in the building matches the predetermined image. An evaluation unit that evaluates the state of air in the building based on an image taken by the photographing unit against the background of the projection destination on which the projection pattern is projected by the projector.
An information processing device including an information output unit that outputs information regarding the evaluation result of the air condition by the evaluation unit. On the computer
A light blocking unit that blocks light with a shading pattern according to the projection pattern generated by adjusting the predetermined image so that the captured image obtained by shooting the projection pattern projected by the projector on the projection destination in the building matches the predetermined image. Based on the image taken by the photographing unit against the background of the projection destination on which the projection pattern is projected by the projector, the state of air in the building is evaluated.
A computer program that executes a process that outputs information about the evaluation result of the air condition.

Images