JP7335850B2 - High fever detection method, high fever detection device, and high fever detection system - Google Patents

Description

Embodiments of the present invention relate to a hyperthermia detection method, a hyperthermia detection device, and a hyperthermia detection system for detecting a hyperthermic health condition.

A thermography technique using an infrared camera is known as a technique for detecting the skin surface temperature of a person to be measured without contact at high speed. As an example of using the thermography technology, the following technology has been proposed.

JP 2013-46668 A

According to one embodiment, the high-temperature person detection device includes a first image recognition unit, a second image recognition unit, a person's body temperature determination unit, and an operation instruction at the time of detection that determines the rule at the time of high-temperature person detection. a detection operation setting unit for setting whether or not to operate the mobile terminal and other external devices; and a microservice for giving specific operation instructions to the mobile terminal and other external devices. , and
the first image recognition unit performing first image recognition from a first signal corresponding to each head of a plurality of persons obtained from a first optical sensor sensitive to infrared light;
The second image recognition unit recognizes a second image from a second signal corresponding to each head of the plurality of persons obtained from a second optical sensor including an image sensor having sensitivity to light of a wavelength other than infrared light. performing image recognition;
The person's body temperature determination unit detects the presence or absence of a hyperthermic person whose body temperature exceeds a predetermined value from the result of the first image recognition and the result of the second image recognition;
The detection-time operation instruction unit that responds to the detection of the presence or absence of a hyperthermic person whose body temperature exceeds the predetermined value outputs an instruction for notification according to a set rule, and outputs an instruction for notification to the detection-time operation setting unit Notifying that the presence or absence of the high fever person is also detected,
The detection operation setting unit sets the microservice for giving specific operation instructions to the mobile terminal and other external devices, and in response to the notification of detection of the presence or absence of the high fever person, the Controlling the microservice corresponding to the portable terminal and other external devices to output the specific operation instruction is provided.

In the present embodiment, a first signal obtained from a first photosensor sensitive to infrared light and a second signal obtained from a second photosensor including an imaging device sensitive to light of wavelengths other than infrared light are used. 2 signal is used to detect a high fever person whose body temperature exceeds a predetermined value, and to provide a high fever detection method, a high fever detection device, and a high fever detection system for notifying the detection result.

FIG. 1 is an explanatory diagram showing the basic concept of this embodiment. FIG. 2 is an explanatory diagram showing the basic system in this embodiment. FIG. 3 is an explanatory diagram showing a problem when using a conventional technique that is not combined with an image recognition technique. FIG. 4 shows an explanatory diagram of the effect when only the head temperature of a person is used for determination. FIG. 5 is an explanatory diagram showing the operation flow executed by the basic system shown in FIG. FIG. 6 is an explanatory diagram illustrating an example of a usage scene of this embodiment. FIG. 7 is an explanatory diagram showing another example of this embodiment. FIG. 8 is an explanatory diagram showing a program configuration used in an Android-based smart phone. FIG. 9 is an explanatory diagram showing the relationship between the configuration of the edge computer and the cloud server. FIG. 10 is a table showing a function list of the application (IF-THEN) engine. FIG. 11 is a block diagram expressing the concept of an application example in this embodiment. FIG. 12 is an explanatory diagram illustrating the effects of writing microservices in an OS-independent program language. FIG. 13 is an explanatory diagram showing the class structure of microservices. FIG. 14 is an explanatory diagram of the functions of microservices set in the individual device control package. FIG. 15 is an explanatory diagram showing a list of APIs provided by the microservice control template class (BaseIms Class). FIG. 16 is an explanatory diagram showing a list of APIs provided by the device control template class (Base Device Class). FIG. 17 is a device state transition diagram. FIG. 18 is an explanatory diagram showing the relationship between device states and microservice states. FIG. 19 is an explanatory diagram showing the relationship between the blackbody radiation source temperature and the spectral radiance characteristics. FIG. 20 shows an explanatory diagram of the internal structure of the trinocular infrared camera. FIG. 21 explains inter-class linkage operations within the class configuration shown in FIG. FIG. 22 is an explanatory diagram showing a setting method for notifying multiple media when a person with high fever is detected. FIG. 23 is an explanatory diagram showing the class configuration of microservices related to information provision. FIG. 24 is an explanatory diagram showing processing steps up to notification of information to the user.

The basic concept of this embodiment will be described with reference to FIG. As shown in FIG. 1(a), assume a place where a plurality of people have gathered. If there is a person with a high fever whose body temperature is a predetermined value (for example, 37.5 degrees Celsius) or more among them, the person with a high fever is automatically detected individually. In the display example of FIG. 1(b), the hatched person corresponds to a person with a high fever of 37.5 degrees Celsius or higher. Then, the information is automatically notified (FIG. 1(c)). A plurality of options exist as the notification method, and in this embodiment, the option can be selected by a very simple method. Here, regarding the notification method, a plurality of methods may be selected at the same time.

In FIG. 1(a), since the number of people gathered at the meeting place is relatively small, each person is displayed separately. But as the population density of congregating people increases, partial overlap occurs. A conventional thermography technique for measuring the surface temperature distribution of an object to be measured cannot identify individual persons in a place where a plurality of persons overlap each other. Therefore, if only the conventional thermography technology is used, it is impossible to individually detect (individually extract from a crowded place) only high-temperature persons in a crowded place.

In this embodiment, image recognition technology is combined with thermography technology using an infrared camera to enable individual detection (individual extraction) of only high-temperature individuals. As a specific combination method, in this embodiment, a program having an image recognition function using AI technology (image recognition class AI Analyze Class) 2018 is incorporated 76 . Details will be described later with reference to FIGS. 11 and 13. FIG.

In order to reduce the risk of virus infection in places where people gather and work, there is a need for technology that easily detects and notifies people with high fever who are suspected of being infected. In this embodiment, a commercially available infrared camera 806 is attached to an edge computer 6 such as an existing smartphone or tablet, and software (service providing application program) 92 called ifLink (registered trademark) application, which will be described later in detail, is installed. available.

As the commercially available infrared camera 806 described above, an infrared camera 806 that can be connected to the edge computer 6 such as a commercially available smartphone or tablet may be used. By using this infrared camera 806, it can be very easily connected to a commercially available edge computer 6 such as a smart phone or a tablet, so that the high fever detection system in this embodiment can be easily constructed.

Also, the ifLink application 92 described above operates as an application for smartphones and tablets owned by many people. As a result, it is possible to use existing smartphones, tablets, and the infrastructure (usage environment) of the infrared camera 806 and install the ifLink application 92, which is a very simple method to reduce the risk of infection at each activity site. There is

In this embodiment, image recognition is combined with the video or still image captured by the infrared camera 806 to individually detect (individually extract) only those with high fever from the group, and automatically notify the information.

There is a conventional method of detecting the surface temperature of an object to be measured using thermography technology using an infrared camera 806 . However, as described above, when people are densely packed and multiple people overlap, it has been necessary until now for the viewer of the video/still image to visually check and separate them into individuals (individual extraction). Furthermore, since it is not possible to distinguish whether the object to be measured is a person or another object with only the conventional thermography technology, a person who identifies the object to be measured is required. For this reason, conventional thermography technology could only be used at the reception desks of companies and facilities where full-time observers can be assigned.
In this embodiment, the image recognition class (AI Analyze Class) 2018 using AI technology is incorporated 76 so that the computer (edge computer 6) recognizes the high fever person in the image (video or still image) obtained from the infrared camera. Since it is automatically recognized inside the edge computer 6 in this way, a person with high fever can be detected and notified without a supervisor. Therefore, in this embodiment, there is an effect that the cost of monitoring (automatic individual detection of a person with high fever) is greatly reduced.
In this embodiment, the method for recognizing a person with high fever is not limited to the use of the image recognition class (AI Analyze Class) 2018 using AI technology, and any image recognition method may be employed. For example, unique hardware having a dedicated image recognition function may be prepared in advance. In this case, an image obtained from the infrared camera may be transmitted to this hardware for image recognition, and the result may be sent back.

In FIG. 1(c), when a high fever person whose body temperature exceeds a predetermined value (for example, 37.5 degrees Celsius) is detected, the result is automatically notified. As an example of options for this automatic notification method, an image (video or still image) clearly indicating the presence of a person with high fever may be displayed 1210 on the display screen 12 of the edge computer 6 such as a smartphone or tablet. Alternatively, a warning may be displayed 1220 on a signage (electronic bulletin board) 804 installed near the edge computer 6 that has detected a high fever person.

Further, the display format is not limited to a specific image, and a warning screen in which “a warning message is written in text format” may be displayed 1230 on the display screen 12 of the edge computer 6 or on the signage 804 . Furthermore, notification may be made not only by a method that appeals to the sense of sight such as a predetermined image or a text warning, but also by a method that appeals to the sense of hearing, such as the voice warning 1240, or the sense of touch or smell.

Further, remote notification may be performed by using network communication or the like to notify a place away from the place where the high-temperature person is detected. As this remote notification method, mail communication (230) 1250 or information posting 1260 on the web screen (210) may be performed. The information to be posted 1260 on this web screen is not limited to the individual information of the high fever person, and may be the aggregate result of information obtained at many different places.

Further, the form of notification is not limited to the above-described "information transmission", and a physical operation accompanied by the device operation 1290 such as the blocking process 1270 of the passage gate (820) may be performed.

The presence of a high fever person may be notified by any notification method, not limited to the options described above. As a notification method, a plurality of options may be notified at the same time.

Conventional surveillance systems have a fixed notification mechanism after detection, and there was no flexibility in how to use it. In contrast, the notification method after detection of a high fever person in the present embodiment can be flexibly set according to the equipment, software, etc. possessed by the user (user). Options that can be set flexibly include turning on a patrol light (registered trademark), sounding an alarm sound 1240, voice guidance 1280 using voice synthesis (audio warning 1240), mail transmission 1250, and SNS (Social Networking Service). ), etc., can be arbitrarily set for notification, screen display 1210, 1230, and the like. As this notification method, a method set by the user in advance is used.
As a technique for flexibly setting according to the device, software, etc. possessed by the user (user), in this embodiment, as will be described later with reference to FIGS. 7, 11 and 22, an application (IF-THEN) The rule 70 is set and the application (IF-THEN) engine 90 using it is set. In other words, the user can flexibly set one or a plurality of notification methods from among the multiple notification method mechanisms described above on the screen of a smartphone or the like.

A basic system (high-temperature person detection system) in this embodiment will be described with reference to FIG. In this basic system (high fever detection system), an infrared camera 806 corresponding to a first optical sensor sensitive to infrared light and a second optical sensor including an imaging device sensitive to light of wavelengths other than infrared light are used. A common camera 816 is also used.

Generally, the wavelength range from 380 nm to 810 nm is called "visible light", and the wavelength range from 810 nm to 1 mm is generally called "infrared light" or "infrared light". However, as shown in FIG. 19, the wavelength range of "infrared light" suitable for "measurement of human body temperature" using black body radiation is within the range of 1 μm to 100 μm, preferably 2.5 μm. to 20 μm is considered appropriate. Therefore, the wavelength range of "infrared" or "infrared light" used herein by the infrared camera 806 as the first photosensor means within the range of 1 μm to 100 μm (preferably within the range of 2.5 μm to 20 μm). do.

Therefore, light with wavelengths outside the above range is referred to herein as "light with wavelengths other than infrared rays". Further, in this embodiment, the second optical sensor (ordinary camera 816) including an imaging element having sensitivity to the above-mentioned "light of wavelengths other than infrared rays" is used to detect a hyperthermia person whose body temperature exceeds a predetermined value. A second signal is also used. Light suitable for this purpose includes ultraviolet light (wavelength of 10 nm or more), visible light, near-infrared light (wavelength of 2.5 μm or less), and microwave (wavelength of 10 cm or less). Considering the above information, the wavelength range of "light of wavelengths other than infrared rays" is the range from 10 nm to 2.5 μm or the range from 20 μm to 10 cm (preferably the range from 380 nm to 1 μm or the range from 100 μm to 10 cm).

The “ordinary camera 816” generally indicates a second optical sensor including an imaging element sensitive to “visible light” in the wavelength range of 380 nm to 810 nm. However, the "normal camera 816" described in FIG. shows the sensor.

Most edge computers 6 such as smartphones and tablets usually have a built-in camera 816 as standard. Therefore, a camera standardly built in the edge computer 6 such as a smart phone or a tablet may be used as the normal camera 816 shown in FIG.

On the other hand, when a commercially available infrared camera 806 is used, the first signal (temperature characteristic signal) obtained from the infrared camera 806 (first optical sensor sensitive to infrared light) is transmitted to the wireless communication means. Alternatively, it may be communicated to the edge computer 6 using wired communication means. Alternatively, signals may be directly transferred via the connection unit 10 as will be described later with reference to FIG.

As shown in FIG. 2, an application 100 (software program) of this system includes a function of a recognition setting unit 110 capable of setting a recognition logic according to a usage scene, a function of a determination temperature setting unit 120, and an application (IF-THEN ) detection rule setting unit 130 for setting rules 70 (described later with reference to FIGS. 7, 11, and 22), detection operation setting unit 140, and detection information transmission unit 150. In the example of FIG. 2, this system application 100 takes the form of software installed on the edge computer 6 . However, it is not limited to this, and part of the functions of each configuration 110 to 150 may be configured by dedicated hardware (electronic circuits).

Therefore, the edge computer 6 in which the application 100 (software program) of the system described above is installed or the edge computer 6 in which dedicated hardware (electronic circuits) for executing a part of each of the functions 110 to 150 is installed is used in this embodiment. It corresponds to a high fever detection device.

A first signal (temperature characteristic signal) obtained from the infrared camera 806 (first optical sensor sensitive to infrared light) is used by the infrared image recognition section 116 (first image recognition unit). The infrared image recognition unit 116 performs image recognition (first image recognition) of the spatial temperature distribution characteristics.

An image signal (corresponding to a video signal or a still image signal and corresponding to a second signal) obtained from an ordinary camera 816 (a second optical sensor including an imaging device sensitive to light of wavelengths other than infrared rays) is used for recognition. It is transmitted to the normal camera image recognition unit 112 (second image recognition unit) of the setting unit (function of) 110 . As the second image recognition performed by the normal camera image recognition unit 112,
1) perform image recognition (image analysis) of the image signal,
2) Determining whether or not there is a person in the screen (person detection),
3) If there are people in the screen, calculate the number of people (detect the number of people),
4) Recognize the position of the head of each person (head detection) only when there are people in the screen. At the same time, 5) for each person identified in the screen, the distance between the person and the infrared camera 806 may be calculated (distance detection).

The determination temperature setting unit 120 also includes a person's head temperature determination unit 125, in which the temperature of the person's head facing the infrared camera 806 is measured as body temperature. Then, a high-temperature person whose body temperature exceeds a predetermined value (for example, 37.5 degrees Celsius) is detected (determination of the presence or absence of a high-temperature person). If a person with high fever is detected as a result of the temperature determination, notification is automatically made. Specifically, detection information is transmitted to a detection operation instruction unit (application (IF-THEN) engine) 90 of the detection rule setting unit (function of) 130 . Then, the detection unit operation setting unit (function of) 140 operates to execute notification processing.

3 and 4, "2) person detection" performed by the normal camera image recognition unit 112 will be described. Only from the first signal (temperature characteristic signal) obtained from the infrared camera 806, the type of heat source that emits black body radiation (from what is heat radiated?) cannot be identified. Therefore, when the image recognition technology normally performed by the camera image recognition unit 112 is not combined, the temperature of objects other than people is also determined.

As an example illustrated by FIG. 3(a), consider a person drinking hot coffee from a coffee cup held in the hand. At this time, it is assumed that the surface temperature of the coffee cup is 50°C. On the thermography screen (output of the infrared image recognition unit 116) obtained from this scene, a surface temperature of 50° C. of the coffee cup is detected as shown in FIG. 3(b). If the determination temperature for determining whether a person has a high fever is "37.5°C", the surface temperature of the coffee cup of 50°C exceeds this determination temperature. If the type of heat source that radiates black body (from what is the heat radiated?) cannot be specified, it is erroneously determined that "a person with high fever exists" (Fig. 3(c)).

FIG. 4 shows the effect of "using only the temperature of a person's head for determination". Here, FIG. 4(a) shows the same view as FIG. 3(a). When "only the temperature of the person's head is used for determination", the coffee cup surface temperature of 50°C is excluded from the detection target (Fig. 4(b)).

The results of “2) human detection” performed by the normal camera image recognition unit 112 and the spatial temperature distribution characteristics in the screen output from the infrared image identification unit 116 are obtained by the human head temperature determination unit 125 (FIG. 2). Integrated processing. Then, as shown in FIG. 4C, the person's head temperature determining unit 125 uses "only the temperature of the person's head" for "determining the presence or absence of a high fever person". Here, the head temperature of the person is used for body temperature measurement, but not limited to this, any arbitrary location of the person may be used for body temperature measurement. When the body temperature is measured at a part other than the head of a person, the name of the "person's head temperature determination unit 125" may be changed to "person's body temperature determination unit" having the same function. In this case, "from the result of the first image recognition and the result of the second image recognition, the person's body temperature determination unit detects the presence or absence of a hyperthermia person whose body temperature exceeds a predetermined value".

Thus, in this embodiment, the first signal obtained from the first optical sensor (infrared camera 806) sensitive to infrared light and the second optical sensor including the imaging element sensitive to light of wavelengths other than infrared light are used. The second signal obtained from the optical sensor (normal camera 816) is also used to detect a high fever person whose body temperature exceeds a predetermined value. As a result, the accuracy of determination of normality (FIG. 4(d)) is significantly improved.

Next, "4) head detection" performed by the normal camera image recognition unit 112 (FIG. 2) described above will be described. As described above with reference to FIGS. 3 and 4, acquiring only the temperature of the head has the effect of eliminating erroneous detection due to other heat sources. Furthermore,
A) Since the trunk and limbs are covered with clothing and are not exposed, the error in the body temperature measurement results from there is large. B) The head is relatively exposed, so it is relatively easy to measure the body temperature.
Therefore, acquiring only the temperature of the head has the effect of improving the accuracy of body temperature measurement.

Furthermore, the relationship with "3. People detection" performed by the camera image recognition unit 112 will also be described. In images (video or still) obtained from a crowded place with multiple people,
C) A part of the body (torso or limbs) tends to overlap with an adjacent person, but the position of the head is relatively separated. Therefore, in this embodiment, by using the head position in particular, not only can the image recognition processing of "3) detection of the number of people" be simplified, but also the accuracy of "3) detection of the number of people" can be improved.

A specific method of "2) person detection" and "3) number of people detection" using "4) head detection" normally performed by the camera image recognition unit 112 will be described. A plurality of standard samples relating to the shape of a person's head and hairstyle are normally stored in advance in the camera image recognition unit 112 . Then, the degree of pattern matching between the image (second signal) obtained from the normal camera 816 (second optical sensor including an imaging device sensitive to light of wavelengths other than infrared rays) and the plurality of standard samples is calculated. Then, an area where the degree of pattern matching exceeds a predetermined value is estimated as a "human head candidate".

In many cases, the torso exists under the "human head", and the limbs (limbs) are arranged around the torso. "2) Human detection" is performed only when the body or limbs are detected directly under the "human head candidate" estimated in this way. Then, the number of areas in the screen where "2) person detection" is performed becomes the result of "3) number of people detection".

By the way, when "4) head detection" is performed on the "back side of the head", accurate body temperature measurement becomes difficult. Consider a case in which the body temperature measurement subject faces the back. In this case, black body radiation emitted from the epidermis of the head is scattered by hair on the optical path toward infrared camera 806 . Since the amount of black body radiation detected by the infrared camera 806 is greatly reduced, the body temperature measurement accuracy is greatly reduced (the detection principle of thermography will be described later with reference to FIG. 19). Therefore, it is desirable to detect only "the head facing forward toward the infrared camera 806".

Only the "head facing the direction of the infrared camera 806" may be used as the "4) head detection" method, and the positions of the "eyes", "nose", and "mouth" arranged in front of the head may be used. . As a specific method, a plurality of standard samples relating to the shape of eyes, nose, and mouth are normally stored in the camera image recognition unit 112 in advance. Then, the image obtained from the normal camera 816 is divided into fine regions, and the degree of pattern matching between each divided region and standard samples of eyes, nose, and mouth is calculated. Then, using the obtained positional relationship among the "position of eye candidate region", "position of nose candidate region", and "position of mouth candidate region", "infrared camera 806 Detects the head facing the direction.

In the above description, "4) head detection" is performed with respect to "the head facing the direction of the infrared camera 806",
The human head temperature determining unit 125 measured the body temperature of the head. However, the temperature is not limited to this, and the body temperature may be measured using a region other than the head.

Also, as the method of "2) person detection", it is not always necessary to use the result of "4) head detection". In "2) person detection" in this embodiment, a flexible method can be applied according to the usage scene. For example, since the number of visitors who visit the reception counter at the same time is often less than several, individual identification of persons is relatively easy. However, the normal camera 816 installed behind the reception counter can only photograph the upper body of the visitor.

Also, when the normal camera 816 is used to photograph the aisle, it is possible to photograph the whole body, but on the other hand, it is necessary to recognize a moving person. In other words, high-speed detection is required for high-temperature person detection in passages. On the other hand, when detecting a person with a high fever in a classroom where about 40 students have gathered, it is necessary to simultaneously measure the body temperatures of many people.

In this way, it is preferable to preset the optimum person detection logic for each different usage scene and select the optimum logic for each usage scene to enable flexible handling. As a method for enabling flexible handling according to this usage scene, for example, a selection menu may be provided that allows the user to select a usage scene, and optimal person detection logic may be executed. Also, as will be described later with reference to FIG. 13, a plurality of methods (optimal person detection logic for each usage scene) are arranged in an image recognition class (AI Analyze Class) using AI. Then, an API command is used to selectively call 76 the most appropriate method for each usage scene.

Next, description will be made regarding "5) Distance detection for each person discriminated in the screen" performed by the ordinary camera image recognition unit 112 in FIG. The infrared camera 806 tends to read the temperature lower than the actual temperature when the distance from the heat source (human head) emitting black body radiation is long (this will also be described later with reference to FIG. 19). Therefore, using the result of “5) Distance detection,
α] The spatial temperature distribution characteristic obtained by the infrared image recognition unit 116 is subjected to body temperature correction (temperature correction) by the human head temperature determination unit 125, or β) the distance from the infrared camera 806 is determined. For a person who is farther than a certain distance, even if the temperature is lower than the judgment temperature, the detection-time operation instructing unit (application (IF-THEN) engine) 90 can perform processing such as calling attention or issuing an instruction.

Therefore, when "5) distance detection" is used, the body temperature detection accuracy is improved, and the effect of improving the instruction accuracy from the detection operation instruction unit 90 is produced.

As the method of "5) distance detection", individual images obtained from a plurality of normal cameras 816 are used, and "triangulation" (used in surveying, etc., uses the relationship between the sides and angles of a triangle to determine the distance, etc.). ) may be used to calculate the distance. For example, when a smart phone or tablet is used as the edge computer 6, the edge computer 6 usually has a built-in camera 816 as standard. Therefore, if individual images obtained from a plurality of edge computers 6 are shared by communication means, the distance can be calculated using "trigonometry". In addition, if a three-lens infrared camera with two built-in visible light sensors 46 and 48, which will be described later with reference to FIG.

Alternatively, the maximum temperature position and the minimum temperature position may be pinpointed from the image analyzed by the normal camera image recognition unit 112 in FIG. 2 and used. That is, using the information of the maximum/minimum temperature position estimated by the normal camera image recognition unit 112 and the information of the maximum/minimum temperature position extracted by the infrared image recognition unit 116, the distance is calculated using "trigonometry". You may

As another example, location information using GPS (Global Positioning System) or beacons may be used. For example, as a result of image recognition by the normal camera image recognition unit 112, it is possible to identify (identify) each detected person. In that case, it is possible to communicate with a mobile terminal (smartphone) owned by the identified (identified) person and collect the position information of the other person.

The detection operation setting unit 140 has a function of setting a control method for various notification means for notifying the presence of a hyperthermia person whose body temperature exceeds a predetermined value. As a specific notification means, there are options such as generation of alarm sound 1240, voice guidance 1280, screen display 1230, mail transmission 1250, and device operation 1290 (1270). Among these, the options used at the time of automatic notification (simultaneous multiple selection is also possible) are the application (IF-THEN) rules 70 (Fig. 7 and FIG. 11). Necessary information for notification means (options) specified according to the application (IF-THEN) rule 70 is communicated (transmitted) from a detection operation instruction unit (application (IF-THEN) engine) 90 .

The application 100 of this system incorporates the function of the detection information transmission unit 150 . Necessary information directed to the cloud server 2 that manages, monitors, and sets the execution of the application 100 of this system is sent from (the function of) the detection information sending unit 150 as appropriate.

The application (IF-THEN) rule 70 used in (the function of) the detection rule setting unit 140 may be set from the cloud server 2 . In this case, the cloud server 2 is required because it is necessary to monitor the execution status of the application (IF-THEN) rule 70 at multiple locations.

Alternatively, the application (IF-THEN) rule 70 may be set locally from the display screen of the application 100 of this system. In that case, the application 100 of this system can be executed without the cloud server 2 .

FIG. 5 illustrates the operational flow performed within the basic system shown in FIG. Here, the flow of operations shown in FIG. 5 will be described with reference to FIG.

During the operation of the application 100 of this system, the periodic processing S01 that repeats the operation shown in FIG. 5 continues. In step 02 (S02), first, the image signal (video or still image) picked up by the normal camera 816 (second optical sensor) is transferred to the normal camera image recognition unit 112 (second image recognition unit) as a second signal. ). Then, the ordinary camera image recognition unit 112 performs "1) image recognition", and "3) number detection" and "head position calculation (4) head detection" using the result of "4) head detection". Part)” and “5) Distance detection” are executed (up to this point, S02). Here, if the number of persons is not 1 or more as a result of "3) detection of the number of people" (that is, no persons are included in the image obtained by the normal camera 816) (No in S03), step 04 ( The process proceeds to the end of the process in S04).

On the other hand, if the number of persons is 1 or more (a person is included in the image obtained by the normal camera 816) (Yes in S03), the person's head temperature determination unit (person's body temperature determination unit) 125 Temperature information is obtained at the head positions of each person corresponding to the number of persons (S05). Prior to this, the infrared image recognition unit 116 (first image recognition unit) uses the temperature characteristic signal (first signal) obtained from the infrared camera 860 (first optical sensor sensitive to infrared light). Then, the spatial temperature distribution characteristics within the screen are calculated. A person's head temperature determination unit (person's body temperature determination unit) 125 calculates the individual head position obtained from the normal camera image recognition unit 112 (second image recognition unit) on the spatial temperature distribution characteristics in the screen. The information is superimposed to obtain the head temperature (body temperature) of each individual.

Next, as shown in step 06 (S06), inside the human head temperature determination unit (human body temperature determination unit) 125, the head temperature (body temperature) of each individual exceeds a predetermined value (for example, 37.5 degrees Celsius). "Set temperature determination" is performed to determine whether or not As a result of this set temperature determination, if there is no person whose body temperature exceeds the predetermined value, the process proceeds to step 07 (S07) to end the process.

In the embodiment of FIG. 5, two types of "alarm value" (for example, 37.5 degrees Celsius) and "caution value" (for example, 37 degrees Celsius) are set as judgment values (predetermined values) for "setting temperature judgment". Here, the "warning value" is set to a value higher than the "caution value".

If the determination result in step 06 (S06) is equal to or greater than the "alarm value", an alarm is notified as notification processing shown in step 10 (S10). This specific notification method is based on the application (IF-THEN) rule 70 set by the detection operation instruction unit (application (IF-THEN) engine) 90, and the detection operation setting unit 140 sets “alarm notification”. Select the appropriate option 1240-1290.

On the other hand, if the determination result in step 06 (S06) is equal to or less than the "alarm value" and equal to or more than the "caution value", the result of "5) distance detection" is taken into account in the determination. As an example of how to use the result of “5) Distance detection,
[β] For a person whose distance from the infrared camera 806 is longer than a certain distance (set value), even if the temperature is lower than the judgment temperature, the detection operation instruction unit (application (IF-THEN) engine) 90 warns the person. I will issue an instruction to urge you.”

As the distance between the infrared camera 806 and the person to be measured increases, the measured result (body temperature) becomes lower than the actual temperature (details will be described later with reference to FIG. 19). Therefore, as shown in step 08 (S08), it is determined whether or not the distance between the head position of the person to be measured and the infrared camera 806 is greater than a predetermined set value. If this distance is closer than the set value (No in S08), it is considered that 'the accuracy of the measured body temperature of the person to be measured is high'. In this case, it is judged that "there is no person with a high fever", and the process ends (S09).

On the other hand, when the corresponding distance is farther than the set value (Yes in S08), it is considered that "the measured value accuracy of the body temperature of the person to be measured is low". In other words, it is presumed that 'even though the subject of measurement is a person with a high fever and his body temperature exceeds the "alert value" (37.5 degrees Celsius), the correct body temperature could not be measured due to the poor measurement accuracy of the thermography.' do. Then, as shown in step 11 (S11), "caution" notification processing is performed.

In this case as well, according to the application (IF-THEN) rule 70 set by the detection operation instruction unit (application (IF-THEN) engine) 90, the detection operation setting unit 140 selects an option suitable for “notification”. Operate 1240-1290.

FIG. 6 shows a usage example (use case) of the edge computer 6 of FIG. 2 corresponding to the high fever detection device or the high fever detection system including it. The high-temperature person detection system shown in FIG. 6 is composed of a three-lens infrared camera 806 having a structure described later in FIG. 20 and an edge computer 6 (high-temperature person detection device) such as a smartphone. Both are connected via the connection section 10 .

Here, many people operate smartphones while riding in a vehicle (such as a train) or in a place where people gather (such as a restaurant). The upper part of FIG. 6 shows a situation in which a group of people are nearby in a vehicle or in a place where people gather. The lower part of FIG. 6 shows a situation in which another application program 22 is used on the smart phone (edge computer 6). If there is a high fever person in the nearby group, the detection result is displayed on the display screen 12 (1230).

As an example of the warning screen display 1230 controlled by the detection operation setting unit 140, the detected high fever person is indicated by diagonal lines. At the same time,
"Warning There is a person with a high fever nearby! !
Please be careful! 』
is also displayed.

In the method shown in FIG. 6, the user is automatically notified when another application program 22 is used, so that the immediacy of the notification and the user's convenience are greatly improved.

Note that in the example of use scene (use case) of the high fever person detection system shown in FIG. Not limited to this, for example, a warning screen display 1230 may be displayed on an HMD (Head Mounted Display) 810, which will be described later with reference to FIG.

Another example of this embodiment will be described with reference to FIG. In the system configuration shown in FIG. 7, the cloud server 2 and the edge computer 6 are configured to be communicable 18 with each other. The edge computer 6 also includes various devices 8 such as a sensor 802, a signage (electronic bulletin board) 804, an infrared camera 806, a mobile terminal 808 such as a smartphone or a tablet, an HMD (Head Mounted Display) 810, and an IoT device (such as a gate) 820. can be controlled to provide various services to users. If these various devices are sensors such as a temperature sensor, a pressure sensor, a gyro, or a normal camera 816, they may be incorporated in the edge computer 6. FIG. Not limited to that, some of these various devices may be arranged outside the edge computer 6 and controlled by communication.

The edge computer 6 has an application (IF-THEN) engine 90 inside, and performs processing (various device 8) to provide services to the user.

Further, as shown in FIG. 7, various devices such as a sensor 802, a signage (electronic bulletin board) 804, an infrared camera 806, a mobile terminal 808 such as a smartphone or a tablet, an HMD (head mounted display) 810, and an IoT device (gate, etc.) 820 8 are pre-installed in the edge computer 6 (pre-installed from the cloud server 2). An application (IF-THEN) engine 90 operates these microservices 80A to 80F to perform cooperation control among various devices 8. FIG. At this time, an API (application program interface) command is issued from the application (IF-THEN) engine 90 to the microservices 80A to 80F to be operated.

In this embodiment, the IoT server microservice 80G that cooperates with and controls the IoT service server 220, the mailer microservice 80H that cooperates with the mail server 230, and the Web service server 210 for obtaining Web services. The web microservice 80I or the like may be owned (pre-installed in the edge computer 6). As a specific control example of this Web microservice 80I, the control of automatically writing necessary information in the frame of a form element (Form Element) specified in HTML (Hyper Text Mark-Up Language) and the transition between Web pages. You may perform control etc. By setting the area of the web microservice 80I in the software architecture of the edge computer 6 in this way, users can enjoy advanced services such as web services. Not limited to the IoT server microservice 80G and the mailer microservice 80H shown in FIG. 7, the microservices 80 that can individually access all services available in the cyber space (not shown) are placed in parallel with the Web microservices 80I. You may Examples of services in the cyberspace not shown in FIG. 7 include automatic trading of securities (stocks of a specific brand), ordering of specific products and automatic transfer of received product prices (billing processing), movie and music appreciation tickets. User services such as a purchase application for a mobile phone or a reservation application for a predetermined trip may be provided.

In this way, if the microservices 80 that can individually access each service on the cyberspace can be arranged (installed in advance) within the software architecture of the edge computer 6, the user can create a cooperative service that connects different services on the cyberspace. can enjoy

By the way, conventionally, cooperation between services using various devices 8 existing in the real space and services in the cyber space was relatively weak. As shown in FIG. 7, microservices 80A to 80F that control various devices 8 can be arranged in the same line as the Web microservice 80I. Users will be able to enjoy advanced cooperative services that seamlessly span services in cyber space without any sense of incongruity.

When technology was born to handle video and audio, which had been handled as analog data in the past, as digital data, the word "multimedia", which refers to the ability to handle all types of media in a digital and integrated manner, became popular. became. And today, such a digital world has become a natural world.

Dedicated control software (microservice 80) that individually controls various devices 8 responsible for detecting and manipulating objects and/or instances existing in real space and their states is newly provided in this embodiment. By defining it, we will build a "multi-service" world that will enable the provision of more advanced services that combine (mix) services that can be provided in real space/physical space with services that can be provided in cyberspace. can. Application (IF-THEN) rules 70 stipulate "methods of combining various services" for providing advanced services to users.

The relationship between FIG. 7 and FIG. 2 described above will be described. The normal camera 816 in FIG. 2 corresponds to one type of the sensor 802, and the infrared camera 806 in FIGS. 2 and 7 means the same device 8. Therefore, the normal camera image recognition unit 112 in FIG. 2 corresponds to the sensor microservice 80A. 2 correspond to the infrared camera microservice 80C. In FIG. 2 , the result of image recognition performed by the normal camera image recognition unit 112 is transferred to the human head temperature determination unit 125 . In FIG. 7, information is exchanged between the sensor microservice 80A and the infrared camera microservice 80C via the application (IF-THEN) engine 90. FIG.

A detection operation instruction unit (application (IF-THEN) engine) 90 of the detection rule setting unit 130 corresponds to the application (IF-THEN) engine 90 shown in FIG.

On the other hand, the function of the detection operation setting unit 140 that controls the mail transmission 1250 in FIG. 2 is processed by the mailer microservice 80H. When the screen display 1230 is performed on the display unit of the HMD 810, the function of the detection operation setting unit 140 for controlling the screen display 1230 is processed by the HMD microservice 80E.

Then, for example, when performing a device operation 1290 (1270) such as "blocking the passage gate for restricting the passage of a person with a high fever detected", the function of the corresponding detection operation setting unit 140 is set to the individual device microservice 80F processed by

Regarding the application (IF-THEN) rules 70 and various devices 8 used by the individual edge computers 6, the functions executed by the cloud server 2 include a rule setting/distribution unit (function) 202 and a device management unit (function) 206. have However, the cloud server 2 may have a function as (a function of or a module of) a microservice registration unit 204 (the terminology definition regarding [module] will be described later).

Various data obtained when the edge computer 6 provides services to the user according to the application (IF-THEN) rules 70 are managed/accumulated 208 in the cloud server 2 and stored in the data/information storage device 200 as appropriate.

FIG. 8 shows an example program configuration used in an Android (registered trademark)-based smart phone. The ifLink™ widget 1100 displays the state of the ifLink.

Microservices 80A to 80F written in the Java (registered trademark) language are installed in a lower area of the ifLink application 1000. FIG. As part of this ifLink application 1000, an application (IF-THEN) engine 90 operates. Application (IF-THEN) rules 70 are also used in this ifLink application 1000 . That is, the ifLink application 1000 manages the microservices 80A to 80F connected to the ifLink and the application (IF-THEN) rule 70. FIG.

The relationship between the inside of the edge computer 6 shown in FIG. 7 and the cloud server 2 will be described in detail with reference to FIG. An ifLink application 92, which is a service providing application program, is installed in the edge computer 6 in advance. As a software architecture of the ifLink application 92, an application (IF-THEN) engine unit 90, a setting/management screen configuration unit 98, a data transmission unit 96, and a rule reception unit 94 exist as components. The edge computer 6 also has a memory area for storing application (IF-THEN) rules 70 described in XML (Extensible Markup Language) format, for example. The application (IF-THEN) engine unit 90 then operates the corresponding microservice 80 according to the application (IF-THEN) rules 70 .

Although the ifLink application 92 is described as a "software program" here, the ifLink application 92 may be configured by "hardware". In this case, the memory area for storing the application (IF-THEN) rule 70, the application (IF-THEN) engine unit 90, the setting/management screen configuration unit 98, the data transmission unit 96, and the rule reception unit 94 are physically arranged. It may be composed of physically separated electronic circuits.

The software architecture of the cloud server 2 is, as shown in FIG. ) 214 , a Web-API control unit 218 , and a data/information storage device 200 .

The rule setting/distribution unit 202 manages the setting of the application (IF-THEN) rule 70 according to the user's request and the distribution of the application (IF-THEN) rule 70 to the user.

Data collected by the ifLink application is transferred from the data transmission unit 96 of the edge computer 60 to the data collection service unit (DDS) 216 of the cloud server 2 via the communication path 18 . The data organized by the data collection service unit (DDS) 216 is then sequentially stored in the data/information storage device 200 .

Application (IF-THEN) rules set according to user requirements are also stored in the data/information storage device 200 . Then, the application (IF-THEN) rule 70 is read out from the data/information storage device 200 at the necessary timing, and sent from the endpoint management unit (EPM) 214 via the communication path 18 to the rule reception unit 94 of the ifLink application. It will be delivered. The application (IF-THEN) rule 70 received by the rule receiver 94 is stored in the storage area of the edge computer 6 (not shown). A microservice 80 that has been authenticated and registered by a predetermined organization is also installed in the edge computer 6 via the endpoint management unit (EPM) and the rule reception unit 94 .

On the other hand, various data collected from the ifLink application and stored in the data/information storage device 200 are processed by various application services 1800 via Web-API. As specific service contents of this application service 1800,
〇Visualization of data collected from the ifLink app 1802
〇 Management of data collected from the ifLink app 1804
〇 Status monitoring and abnormality handling 1806 regarding the status of the edge computer 6 and the status of various modules 9 using data collected from the ifLink application (description of module 9 will be described later)
〇 Analysis of data collected from the ifLink app 1808
〇 Optimization 1810 of services for users using the ifLink application and optimization 1810 of module 9 setting conditions
o Information service 1812 using information obtained as a result of data analysis 1808
etc.

FIG. 10 shows functions implemented in the application (IF-THEN) engine 90 shown in FIGS. Specific functions include activation 902, rule management control 904, rule execution control 906, sensor data control 910, JOB control 912, log output function 914, same event ignoring time setting function 916, IF across other edge computers 6 -THEN function 918, rule enable/disable function 920, execution time control function 922, intermediate data holding object function 924, and so on.

Activation 902 establishes a connection with the ifLink application 92 when the application (IF-THEN) engine 90 is activated from the ifLink application 92 .

The rule management control 904 reads the pre-stored application (IF-THEN) rules 70 when the application (IF-THEN) engine 90 is started, and constructs the internal state. That is, for example, it is in a state of waiting for an actual detection output from the sensor 802 or an instruction (a state of preparation with several options).

Based on the sensor data information notified from the ifLink application 92, the rule execution control 906 extracts the application (IF-THEN) rule 70 that matches the conditions based on the internal state. That is, a rule is selected according to the content of the detection output (command content).

The sensor data control 910 receives the device information 68 from the ifLink application 92 and activates execution control corresponding to the application (IF-THEN) rule 70 .

JOB control 912 notifies the ifLink application 92 of necessary JOB information.

The log output function 914 outputs a log (for example, service history) according to the log level (importance or detail of log contents).

The same event ignoring time setting function 916 means a function of suppressing occurrence for a specified time even if similar sensor data is input.

The IF-THEN function 918 across other edge computers 6 notifies the JOB to another specific edge computer 6 and causes the JOB to be executed there.

The rule enable/disable function 920 means a function for switching ON/OFF of a rule designated as an execution (THEN) 78 (described later with reference to FIG. 11) in the application (IF-THEN) rule 70 .

The execution time control function 922 means a function that can shift the timing of a specific execution (THEN) 78 and perform subsequent execution (THEN) 78 (described later using FIG. 11) after a certain period of time.

The intermediate data holding object function 924 means a function that holds intermediate data generated during processing in a storage device and enables input to a condition (IF) 72 (described later using FIG. 11) of another rule. do.

FIG. 11 is a block diagram expressing the concept of an application example in this embodiment. In the world of "multi-services" proposed by this embodiment, for example, change detection results of entities and states occurring in the physical real world (or information related to them) 7 and services in cyber space (Web services 210) are directly By cooperating, it is possible to provide services to users.

In other words, in FIG. 7, the web service server 210 and the mail server 230 are explicitly shown as a form of service to users using devices other than the device 8 . On the other hand, here, various devices 8 (equivalent to 802-820 in FIG. 7) as means for providing services to users in physical real space and means for providing services to users in cyber space (for example, Web screens provided by the Web service server 210) are collectively referred to as "human recognizable entities, status, and/or information (instance, status, and/or information) 7 (corresponding to 7A-7F)". In addition, the device 8 described above corresponds to one type of "human recognizable entity/state/information 7". General-purpose means for controlling these individually is defined as "control means 4 (4A-4F)".

As described above, FIG. 7 explicitly shows the web service server 210 and the mail server 230 as a form of service for users using devices other than the device 8 . On the other hand, here, "human recognizable entity or state, information (instance, status, and/or information) 7 (7A-7F)" is newly proposed. The aforementioned device 8 corresponds to one type of "human recognizable entity/state/information 7". General-purpose means for controlling these individually is defined as "control means 4 (4A-4F)".

For example, using an AI (Artificial Intelligence) program to control a web screen (control to automatically write necessary information in the frame of form elements specified in HTML, control transitions between web pages, etc.), or use a web screen Consider an example of order processing, contract processing, and application processing. In this case, the information on the web screen (for example, the “format” on the web screen to be written at the time of order processing, contract processing, application processing) is a kind of “human recognizable entity or state, information 7” and controls the information (as a specific example, the process of automatically filling in the necessary items in the “form” on the web screen and pressing the “execute” button on the web screen after confirmation by the user, etc.) perform) corresponds to the control means 4 .

As another example, consider a case where big data collected using the sensor 802 is stored in the data/information storage device 200 and then analyzed using statistical analysis software. In this case, the single sensor 802 is classified as the device 8, and therefore corresponds to one type of "human recognizable entity/state/information 7". A sensor microservice 80A that controls the operation of the sensor 802 is a kind of control means (microservice) 4. FIG.

Further, the data/information storage device 200 used for storing big data is classified as a device 8, and therefore corresponds to one type of "human recognizable entity/state/information 7". The big data stored in the data/information storage device 200 and the information obtained after data analysis can also be classified as one type of "human recognizable entity/state/information 7". In comparison, data analysis processing performed by statistical analysis software falls under the broad definition of "data control." Therefore, this statistical analysis software corresponds to one type of control means (microservice) 4 .

Here, as a function implementation mode of the control means 4 shown in FIG. 11, it may be implemented by a hardware configuration (for example, a combination of logic circuits) or by a software program. For example, the control means 4 that takes the form of a program using an object-oriented programming language is called a microservice 80 . Of course, microservices created in a non-object-oriented programming language (for example, assembler) may also be used.

The device 8 and the cyberspace service providing means are collectively (generalized) named "recognizable entity/state/information 7", and the generalized generic name including the microservice 80 is named "control means 4". Calls and all forms of service to users will be described generically hereinafter.

For example, the sensing function can be realized only after the "control means 4" controls the sensor 802 alone. A means for realizing a predetermined function related to the service provided to the user is called "module 9 (9A-9F)". The module 9 basically consists of "recognizable entity/state/information 7" and "control means 4".
As an installation form of this module 9, the entire module γ9C may be built in the edge computer 6. FIG. Alternatively, only the recognizable entities/states/information 7D, 7E of the modules 9D, 9E may be arranged outside the edge computer 6. FIG. In this case, the control means (microservices) 4D, 4E arranged (pre-installed) in the edge computer 6 and the recognizable entities/status/information 7D, 7E are signal-connected by the communication means 18. ing.

Further, the control means (microservices) 4A and 4B may be present not only in the edge computer 6 but also in the cloud server α2A.

An application rule β70B that defines the method of cooperation between the module γ9C and the module δ9D that is performed to provide a predetermined service to the user is basically a basic transition over time from condition (IF) β72B to execution (THEN) β78B. Consists of logic.

Using the logic example (application rule β70B) of "turn on the lights when it gets dark", the "condition (IF) β72B⇒execution (THEN) β78B" logic and its specific processing will be described. In this case, an API command 75 is issued from the application (IF-THEN) engine 90 to the illuminance sensor control means (microservice) γ4C as processing corresponding to the condition (IF) β72B. Correspondingly, the illuminance sensor control means (microservice) γ4C controls the illuminance sensor γ7C of the recognizable entity/state/information 7 to measure the illuminance of the surrounding environment. When the obtained illuminance is lower than the preset value, the return value of the API command 75 is returned to the application (IF-THEN) engine 90 from the control means (microservice) γ4C of the illuminance sensor.

Then, the application (IF-THEN) engine 90 starts the execution (THEN) β78B operation according to the application rule β70B. At this time, the application (IF-THEN) engine 90 directs the signal received from the microservice to the necessary microservice based on the interpretation of the IF-THEN rule. In other words, the application (IF-THEN) engine 90 does not autonomously execute processing, but compares the data received from the microservice specified by the IF with the conditions of the rules, and when the conditions are met, Issue an execution instruction to the microservice specified by THEN.

Then, the application (IF-THEN) engine 90 issues the following API command 75 to the lighting switch control means (microservice) δ4D. Then, the lighting switch control means (microservice) δ4D controls the lighting switch δ7D, which is the recognizable entity/state/information 7, to turn on the lighting switch. Thereby, the function of the module δ9D having the function of controlling the lighting switch is executed.

Next, a case where the user purchases new modules .epsilon.9E and .zeta.9F and expands the system will be described. The control means (microservices) ε4E and ζ4F of the modules ε9E and ζ9F and the application rule γ70C combining them are generated in the cloud server β2B of the Web server. After that, they are installed 180 via the communication line 18 connecting the cloud server β2B and the edge computer 6 .

The control means (microservices) ε4E and ζ4F and the application rule γ70C installed 180 are stored in a predetermined recording area of the edge computer 6 in the form of files (for example, as a microservice file 2602 or an application rule file).

At the same time, the cloud server β2B generates an integrated rule 700 that integrates the existing application rule β70B and the newly created application rule γ70C. Then, the cloud server β2B verifies consistency between the existing application rule β70B and the newly created application rule γ70C. If any problem is found as a result of the verification, the user is notified during the above installation 180 or during service provision to the user.

The cloud server β2B can verify the application rules of each edge computer against many edge computers. As edge computers, there are edge computers in factories, hospitals, schools, government offices, farms, homes, etc., and application rules suitable for each facility are adopted.

The terms used in this patent specification, including the terms used in the above description, are defined below.

● [Module 9]--means means for realizing a predetermined function, and in the real space where this predetermined function is ) 7] and its [controller 4]. In this embodiment, basically, services are provided to users by combining operations (operations) of a plurality of [modules 9] controlled by individual [control means 5]. [Module 9] described in the present embodiment merely indicates the "concept" of "a combination for realizing a predetermined function". Therefore, the [recognizable entity or state, information 7] that constitutes the [module 9] and its [control means 4] need not be "physically integrated". For example, the [recognizable entity or state, information 7] and its [control means 4] may be installed at physically separated positions, and the communication means 18 may be used to cooperate with each other. Neither need necessarily form a "physically real object", as further explained below.

● [Human recognizable entity/state/information (instance, status, and/or information) 7]--refers to an object for realizing a predetermined function in the real space. Among the realization objects (objects) mentioned here, measurable [states] such as temperature, humidity, and pulse are also included. In this patent specification, the types of content, data/information, processing, and devices described below are recognizable in the real space [entity or state, information (instance, status, and/or information) 7] include.

• [data and/or information]--includes both acquisition data/information and provisional data/information. Here, data collected by a [device] such as a sensor corresponds to one type of acquired data, and information obtained as a result of analyzing this acquired data corresponds to one type of acquired information. Naturally, this [data] also includes measurement results of [state] such as temperature, humidity and pulse. Data and information provided to the user as [service] are classified as data/information for provision. As a specific example, educational materials, teaching materials, etc. correspond to this data/information for provision.

● [Processing]--means the process of realizing a form of implementation of a given function, and the intervention of an electronic device is a prerequisite. In other words, it includes all specific operations using electronic equipment. For example, specific operations such as order processing, remittance/remittance processing, exchange processing, contract processing (such as insurance), travel and business scheduling, and travel arrangements using a smartphone or a web screen are applicable. By the way, manual processing such as "inserting a handwritten document on an application form into a postbox" does not correspond to the [processing] described in this specification because it does not involve an electronic device.

● [Device 8]--includes all electronic devices that actually exist in the real space and are used to implement a predetermined function. Therefore, it also includes an electronic device that executes the above [processing], an electronic device that displays the above [content], and a storage device that stores the above [information/data]. This [device 8] alone may incorporate a communication function for communicating with an external electronic device. Also, this [device 8] may have a function to act on physical (or chemical) states and phenomena in the natural world. Devices that act passively on the natural world include sensors and the like. Devices that actively act on the natural world include robots, drive mechanisms, display devices (display elements), and the like. The physical form of this [device 8] is not limited to a stationary type, but may be a movable portable type. For example, a band-type sensor fixed to the user's arm or leg, a belt-type sensor fixed to the waist, or a VR (virtual reality) type or AR (augmented reality) type HMD 810 worn on the head. can take any form.

● [controller 4]--refers to means for controlling [human recognizable entity/state/information 7] in order to achieve a given function. A communication function may be incorporated therein, and the [human recognizable entity/state/information 7] may be controlled using the communication means. As an embodiment using this [control means 4], transition control between Web pages, automatic input control of necessary parts in the Web page, data analysis control of accumulated data, trade processing control of goods, automatic travel arrangement, etc. Processing control, robot motion control, etc. are included. In addition, [human recognizable entity/state/information 7] is a control means for sensors such as temperature and humidity. do Here, in this control means, detailed procedures for realizing individual detailed functions are defined. For example, detailed procedures may be defined in a “hardware form” in which control logic is formed by combining logic circuits. Further, without being limited to this, detailed procedures may be specified in a “software form” such as a program that can be installed in [edge computer 6] or [cloud server 2].

● [Micro-service 80/IMS]--means the [control means 4] formed by a program using an object-oriented programming language. Objects controlled by these microservices are not limited to devices as described above, but include processes, content, and information/data. Therefore, for example, program software using AI technology for controlling transitions between web pages and controlling screen operations corresponding to specific web pages is also included in one type or part of microservices. Similarly, program software that uses AI technology to automatically extract the number of people appearing in a specific video content, for example, and program software that analyzes acquired data (using statistical analysis, etc.) are also a type of microservice or included in some of them. The information obtained as a result of the analysis here and the recording device that stores the information are treated as [human recognizable entity/state/information 7]. Also, [microservice 80] is a form of [control means 4], so by installing [microservice 4] as described above, the control target (human Control of recognizable entities/states/information 7) becomes possible. Here, when defining detailed procedures in this "software form", if [control means 4] is formed by a program using an object-oriented programming language such as Java (registered trademark) language or Objective-C language, , effective use of already created program assets (incorporation (import) in other programs, calling (issuance of API command 75), etc.) can be performed (details will be described later in Chapter 2). Furthermore, if the program is written in OS (operating system)-independent Java language, the versatility of the [control means 4] will be improved. In the following description, it is also called IMS (ifLink Micro-service) instead of microservice.

● [Control Method]--means a program that constitutes a part of [microservice 80] and defines detailed procedures for realizing a specific individual detailed function. Individual control methods are treated as minimum functional units (ie, subprograms) that implement individual detailed functions. As an API command 75 from [application engine 90], a control method name corresponding to a specific individual detailed function can be called as a corresponding function (command). As a result, the [application engine 90] can execute the individual detailed function corresponding to the control method name. In this embodiment, the [control method] can be linked to the [application rule 70] as described above by describing the [control method] program (regulating the program content) using an object-oriented programming language. (Detailed explanation with specific examples will be described later in Chapter 2.)
● [Control Class]--In each [Control Method], it means a group of aggregates collected for each common similar function. A control method corresponding to the initial setting function of an instance (entity) to be created is called a constructor. Constructor names (control method names corresponding to constructors) included in the same aggregate may be matched with control class names. In the [microservice 80] described above, a management unit in which control methods are grouped for each common similar function corresponds to a [control class]. Therefore, a plurality of different [control classes] may exist in the [microservice 80] for the same purpose.

● Control Package--Represents a collection of classes with similar functionality. That is, they are defined according to a mechanism for managing groups of classes having similar functions by dividing them into folders. Particularly in this embodiment, control packages are divided for each entity/status/information 7 included in the same module 9, and different control package names (identification information for each control package) are individually set. Therefore, it is possible to identify the contents of each entity/state/information by using the individually set control package name (identification information for each control package). Here, when setting the name of the [control package] for controlling the [device] belonging to the entity/state/information, the identification information of the manufacturer or seller of the module (or device), the module (or device) ), individual manufacturing numbers, etc. may be used (set) as identification information for each control package.

● [microservice file 2602 ] -- indicates a storage unit (storage format/storage format) for storing [microservice 80 ] in the storage area of the cloud server 2 or edge computer 6 . When the [microservice 80] is described in Java language, for example, the content of the [microservice 80] is a chain of Java language descriptions. In order to store this information, a file is configured for each control class (which also describes the processing details for each control class). The extension of this control class file is ".class". A "folder" for each control package is placed in the storage area of the cloud server 2 or the edge computer 6, and the control class file is stored therein. This folder name should match the control package name. Then, since the manufacturer/seller identification information of the module (or device) and the type information of the module (or device) can be known from this folder name, it becomes easy to search for the necessary control class file.

[API (API command) 75]--[application (IF-THEN) engine 90] and [microservice 80] for operating individual microservices 80 from [application (IF-THEN) engine 90] indicates a means of communication (communication tool) between Each [microservice 80] has a hierarchical structure of control package/control class/control method. In [API (API command) 75], in many cases, a specific control method name indicating a detailed procedure (program) for realizing an individual detailed function is specified, and the individual detailed function operation is performed for each control method. . An example of a method for executing the individual detailed function of this specific control method with an application program 22 (including [ifLink application (service providing application program)] described later) written in Java language, for example, will be described. In the class of this application program 22, first, the control class of the control package containing the corresponding control method is incorporated (imported). By designating the corresponding control method name within the class or within the method, this individual detailed function can be executed.
● [Application (IF-THEN) rule 70]--Indicates stipulation information of a method (rule) for combining a plurality of different control means 4 (or modules 9), which indicates a method of providing a predetermined service to a user. When [application (IF-THEN) rule 70] is composed of hardware composed of, for example, a combination of logic circuits, the output terminal of this combinational logic circuit is composed of a combination of logic circuits of [control means 4]. It is electrically connected directly to the input terminal side of the control logic to be connected. On the other hand, if this [application (IF-THEN) rule 70] is defined in a description format (including HTML that constitutes a web screen) in accordance with a predetermined description method, the cloud server 2 or the edge computer 6 application ( The IF-THEN) engine 90 decodes the contents of the [application (IF-THEN) rule 70]. Based on the decryption result, an API command 75 is issued from the application (IF-THEN) engine 90 to the corresponding microservice 80, and the microservice 80 controls the recognizable entity/state/information 7. be started.

● [Unified Rule 700] -- Generated when multiple different [Application (IF-THEN) Rules 70] are defined for the same edge computer 6 or the same ifLink application. The purpose of generating this [integration rule 700] is to discover in advance contradictions and problems that may occur when multiple different [application (IF-THEN) rules 70] are combined, and to solve problems when providing services to users. in avoidance. When a user desires to specify a plurality of different [application (IF-THEN) rules 70], cloud server 2 first generates [integration rule 700], and cloud server 2 follows this [integration rule 700]. Manipulation is simulated. If a contradiction or problem occurs in the simulation results, the user is notified and a remedy is proposed. After confirming that no inconsistency or trouble occurs in the simulation results, this integration rule 700 is installed 180 from the cloud server 2 to the edge computer 6 . After this installation 180, the edge computer 6 operates various microservices 80 (control means 4) according to this [integration rule 700].

● [Edge computer 6]--has [application (IF-THEN) engine 90], [application (IF-THEN) rule 70] and various [microservices 80] necessary for providing services to users corresponding to it stored in advance. When a user wants to receive a predetermined service from [Place 1], [application (IF-THEN) rule 70] and [microservice 80] corresponding to it are generated based on the request from the user to [Place 1]. is installed in advance from the cloud server β2B. [Edge computer 6] is composed of a processor, a memory unit, and a communication unit. [Application (IF-THEN) rule 70] installed here and [Microservice 80] corresponding thereto are stored in the memory unit. In addition, the above-mentioned processor bears the function of [application (IF-THEN) engine 90] according to this [application (IF-THEN) rule 70]. As long as the edge computer 6 is composed of a processor, a memory unit, and a communication unit in this way, the [edge computer 6] can take any form. As a specific form example, a personal computer, a smart phone, a tablet, a signage, a gateway, a router, etc. may function as the [edge computer 6].

● [ifLink application (service providing application program) 92 ]--means a program processed by the cloud server α2A or the edge computer 6 in order to provide a predetermined service to the user 1700 . This is involved in the communication between the [application (IF-THEN) engine 90] that executes processing according to the [application (IF-THEN) rule 70], the setting/management screen unit 98, and the cloud server 2. It consists of a data transmission 96 and a rule reception section 94 . The [ifLink application (service providing application program) 92] is programmed to first refer to the contents of a pre-stored predetermined [application (IF-THEN) rule 70]. Then, the cloud server α2A or the [application (IF-THEN) engine 90] of the edge computer 6 executes processing according to the contents programmed in this [ifLink application (service providing application program) 92], and the user 1700 provide the prescribed service;

● [Application (IF-THEN) engine 90]--It is built in the cloud server α2A or the edge computer 6, and means a place (function) where processing/execution within the cloud server α2A or the edge computer 6 is performed. In terms of hardware, an arithmetic processor (or its processing state) may correspond. This [application (IF-THEN) engine 90] reads the [application (IF-THEN) rule 70] and decodes its contents. Then, referring to the decoded contents, the processing according to the contents programmed by the [ifLink application (service providing application program) 92] is executed. In the middle of the processing/execution, the necessary [API command 75] is issued to the microservice 80. The inside of this [application (IF-THEN) engine 90] is a programming language interpretation engine corresponding to the description format (corresponding programming language) in which the [application (IF-THEN) rule 70] is described, and the interpretation result. It is composed of a control engine that executes processing according to a program defined in [ifLink application (service providing application program)] while referring to it. A web browser function may be built in to correspond to the case where the [application (IF-THEN) rule 70] is described (expressed) in HTML.

● [Cloud server 2]--has [application (IF-THEN) engine 90], [application (IF-THEN) rule 70] and various [microservices 80] necessary for providing services to users corresponding to it stored in advance. When a user wants to receive a predetermined service from [Place 1], [application (IF-THEN) rule 70] and [microservice 80] corresponding to it are generated based on the request from the user to [Place 1]. to the edge computer 6. As an example of this transmission method, if a link function (Anchor Element) specifying a URL in HTML is used, the [cloud server 2] may have a Web server function.

● [Service] --- Combining the control of the prescribed [entity/state/information 7], the prescribed [processing] or [state] changes, or the [contents] or [data/ information]. In this embodiment, the [service] is provided to the user according to the [application (IF-THEN) rule 70]. As a form of providing this [service], in this embodiment, a service that transcends (straddles) a barrier between a service in a real space or a physical space using various devices 8 and a service in a cyber space using a web service or the like is provided. Combined (composite) services may be provided.

Concepts of application examples in this embodiment described with reference to FIG. 11 will be collectively described below. That is, the module 9 consists of a human recognizable entity or state/information 7 and a control means 4 for controlling it. A plurality of modules 9 can be defined. That is, the first control means γ4C for the first module γ9C and the second control means δ4D for the second module δ9D are individually defined.

Here, an application rule 70 for providing services to the user is set by the combination of the first module γ9C and the second module δ9D. API commands 75 are individually issued from the application (IF-THEN) engine 90 to the first control means γ4C and the second control means δ4D in accordance with the content of the application rule 70 to perform the first or second control. Operate means γ4C and δ4D. The module control method in this embodiment is used.

A microservice 80 is included in the control means 4 described above. The microservice 80 may be formed (written) by a program using an object-oriented programming language. (However, the microservice 80 is not limited to this, and may be written in any programming language.) Here, the first microservice 80 (control means γ4C) and a second human-visible entity or state, a second microservice 80 (controller δ4D) that controls information δ7D is defined. An application rule β70B is set for providing a service to the user by combining the first microservice 80 and the second microservice 80. FIG. Then, in response to the API command 75 issued from the application (IF-THEN) engine 90 according to the application rule β70B, the first entity or state recognizable by these first/second microservices 80, information γ7C, δ7D individually controlled.

Further, the edge computer 6 in this embodiment (application example) receives the application rule γ70C (and the integration rule 700) and the microservice 80 (control means 4F) from the cloud server β2B, and installs them 180 in advance. As a result, a first microservice 80 (controlling means ε4E) that controls a first human-recognizable entity or state, information ε7E, and a second human-recognizable entity or state, information ζ7F. A second microservice 80 (controlling means ζ4F) and related application rule γ70C (and integration rule 700) are built in the edge computer 6 in advance. Here, the application rule γ70C (and the integration rule 700) is a rule for providing services to users by combining the first microservice 80 (control means ε4E) and the second microservice 80 (control means ζ4F). stipulated. Then, the edge computer 6 operates the first microservice 80 (control means ε4E) and the second microservice 80 (control means ζ4F) to obtain a human-recognizable second entity or state, information ζ7F or human separately controls a second entity or state recognizable by , information ζ7F, to provide services to the user.

In particular, the edge computer 6 in this embodiment (application example) has an application (IF-THEN) engine 90 . Then, according to the rules defined by the application rule γ70C (and the integration rule 700), the application (IF-THEN) engine 90 sends the API command 75 to the first microservice 80 (control means ε4E) and the second microservice 80 ( Issue to the control means ζ4F). As a result, the first or second microservice 80 (control means ε4E, ζ4F) individually controls recognizable entities or states, information ε7E, ζ7F to provide services to users. Then, in the service providing method in this embodiment (application example), the above-described method is used to provide the service to the user.

In this embodiment (application example), as described above, the contents of the microservice 80 can be described in an OS-independent programming language such as Java language.

The effect will be described with reference to FIG. In the case of Java, which is an OS-independent general-purpose programming language, translation regions α1300A and β1300B (Virtual Machines) corresponding to individual OS layers α30A and β30B are prepared in advance. The contents of the microservice 80 written in Java language are translated via the translation areas α1300A and β1300B (Virtual Machines) and delivered to the individual OSs α30A and β30B.

In particular, commands of the microservice 80 in this embodiment are described by a combination of basic API commands 75 of the OS layer α30A and β30B levels. As a result, there is an effect that the basic control of each device can be performed in detail.
FIG. 13 shows the class configuration (software architecture) of the microservice 80. As shown in FIG. In this chapter, the microservice 80 will be mainly described, but the content described below can be applied not only to the microservice 80 but also to the control means 4 in a broad sense.

As a basic function required for the microservice 80, "control function of each corresponding device 8" is an essential condition. However, not limited to this, "peripheral functions related to control of individual devices 8" are also required as functions of microservices 80. FIG. As a specific example, it is necessary to perform interface processing with an application (IF-THEN) engine, and to check whether the corresponding device 8 is in an operable state.

In the present embodiment, the microservices 80 are separated into different classes according to functions different from the above, and the functional expandability of the microservices 80 is improved. In other words, the Custom Device Class 2102 is set in order to execute the "control function of each corresponding device 8". Then, a plurality of methods in which detailed procedure program contents for each individual detailed function of the corresponding device 8 are described are arranged (described) therein.

On the other hand, as a program for executing "peripheral functions related to control of individual devices 8", a control class (Customims Class) 2110 within microservices corresponding to individual devices is installed.

Here, the individual control method of the individual device control class (Custom Device Class) 2102 is called (incorporated) from the control class (Customims Class) 2110 within the microservice corresponding to the individual device described above to make it executable 76 . Specifically, using an import sentence before the control class (Customims Class) 2110 in the microservice corresponding to individual devices
import [Individual device control package name].[Individual device control class name (Custom Device)];
, it is possible to incorporate 76 the individual control class in the individual device control class 2102 in the individual device corresponding microservice control class (Customims Class) 2110 . In the individual device control class (Customims Class) 2110 or a specific control class therein, the individual control method name, argument, and return value format (type) in the individual device control class 2102 are described. This allows invocation/use 76 of custom control methods in the custom device control class 2102 within the custom control class (Customims Class) 2110 or a specific control class therein.

In this embodiment, the optimum data E60 that matches the characteristics of each device 8 to be controlled can be transmitted to the device 8. FIG. In order to make this possible, in the present embodiment, different individual device control packages 2100 can be set for different manufacturers/sellers and device models.

On the other hand, it is necessary to guarantee compatibility and future expandability between systems delivered to users. In order to ensure such inter-system compatibility and future expandability, a template class may be provided as a reference for the description content (program) for each class. That is, for the individual device control class (Custom Device Class) 2102, a device control template class (Base Device Class) 2002 is provided to any programmer as a template class. In addition, for the control class (Customims Class) 2110 within the microservice corresponding to individual devices, a control template class (BaseIms Class) 2010 within the microservice is provided to any programmer as a template class.

A programmer who intends to program a new microservice 80 creates an individual device control class (Custom Device Class) 2102 and a control class (Customims Class) 2110 within the microservice corresponding to individual devices only by minor modification (customization) 74 of the above template contents. Therefore, a programmer who creates a new microservice 80 can efficiently create the microservice 80 in a short period of time.

And to show that each class extends 74 the function of the original template class, at the place where each class name created individually is defined, for example
public class Custom Device extends Base Device {
, the relationship of function inheritance (Extends) 74 between classes is specified. Using such a description method not only facilitates management of individual microservices 80, but also facilitates compatibility and extensibility between systems.

By the way, the above "public" has the meaning of "allowing the use (diversion) of the Custom Device Class in other systems". If you write "private" instead of "public" in this place, use other than your own class (Custom Device Class in the above example) is prohibited. Also, if it is described as "protected", it can be used only in the class of the package and the class that inherits the function 74. If many programmers are involved in the creation (modification) of the microservice 80, troubles such as unintended tampering are likely to occur. Making rules as described above has the effect of reducing the frequency of troubles occurring between multiple programmers.

As shown in FIG. 13, an API command (cooperation designation) 75 from the application (IF-THEN) engine 90 is processed by the microservice control template class (BaseIms Class) 2010 .

Also, during the provision of a predetermined service to the user 1700, for example, an abnormal situation such as the device 8 running out of battery may occur. An anomaly detection class (Health Check Task Class) 2016 having a function to cope with an emergency that occurs in an emergency is arranged to enable quick response to the anomaly. This anomaly detection class (Health Check Task Class) 2016 is called/implemented 76 from the microservice control template class (BaseIms Class) 2010 . The invocation/insertion processing 76 here is handled in the same manner as described above.

Especially in this embodiment, AI processing such as image recognition is required. Therefore, in the basic control package 2000, an image recognition class (AI Analyze Class) 2018 using AI that specializes in image recognition processing using AI technology is arranged.

The normal camera image recognition unit 112 in FIG. 2 (corresponding to the sensor microservice 80A in FIG. 7) 1) performs image recognition (image analysis) of the image signal,
2) Determining whether or not there is a person in the screen (person detection),
3) If there are people in the screen, calculate the number of people (detect the number of people),
4) Recognize the position of the head of each person (head detection) only when there are people in the screen,
5) It has been described that a series of image recognition processes are performed to calculate the distance (distance detection) between each person identified in the screen and the infrared camera 806 . Correspondingly, in an image recognition class (AI Analyze Class) 2018 using AI, individual methods for performing the above individual processes [1] to [5] are arranged (programmed).

Then, using an API command corresponding to each method, each corresponding method is individually called from within the microservice control template class (BaseIms Class) 2010 or the individual device corresponding microservice control class (Customins Class) 2110, and 76 processes are performed. be done. Here, each corresponding method is sequentially called 76 according to the flow procedure shown in FIG.

It has already been explained that the image recognition processing related to "2) person detection" performed by the normal camera image recognition unit 112 in FIG. 2 requires flexible processing according to the usage scene. For example, since the number of visitors who visit the reception counter at the same time is often less than several, individual identification of persons is relatively easy. However, the normal camera 816 installed behind the reception counter can only photograph the upper body of the visitor.

Also, when the normal camera 816 is used to photograph the aisle, it is possible to photograph the whole body, but on the other hand, it is necessary to recognize a moving person. In other words, high-speed detection is required for high-temperature person detection in passageways. On the other hand, when detecting a person with a high fever in a classroom where about 40 students have gathered, it is necessary to simultaneously measure the body temperatures of many people.

In this way, image recognition classes (AI Analyze Class) 2018 may be arranged in the same basic control package 2000 at the same time. (However, in this case, the class name changes for each class.) Then, from the microservice control template class (BaseIms Class) 2010 or the individual device corresponding microservice control class (Customims Class) 2110, the corresponding class name is It is processed 76 by calling each method specified and individually arranged (described) therein.

Not limited to this, the AI Analyze Class 2018 can be used for the individual applications of [1] to [5] according to the usage scene such as "reception counter", "inside the passage", and "inside the classroom". All individual methods for processing may be arranged (programmed).

When the image recognition class (AI Analyze Class) 2018 using AI is placed in the same basic control package 2000 in this way, each corresponding class in the microservice 80 can be called 76/utilized. As a result, the optimum person detection logic can be selected for each different usage scene, enabling flexible and highly accurate image recognition and AI processing.

A method of moving data used for image recognition (or used for AI processing) in the same microservice 80 will be described. The inter-block movement of data shown in FIG. 2 and the inter-method movement of data within the same microservice 80 are substantially synonymous. For example, in FIG. 2 , the image recognition result within the block of the normal camera image identification unit 112 moves within the block of the human head temperature determination unit 125 .

The detection data of the sensor 802 acquired by the device control template class (Base Device Class) 2002 or the individual device control class (Custom Device Class) 2102 in FIG. used within the method. Data movement at this time is controlled and managed by a predetermined program of the microservice control template class (BaseIms Class) 2010 or the individual device corresponding microservice control class (Customins Class) 2110 .

Not limited to this, the detection data of the sensor 802 acquired by the device control template class (Base Device Class) 2002 or the individual device control class (Custom Device Class) 2102 is temporarily stored in the buffer memory 2008 (for temporary storage of stream data). You may save. A corresponding method of the AI Analyze Class 2018 may then read and use the necessary data therefrom.

In FIG. 13, as an example of AI processing, an image recognition class (AI Analyze Class) 2018 (data analysis processing program) that performs specialized image recognition is arranged. However, not limited to that, for example, any AI class (AI program) that specializes in a specific function using any artificial intelligence technology such as data control function, data manipulation function, automatic data collection function (using the Internet) can be used for basic control It may be arranged in the package 2000 .

If an AI class that specializes in data analysis is provided in the basic control package 2000 in this manner, advanced data analysis processing can be executed inside the microservice 80 . As a result, only the data analysis result processed inside the microservice 80 can be passed to the application (IF-THEN) engine 90, so that the processing load of the application (IF-THEN) engine 90 can be greatly reduced. Furthermore, AI classes (AI programs) specialized for specific functions using artificial intelligence technology can be arbitrarily placed in the basic control package 2000, so advanced processing using artificial intelligence technology can be performed arbitrarily in the microservice 80. It becomes possible.

When the device 8 is installed outside the cloud server 2 or the edge computer 6, the microservice 80 controls the device 8 via the communication line 18. Here, for example, if the communication line 18 is congested or disconnected, there is a risk that the system will freeze until a response is returned from the device 8 .

On the other hand, if a response time monitoring class (Timeout Check Task Class) 2006 is installed, the communication status between the microservice 80 and the device 8 can be monitored. And by detecting communication problems and taking prompt action, it is possible to prevent freezes in the system.

Also, depending on the type of the sensor device 802, it is necessary to handle various data ranging from binary data (binary data of "1" or "0") to video streams. By providing the stream control engine class (Stm Engine Class) 2004 here, the effect of efficiently handling diverse data in an integrated manner is produced. A stream analysis class/method using AI technology may be arranged inside this stream control engine class (Stm Engine Class) 2004 or at the same position as this. That is, as described above, a stream analysis class using AI technology is created using an import statement in the stream control engine class (Stm Engine Class) 2004 or the device control template class (Base Device Class) 2002 (stream control engine class (Stm Engine Class) 2002). Class) 2004 when placed in the same position). Accordingly, in the stream control engine class (Stm Engine Class) 2004 or the device control template class (Base Device Class) 2002, it is possible to use (call) /methods in the stream analysis class using AI technology.

As a result, not only is the raw stream data obtained from the sensor device 802 passed to the application (IF-THEN) engine 90, but only the analysis result information obtained as a result of automatically analyzing the stream data is sent to the application (IF-THEN) engine. It becomes possible to pass the application (IF-THEN) engine 90, and the processing load of the application (IF-THEN) engine 90 can be greatly reduced. In this case, the data/information storage device 2 in the edge computer 6 or the cloud server 2 may sequentially store raw stream data obtained from the sensor device 802 as a time-series file. Then, the method of the stream analysis class using AI technology reproduces the time-series file (raw stream data file) at appropriate times. Then, a specific condition (IF) β determination may be performed using analysis information obtained by data analysis using AI technology.

As shown in FIG. 13, the response time monitoring class (TimeoutCheckTask Class) 2006 and the stream control engine class (StmEngine Class) 2004 can be called/installed 76 from the device control template class (Base Device Class). The specific method of this calling/inserting 76 process employs the same method as described above.

In this embodiment, if the microservice 80 is described using an object-oriented programming language (such as Java language), there is an effect that the existing program assets of the basic control package 2000 can be effectively used in the individual device control package 2100 . Then, the programmer's microservice 80 development efficiency is greatly improved.

For example, using an import statement before the control class (Customins Class) 2110 in the microservice for individual devices and the control class (Custom Device Class) 2110 for individual devices
import [name of basic control package 2000].[abnormality detection class name (HealthCheckTask)];
import [name of basic control package 2000].[response time monitoring class name (TimeoutCheckTask)];
import [name of package 2000 for basic control].[stream control engine class name (StmEngine)];
can be used to program the methods defined in each class.

Furthermore, the destination of the API command (cooperation designation) 75 from the application (IF-THEN) engine 90 is changed from the microservice control template class (BaseIms Class) 2010 to the individual device compatible microservice control class (Customims Class) 2110. .

As a result of this series of processing (program change), the device control template class (Base Device Class) 2002 and the microservice control template (BaseIms Class) 2010, which are corresponding classes in the basic control package 2000, are substantially converted into individual devices. An individual device control class (Custom Device Class) 2102 corresponding to the control package 2100 and a control class (Customims Class) 2110 in the microservice corresponding to the individual device are replaced and used 71 . If the microservice 80 is described using an object-oriented programming language in this way, there is an effect that the program can be edited easily and accurately with only a very slight change in the description.

The functions of the microservice 80 set by the individual device control package will be described with reference to FIG. The three items circled in the mandatory column 2204 (initialization 2210, end 2212, start/stop 2214) are minimum essential functions. The remaining two items (sensor data transmission 2218 and job reception 2220) may be unnecessary functions depending on the characteristics of the corresponding device 8. FIG.

Initialization 2210 among the functions of the microservice indicates the function of connecting with the ifLink application and registering the microservice 80 and the device 8 to be controlled when started from the ifLink application (see the definition of terms above). .

The function of termination 2212 is a function of disconnecting and terminating the connection with the ifLink application, the microservice 80 and the device 8 to be controlled when the ifLink application is terminated.

A function of the start/stop 2214 is a function of starting or stopping the device 8 when the rule of the device 8 registered as the application (IF-THEN) rule 70 is enabled or disabled.

Next, the function of the status notification 2216 will be described. The ifLink application manages the status of registered devices. This status notification 2216 indicates a function of notifying the ifLink application to update the status of the device 8 managed by the ifLink application.

A function of the sensor data transmission 2218 is a function of transmitting sensor data sent from the device 8 to be controlled to the ifLink application.

Finally, the function of JOB reception 2220 will be explained. Via the IfLink application, the cloud server 2 or the application (IF-THEN) engine 90 notifies the content of the JOB (instruction of operation content/API). This JOB reception 2220 is a function of processing JOB information viewed from the IfLink application.

FIG. 15 shows a list of APIs provided by the microservice control template class (BaseIms Class) 2010 or the individual device corresponding microservice control class (Customims Class). That is, each control method shown in FIG. 15 is described (arranged) in the microservice control template class (BaseIms Class) 2010 . In other words, each control method described in the method/function column 2302 in FIG.

"void" and "int" described at the head of each control method described in the method/function column 2302 represent the return value type (form/type) of the control method. For example, when this control method is used in a call 76 from a predetermined class or method, after the processing of this control method is completed, "What type of data will be returned to the class or method used?" show.

For example, a control method that starts with the word "void" will have "no return value" (that is, no specific data will be returned to the using class or method after the control method has finished processing). indicates the state of

Similarly, in a control method in which the character "int" or "long" is described first, "an integer whose size is in the range of 32 bits or 64 bits" is returned after processing of the control method ends.

Then, in the control method in which the character "boolean" is written first, the "truth value of 'true' or 'false'" is returned after the processing of the control method is completed.

Also, in the parentheses of the control method, the "argument type (format/type)" and the "argument" to be handed over to the control method are described as a set with a "space" between them. If multiple "arguments" are inherited here, a "," (comma) is placed between the set of arguments.

"String" as the type (format/type) of this argument represents "character string". "Map" represents a "key/value database" in which key and value pairs are stored. "HashMap" means the class (HashMap Class) using the above-mentioned "key/value type database" or its mechanism. By using the "key/value type database" for controlling the device 8 in this way, there is an effect that data/information can be shared between methods across different control packages. Furthermore, by using a "key/value type database" as the data/information storage format, the convenience of data retrieval is improved.

"Message" means a "message" that can be directly sent to the application (IF-THEN) engine 90 or ifLink application. Also, the "constructor" described in FIG. 15 has already been explained in the definition of terms related to [control class].

"EPA", "epa", and "Epa" described in FIG. 15 are abbreviations of "End Point Access" and indicate the application (IF-THEN) engine 90 and ifLink application. In relation to this, "void onActivationResult (boolean result, EPADevice device)" is described at the 28th position in FIG. This process outline 2304 describes "registration result notification of service from ifLink application". Here, as an argument, "EPADevice" is specified as the "argument type (form/type)" and "device" is specified as the "argument". "Service registration" here means "registration of the device 8 operated by the application (IF-THEN) rule 70". And the result is in the argument "result". If "registration was successful" here, "true" is entered in "result" and returned. On the other hand, if "registration failed", "false" is entered in "result" and returned. The above argument "device" means a specific device 8 for which identification information "device" is set. "EPADevice" specifying the "argument type (form/form)" means "the device 8 that can be identified by the application (IF-THEN) engine 90 or ifLink application".

The devices 8 used in this embodiment also include portable devices 8 . Therefore, when attempting to operate the portable device 8 according to the application (IF-THEN) rule 70, there are many cases where the target portable device 8 is taken out of the operation area. Therefore, before providing the service to the user, it is necessary to confirm in advance whether the target portable device 8 exists in the operation area. This requires pre-registration of the portable device 8 whose presence in the operation area should be confirmed in advance.

For the pre-registration, the callback registration "void registerCallback( )" or "void registerCallback (String cookie)" described in the 11th or 12th item is performed. Also, for the portable device 8 that does not require prior confirmation of whether it exists in the operation area,
Execute “void unregisterCallback()” for canceling the callback described in the thirteenth.

When the target device 8 is confirmed in advance, the 14th "boolean createDevice( )" is used to perform "device registration". Conversely, for the device 8 that could not be pre-confirmed, the fifteenth "boolean deleteDevice()" is used to "delete the device".

Further, in the middle of providing the service to the user, it may become difficult to continue the service due to, for example, the battery of the specific device 8 running out. In order to deal with this situation, the 21st "void startHealthCheck (long interval)" can be used to "start health check" for each device 8. FIG. In this method, instead of "continuous health checks", checks can be performed at regular intervals specified by "interval".

When a trouble occurs, a "warning notification" is required from the microservice 80 to the application (IF-THEN) engine 90 or ifLink application. In this case, "alert transmission" can be performed by operating the 17th "int send_alert(-)" method.

Among the high fever detection performed in this embodiment, the functions required from the application (IF-THEN) engine 90 to the sensor microservice 80A and the infrared camera microservice 80C are as follows:
There are two types: a) detection (discovery) of a high fever person and b) provision of information necessary for notifying the user.

For example, in the example of use scene (use case) in FIG. 6, in the case of "a) high fever detected (discovered)", the display screen 12 warns the user by "diagonally displaying the detected high fever". In this way, "b) provision of information" necessary for "visualizing the detected high fever person" is required. By the way, this “image in which the detected high fever person is displayed with diagonal lines” is generated by the image recognition class (AI Analyze Class) 2018 using AI.

In such a case, the above (a) and (b) differ in the format of the data provided to the application (IF-THEN) engine 90 (the type (format/type) of the return value of the control method).

In order to respond flexibly to such a situation, it is necessary to specify the "transmit response data format" from the application (IF-THEN) engine 90 to the corresponding microservices 80A and 80C.

Therefore, as an API provided by the microservice control template class (BaseIms Class) 2010 or the individual device compatible microservice control class (Customims Class), the microservice control template class (BaseIms Class) 2010 or the individual device compatible microservice control Requires void setSendDataFormat(String format) to specify the data format to send from the corresponding method of the class (Customims Class).

The data type (format/type) of the argument format of this API is set as String (character string). That is, the application (IF-THEN) engine 90 designates the data format to be transmitted in text format.

Here, a large amount of data is unnecessary in the above-mentioned "a) report of detection (discovery) result of hyperthermia". In this case, long send_data(Map map) is used for the data transmission process, and the application (IF-THEN) engine 90 is answered with the information "how many people with high fever have been detected (discovered)". Even if a person with high fever cannot be detected, the answer may be that "0 persons with high fever have been detected (discovered)." The Map specified by this API means the format of "data in which key-value pairs are aggregated".

That is, when the application (IF-THEN) engine 90 requests the corresponding microservices 80A and 80C to "a) report the detection (discovery) result of a hyperthermia person", first, the void setSendDataFormat(String format) API Issue a command to execute "image recognition processing for calculating the number of people with high fever". Then, an API command of long send_data(Map map) is issued, and "a) high fever person detection (discovery) result report" is performed.

If a person with high fever is detected (discovered) here, the application (IF-THEN) engine 90 requests the corresponding microservices 80A and 80C to "b) provide information necessary for notifying the user". do. At this time as well, an API command of void setSendDataFormat(String format) is issued, but the difference in request content is specified in text format in the argument format.

As can be easily understood from the usage scene example (use case) in FIG. 6, the format of "b) information necessary for notification to the user" may include stream data such as video and still images. There are many different data compression methods for stream data such as video, still images, and audio information. Therefore, in the argument format of void setSendDataFormat(String format) issued from the application (IF-THEN) engine 90, data type, data compression method, contents of necessary stream data and its display format ) is specified.

For example, when requesting video information, the application (IF-THEN) engine 90 issues a Stream send_data (Stream movie) API command. Here, Stream means "data in stream format", and video information is included in movie.

On the other hand, when requesting still image information, an API command of Stream send_data (Stream image, long time) is issued. In this case the still image information goes into image. Here, the time information can be used to set the "time to capture (take in) still image information".

As can be easily understood from the usage scene example (use case) in FIG. 6, there are cases where an image (video or still image) is used for notification to the user. If the image to be notified to the user is generated by the application (IF-THEN) engine 90 or the microservices 80H and 80F related to the detection operation setting unit 140, the generation processing will be heavily loaded.

As in the present embodiment, the microservices 80A and 80C related to the condition (IF) 72 (FIG. 11) are caused to "b) provide information necessary for notification to the user", so that the high fever detection device ( Alternatively, there is an effect that the processing load of the entire high fever person detection system) can be reduced.

FIG. 16 shows a list of APIs (commands issued from the application (IF-THEN) engine 90) provided by the device control template class (Base Device Class) or the individual device control class (Custom Device Class). Description rules are the same as those in FIG.

Application rules 70 are basically composed of combinations of predetermined executions (THEN) 78 according to predetermined conditions (IF) 72 . For example, the recognizable entity/state/information 7 subject to this condition (IF) 72 determination changes every moment, just like the temperature, humidity, and illuminance of the user's environment change every moment. In order to cope with this time change, in this embodiment, recognizable entities/states/information 7 can be recorded chronologically on a "HashMap" corresponding to a "key/value type database".

For example, if you want to acquire the current recognizable entity/state/information 7 (that is, for example, to acquire sensor data obtained from the sensor device 802), from the microservice control template class (BaseIms Class) 2010 "HashMap createSensorData ( )” is called 76 to perform “sensor data generation”. On the other hand, if you want to “generate sensor data” at a specified time, set the specified time for obtaining data to the argument “time” of “HashMap createSensorData(long time)”. Also, if you want to acquire data (= "generate sensor data") at a specific time interval, the program of the control template class (BaseIms Class) 2010 in the microservice automatically calculates the data acquisition time according to the specific time interval. do. Then, the time obtained by the automatic calculation is sequentially specified as the argument "time" of "HashMap createSensorData(long time)".

The sensor data (recognizable entity/status/contents of information 7) thus accumulated in time series is sent from the device control template class (Base Device Class) 2002 using "long sendSensor (Map map)". “Send sensor data” is performed toward the microservice control template class (BaseIms Class) 2010 .

FIG. 16 shows a list of control classes of the device control template class (Base Device Class) 2002 corresponding to the sensor device 802 as an example. However, as shown in FIG. 7, it is necessary to control various robots (or drive mechanisms) included in the mobile terminal device 808, the HMD device 810, and the IoT equipment device 820. FIG. Therefore, not only the contents described in FIG. 16, but also the processing outline 2304 includes, for example, "execution of driving" and "display of screen/image" designated in advance.

Also, when various devices 8 are arranged outside the edge computer 6 or the cloud server 2, communication control between the devices 8 and the corresponding microservices 80 is required. Although description of the source code of the control method shown in FIG. 16 is omitted here, the control required for communication control may also be described in the source code. For example, when the "Socketlmpl Class" of the "java (registered trademark).net" package is incorporated 76, communication control for the device 8 becomes possible at the IP address level. Specifically, before the device control template class (Base Device Class)
import java.net.Socketlmpl;
describe. Based on this, it is possible to use various methods that perform basic communication control of the Socketlmpl Class.

Using HashMap setDeviceData(HashMap map, long time), which is a function corresponding to void setSendDataFormat(String format) in Fig. 15, the microservice control template class (BaseIms Class) 2010 or the microservice control class for individual devices ( Customs Class) 2110 sets sensor data. In the argument map used at this time, the type of sensor data such as "Do you want to collect video?" or "Do you want to collect still images?" Also, this argument time indicates the time to capture (acquire) the still image.

If it is desired to collect an image, first issue a command Stream createSensorData( ) to cause the corresponding sensor 802 or infrared camera 806 to collect the image. If it is desired to stop the collection of the sensor 802 or the infrared camera 806 in the middle of the process, an API command void removeCreateSensorData( ) is issued to stop (cancel) the sensor data generation. After that, when Stream sendSensor (Stream movie) is issued, the collected video can be used in other programs (methods) of the microservice control template class (BaseIms Class) 2010 or the individual device compatible microservice control class (Customims Class) 2110. It becomes possible.

If a still image is to be collected, a command Stream createSensorData(long time) is first issued to cause the corresponding sensor 802 or infrared camera 806 to collect a still image. After that, when Stream sendSensor (Stream image, long time) is issued, the collection is stopped by another program (method) of the microservice control template class (BaseIms Class) 2010 or the individual device corresponding microservice control class (Customims Class) 2110. Images can be used.

State transitions of the device 8 in this embodiment will be described with reference to FIG. Regarding the state of the device 8, five states are defined: stop state 2400, operation state 2402, running state 2408, preparation state 2410, and error occurrence state 2414. FIG.

Immediately after the device 8 starts controlling 2300 , it is in a stopped state 2400 . Then, when “createDevice( )” (see 14th in FIG. 15), which means activation of device control by the microservice 80 , is executed 2502 , the device 8 shifts to the operating state 2402 . Here, if the response 2506 of “createDevice( )” fails 2510 , an error occurrence state 2414 is entered, and the stop state 2400 returns at the timing of disconnection 2500 of the ifLink application. Also, if the ifLink app disconnection 2500 occurs while the device 8 is in the running state 2402 , the device 8 returns to the stopped state 2400 .

On the other hand, if the “createDevice( )” response 2506 is successful 2512 , device 8 enters ready state 2410 . Then, when an operation instruction 2508 is received from the application (IF-THEN) engine 90, the device starts operating and becomes running 2408. FIG. Here, when a stop instruction 2504 from the application (IF-THEN) engine 90 arrives, the device operation stops and the device 8 returns to the ready state 2410. FIG. Further, when the ifLink application is disconnected, the device 8 enters the stop state 2400 regardless of whether the device 8 is in the ready state 2410 or the running state 2408 .

If the device operation fails 2530 during execution 2408 of the device 8, a retry 2530 to start the device operation is required, so the device 8 enters an error occurrence state 2414. FIG. Similarly, if the device 8 is running 2408 and there is no response from the device 8 for a long period of time, a timeout 2520 occurs and the device 8 enters an error occurrence state 2414 .

A module 9 that implements a predetermined function is separated into a device 8 (or a human-recognizable entity/state/information 7) and a microservice 80 (or control means 4) that controls it. Accordingly, the state of the device 8 and the state of the microservice 80 can be managed separately. On the other hand, the user "changes the contents of the application (IF-THEN) rule 70" (that is, the device 8 operated by the application (IF-THEN) rule 70 is suddenly changed) during execution of the service to the user. There is If the state of the device 8 and the state of the microservice 80 are separately managed in this way, there is an effect that real-time changes can be easily made in response to "change of the application (IF-THEN) rule 70" performed by the user.

As shown in FIG. 18, six states of a static state 2400, connecting 2412, operating state 2402, device control 2404, start 2300, and error occurrence 2414 can be defined as states of the microservice 80 in this embodiment. Then, as shown in FIG. 18, a relationship between the state of the microservice 80 and the state of the device 8 is established.
FIG. 19 shows the relationship between the light source temperature and spectral radiance of radiant light emitted from a blackbody radiation source. It is known that all objects having a given temperature radiate electromagnetic waves (blackbody radiation) corresponding to the temperature. Therefore, by measuring the spectral characteristics of the blackbody radiation emitted therefrom, the temperature of the blackbody radiation source can be determined.

From the spectral characteristic diagram of FIG. 19, it can be seen that the radiance changes with the temperature of the black body radiation source when only the electromagnetic wave with the predetermined wavelength 14 is extracted. In the thermography technique, only electromagnetic waves with a predetermined wavelength 14 are extracted from the radiant light emitted from the object to be measured, and the temperature of the object to be measured is measured from the intensity of the extracted electromagnetic waves.

When the thermography technique is used to measure the scalp surface temperature of a person, the hair in the optical path of the black body radiation light scatters the light. Therefore, when measuring the scalp surface temperature in a place where a lot of hair is distributed in the human head (such as the back of the head), the measurement accuracy is greatly reduced (lower than the actual temperature due to the light scattering loss in the hair). temperature is measured). Therefore, in order to measure the scalp surface temperature of a person using the thermography technique, it is desirable to measure the face where the hair distribution is small.

Also, the intensity of the electromagnetic wave of the predetermined wavelength 14 detected by the infrared camera 806 is inversely proportional to the square of the distance from the object to be measured to the infrared camera 806 . Therefore, when measuring the surface temperature of an object to be measured using thermography technology, the detected temperature decreases as the distance from the object to be measured to the infrared camera 806 increases.

As points to note when measuring the body temperature of a person's head with high accuracy using thermography technology, the above 1) face (in front of the head, where the scalp is exposed) measurement 2) measurement between the measurement object and the infrared camera 806 In addition to consideration of distance, 3) relationship between moisture absorption (effect of sweat and humidity on the face surface) and wavelength used for temperature measurement, and 4) removal of influence of dark current having temperature dependence of infrared camera 806 itself is also considered. There is a need.

From the characteristics shown in FIG. 19, it has already been explained that it is desirable to use light in the wavelength range of 2.5 μm to 20 μm for human body temperature measurement. By the way, in the light in this range, large water absorption peaks are observed near the center wavelengths of 2.955 μm and 6.100 μm. Furthermore, in the wavelength region longer than 6.9 μm, the water absorption tends to increase as the wavelength increases.

A specific example of light absorption by actual water molecules is shown. When light with a wavelength of 2.955 μm passes through a water film with a thickness of 0.3 μm, 80% of the light is attenuated (that is, only 20% of the incident light is transmitted). It is believed that many people have experienced that when they perspire profusely, the sweat turns into water droplets and drips. Considering that the diameter of the water droplet at this time is about 1 to 2 mm, it can be understood how large the amount of light absorption is.

On the other hand, the wavelength dependence of the amount of light absorbed by water molecules takes minimum values near wavelengths of 3.75 μm and 5.30 μm. Therefore, the value of the predetermined wavelength 14 used in thermography is desirably around 3.75 μm or around 5.30 μm. Considering the above light absorption characteristics of water molecules, it is desirable to use light in a wavelength range of 3.3 μm or more and 5.7 μm or less for human body temperature measurement. As the value of the predetermined wavelength 14, if light with a wavelength in this range is used for thermography, body temperature can be measured with relatively little effect of sweat and humidity in the air.

If higher-precision body temperature measurement is required, the detected temperature may be measured with light of two different wavelengths. By combining the detection temperature characteristics of the two-wavelength light and the wavelength dependence characteristics of the optical absorption of water molecules, the amount of water molecules absorbed in the optical path of the infrared light can be estimated. By correcting the detected temperature using this estimated water molecular weight value, an accurate body temperature can be calculated.

It can be seen that the dark current of the infrared camera 806 (pseudo detection current that flows when blackbody radiation is blocked) is unexpectedly large. Therefore, to accurately measure a person's body temperature, it is necessary to subtract the dark current value from the detected current value. Here, the value of this dark current changes with the temperature inside the infrared camera 806 .

For the measurement of the dark current value of the infrared camera 806, it is not preferable to shield the entrance of the infrared camera 806 from light. Blackbody radiation from the room temperature light shielding member enters the infrared camera 806, making it impossible to measure the dark current value. It is desirable to aim the infrared camera 806 at a sufficiently distant object and measure the dark current value frequently between measurements of the body temperature of the person. Since the brightness of blackbody radiation is inversely proportional to the square of the distance between infrared camera 806 and a sufficiently distant object, almost no blackbody radiation from a sufficiently distant object enters infrared camera 806 . In this state, the signal amount (current value) obtained from the infrared camera 806 is approximately equal to the dark current value.

FIG. 20 shows the internal structure of a trinocular infrared camera, which is an application example of this embodiment. Here, "three eyes" means one lens 32 (single eye) for concentrating infrared light 38 on infrared light sensor 34 and one lens 32 (one eye) for concentrating visible light 40 on visible light sensors 46 and 48. It means a total of "three lenses" of the two lenses 42, 44 (two eyes) that allow

In an infrared camera 806 incorporating an infrared sensor 34 that is sensitive to infrared light 38 , the bandpass optical filter 24 is placed immediately before the lens 32 that collects the infrared light 38 . As described above, the band-pass optical filter 24 is designed to pass only light in the wavelength range of 3.3 μm or more and 5.7 μm or less (or light near wavelength 3.75 μm or light near wavelength 5.30 μm). is designed to

A detection signal (first signal) obtained from the infrared light sensor 34 is transferred to the temperature distribution video/image generation unit 36 . The temperature distribution video/image generation unit 36 generates spatial detected temperature distribution characteristics.

Also, the three-lens infrared camera 806 shown in FIG. 20 has a built-in function of correcting a decrease in detected temperature caused by the above-described reason (distance change between the measurement object 20 and the infrared camera 806). That is, the distance L to the measurement object (measurement subject) is measured with two eyes (lenses 42, 44) corresponding to the visible light 40 among the three eyes, and the detected temperature is corrected according to the measured distance L. do.

That is, in the three-lens infrared camera 806, the two lenses 42 and 44 that allow the visible light 40 to pass are arranged at positions separated from each other. Therefore, the imaging positions of the measurement object 20 are shifted between the visible light sensors (imaging devices) 46 and 48 . By detecting this shift amount, the distance L from the measurement object 20 to the three-lens infrared camera 806 can be calculated based on "trigonometry".

Specifically, using detection signals (second signals) obtained from visible light sensors (imaging elements) 46 and 48, visible video/visible image generators 52 and 54 generate an imaging pattern for the measurement target 20. Generate. Then, the distance distribution calculation unit 62 for each object to be measured examines the amount of deviation between the two imaging patterns, and calculates the distance L from the object 20 to the three-lens infrared camera 806 .

As described above, the brightness of the black body radiation emitted from the measurement object 20 is inversely proportional to the square of the distance L. Therefore, in the temperature distribution correcting section 60, the value of the distance L obtained here is used to correct the spatial detected temperature distribution characteristic and calculate the actual body temperature.

Only the "corrected temperature distribution" is obtained from the temperature distribution correction unit 60. FIG. It is difficult to identify the object 20 to be measured from only the temperature distribution characteristics. Therefore, in the synthesizing unit 68, shape images such as the outline of the measurement object 20 and the eyes, nose, mouth, etc. obtained by imaging the visible light are superimposed on the "corrected temperature distribution". Then, this composite image is output to the temperature distribution output terminal 58 .

Here, contours and shape images of the eyes, nose, mouth, etc. to be superimposed on the "corrected temperature distribution" are obtained from the contour shape extractor (including the contours of the eyes) 66 of the visible video/visible image. In particular, the center position of shape images such as contours, eyes, nose and mouth must match the center position of the "corrected temperature distribution". As can be seen from FIG. 20, the position of the lens 32 of the infrared camera 806 and the positions of the lenses 42 and 44 of the normal cameras 816-1 and 816-2 are shifted. Therefore, it is necessary to correct the center position deviation.

After synthesis, the visible light plane video/image generation unit 64 generates a visible image in which the deviation of the center position is corrected. For this, the images obtained by the visible video/visible image generators 52 and 54 and the distance information L obtained from the distance distribution calculator 62 for each object to be measured are used. The visible image obtained here is output from the visible video/visible image output terminal 56 .

The correspondence relationship between FIG. 20 and FIG. 2 described above will be described below. The normal camera 816 in FIG. 2 corresponds to the normal cameras 816-1 and 816-2 in FIG. 20, and FIGS. 2 and 20 have the same infrared camera 806. FIG. 20, from the visible video/visible image generation units 52 and 54 to the distance distribution calculation unit 62 for each measurement object, and the visible light plane video/image generation unit 64 after synthesis, the normal camera image recognition unit 112 in FIG. corresponds to a part of 20 corresponds to the infrared image recognition section 116 in FIG.

Also, the trinocular infrared camera 806 itself shown in FIG. 20 corresponds to the infrared camera 806 in FIG. The human head temperature determination unit 125 in FIG. 2 corresponds to part of the infrared camera microservice 80C in FIG. 2 corresponds to the mailer microservice 80H, the HMD microservice 80E, the individual device microservice 80F, and the like in FIG.

The normal camera image recognition unit 112 shown in FIG. 2 performs advanced processing from "1) image analysis" to "2) person detection", "3) people detection", "4) head detection", and "5) distance detection". I explained how to perform “image recognition”. And the advanced "image recognition" was explained to be executed in the image recognition class (AI Analyze Class) 2018 (Fig. 13) using AI of the sensor microservice 80A.

FIG. 21 shows an example of cooperation between classes in the class structure (software architecture) of the sensor microservice 80A described in FIG. First, Custom Device Class and AI Analyze Class are imported (embedded) 74 and 76 in Customs Class. As a result, the methods defined (described) in the Custom Device Class and AI Analyze Class can be used in the Customims Class.

Then, at step 44 (S44), the transmission data format specification is received by the API command void setSendDataFormat (String format) issued from the application (IF-THEN) engine 90. FIG. If the format of the above argument includes "request for the number of people obtained after recognizing (analyzing) the image signal from the normal camera 816", the image recognition processing using the AI Analyze Class method is executed. be started.

Before starting image recognition processing, it is usually necessary to acquire an image signal (corresponding to a video signal or a still image signal and corresponding to a second signal) from the camera 816 . So call HashMap SetDeviceData() in Customs Class. Then, a corresponding method pre-written in the Custom Device Class operates (S45), and image signal acquisition using the normal camera 816 is started.

In step 46 (S46), an image signal is acquired using the normal camera 816. FIG. Next, at the timing when Stream sendSensor() is called in the Customims Class, the image signal acquired by the normal camera 816 is read into the Customims Class. The image signal read here may be stored in the buffer memory 2008 (FIG. 13) (S48).

A corresponding method previously described in the AI Analyze Class is called in the Customs Class (S49), and the image signal stored in the buffer memory 2008 is read by the image recognition processing side (S50). Similarly, by calling the corresponding method described in advance in the AI Analyze Class in the Customs Class, an instruction to calculate the head position of the person and the resulting number of people calculation (S51) are performed. In response to this, a corresponding method described in advance in AI Analyze Class operates, and data analysis (image recognition) S52 is executed. When the execution of the corresponding method is completed, the number of persons obtained as a result of data analysis (image recognition) is obtained (S53).

In step 54 (S54), according to the calculated number of people, "Is head temperature acquisition necessary? ] is performed. Up to this point, the processing flow of the sensor microservice 80A alone that controls the normal camera 816 has been explained.

The head temperature extraction processing in step 55 (S55) and step 56 (S56) utilizes cooperative processing between the sensor microservice 80A and the infrared camera microservice 80C or the image recognition result of the sensor microservice 80A. Then, the infrared camera microservice 80C performs processing.

FIG. 22 shows an example of an application (IF-THEN) rule setting method for simultaneously notifying multiple media when a person with high fever is detected. In FIG. 22, not only various videos and various still images 128 obtained from the three-lens infrared camera 806 but also position information 138 obtained from the GPS position sensor 830 are used as detection signals.

Here, the setting of the notification destination (medium used for notification) when a person with high fever is detected is defined (described) by application (IF-THEN) rules 70 stored in the memory area 71 of the application (IF-THEN) engine 90. be.

In the example of FIG. 22, both the mail communication/mail transmission 1250 and the warning screen display 1230 are notified. What is important here is not only that "a high fever person has been detected", but also that the user who has received the notification must be able to "identify the detected high fever person". In the example of the warning screen display 1230 in FIG. 6, "the person on the right end is shaded" so that it can be identified. By performing “visualization using an image” at the time of notifying the user in this way, there is an effect that the user who receives the notification can easily identify the hyperthermia (target to be noticed).

For "visualization" at the time of this notification, the high fever identification image 136 may be sent to the screen display microservice 80K that controls the warning screen display 1230. FIG. Also, the high fever person face image 132 may be sent to the mailer microservice 80H that controls the mail communication/mail transmission 1250 . The user who receives the mail here is highly likely to be in a position away from the site. Therefore, the hyperthermia location information 134 obtained from the location information 138 obtained from the GPS location sensor 830 may be transmitted to the mailer microservice 80H.

What is particularly important here is that "information 132-136 to be sent to microservices 80H, 80K on the notification side (that is, associated with THEN 78 in FIG. 11) is sent to the sensor side (that is, the conditions ( IF) Let the microservices 80C and 80J create and send” in relation to 72. Once the information necessary for notification to the user (i.e. execution (THEN) 78 in FIG. 11) is created by microservices 80C, 80J on the sensor side (i.e. associated with condition (IF) 72 in FIG. 11), hyperthermia detection This has the effect of greatly improving the processing efficiency inside the device (or the entire high fever detection system).

An example of the notification information when a high fever person is detected is composed of a "warning message" and a "display image for easy identification of a high fever person" as in the display screen 12 of FIG. In many cases, the content of the "warning statement" can be formatted in a fixed format. Therefore, the standard template (standard format) can be used frequently as a method of "providing information to the user" in the processing performed by the execution unit (THEN) 78 (FIG. 11) of the application (IF-THEN) rule 70. FIG. Therefore, in the microservice 80 related to "providing information to the user", it is possible to execute the process of "creating a standard format using an existing template".

FIG. 23 shows the class configuration (software architecture) of microservices related to information provision. A basic control package 2000 related to information provision includes a standard format generation class (FillFormat Class) 2019 using an existing template. Then, a set of a "fixed form" (for example, the above-mentioned "warning sentence") regarding information to be notified to the user and a "display image" (for example, the above-mentioned "display image for easy identification of a person with high fever") is generated. be.

The created format 77 created in the standard format generation class (FillFormat Class) 2019 (predetermined method) using this existing template is temporarily saved in the created format saving buffer memory. This temporarily saved creation format 77 is displayed on the screen 1230 to the user through the device control template class (Base Device Class) 2002 or the individual device control class (Custom Device Class) 2101 .

FIG. 24 describes a method of communicating information to a user using the information provision (automatic notification) related microservices shown in FIG. Using FIG. 22, "information 132 to 136 to be sent to microservices 80H and 80K on the notification side (that is, related to execution (THEN) 78 in FIG. 11) is transmitted to the sensor side (that is, condition (IF) in FIG. 72) are made and sent by microservices 80C and 80J. A specific processing flow will be described here.

From step 24 (S24) to step 29 (S29) in FIG. 24, the microservice 80C on the sensor side (that is, related to condition (IF) 72 in FIG. 11) is caused to create and receive necessary information. showing. Then, steps 30 (S30) to step 31 (S31) transmit information using the information created by the sensor-side microservice 80C to the notification-side (that is, related to execution (THEN) 78 in FIG. 11) microservice 4D, It is sent to 80H and 80K for notification processing.

In this embodiment, it is necessary to notify the user only after a high fever person is detected. So first for the sensor-side microservice 80C (ie, associated with condition (IF) 72 in FIG. 11), is condition (IF) 72 met? In other words, whether a person with high fever is detected or not? (S27) is made to answer. And if a high fever is detected, the sensor side (i.e. Create/send to microservice 80C (related to condition (IF) 72 in FIG. 11) (S29).

At steps 24 (S24) and 28 (S28), the application (IF-THEN) engine 90 issues an API command of void setSendDataFormat (String format) (FIG. 15) to the microservice 80C on the sensor side. As a result, the microservice 80C on the sensor side issues an instruction to switch the contents of information created/transmitted.

While embodiments of the invention have been described, the embodiments have been presented by way of example and are not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. This embodiment, modifications thereof, and combinations of embodiments are also included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and equivalents thereof.

2... Cloud server, 6... Edge computer, 70... Application (IF-THEN) rule, 80... Micro service, 90... Application (IF-THEN) engine, 92... ifLink app.

Claims (4)

In the high-temperature person detection device, there are a first image recognition unit, a second image recognition unit, a person's body temperature determination unit, a detection operation instruction unit that determines the rule when a high-temperature person is detected, a mobile terminal, and others. a detection operation setting unit for setting whether or not to operate the external device, and a microservice for giving a specific operation instruction to the mobile terminal and the other external device,
the first image recognition unit performing first image recognition from a first signal corresponding to each head of a plurality of persons obtained from a first optical sensor sensitive to infrared light;
The second image recognition unit recognizes a second image from a second signal corresponding to each head of the plurality of persons obtained from a second optical sensor including an image sensor having sensitivity to light of a wavelength other than infrared light. performing image recognition;
The person's body temperature determination unit detects the presence or absence of a hyperthermic person whose body temperature exceeds a predetermined value from the result of the first image recognition and the result of the second image recognition;
The detection-time operation instruction unit that responds to the detection of the presence or absence of a hyperthermic person whose body temperature exceeds the predetermined value outputs an instruction for notification according to a set rule, and outputs an instruction for notification to the detection-time operation setting unit Notifying that the presence or absence of the high fever person is also detected,
The detection operation setting unit sets the microservice for giving specific operation instructions to the mobile terminal and other external devices, and in response to the notification of detection of the presence or absence of the high fever person, the A hyperthermia detection method , comprising: controlling the microservice corresponding to the mobile terminal and the other external device so as to output the specific operation instruction. The first image recognition unit acquires for body temperature determination when the measured distance when actually measuring the temperature of the person is larger or smaller than the measured distance when the predetermined value is set in advance. 2. The method of detecting a high fever person according to claim 1, further comprising correcting the temperature distribution. The high-temperature person detection device includes a first image recognition unit, a second image recognition unit, a person's body temperature determination unit, a detection time operation instruction unit that determines rules for high-temperature person detection, a mobile terminal and other external devices. A detection operation setting unit that sets whether or not to operate the device, and a microservice for giving a specific operation instruction to the mobile terminal and the other external device,
The first image recognition unit performs first image recognition from a first signal obtained from a first optical sensor sensitive to infrared light and corresponding to each head of a plurality of persons,
In the second image recognition unit, a second signal corresponding to each head of the plurality of persons obtained from a second optical sensor including an image sensor having sensitivity to light of wavelengths other than infrared rays is converted into a second image. perform image recognition,
In the person's body temperature determination unit, the presence or absence of a hyperthermia person whose body temperature exceeds a predetermined value is detected from the result of the first image recognition and the result of the second image recognition,
In response to detection of the presence or absence of a hyperthermia person whose body temperature exceeds the predetermined value, the detection operation instruction unit outputs an instruction for notification according to a set rule, and outputs a notification instruction to the detection operation setting unit. Also notify that the presence or absence of the high fever person is detected,
The detection operation setting unit sets the microservice for issuing specific operation instructions to the mobile terminal and the other external devices, and in response to the notification of detection of the presence or absence of the high fever person, the A hyperthermia detection device configured to control the microservice corresponding to the portable terminal and the other external device so as to output the specific operation instruction. Composed of an optical sensor sensitive to infrared light, an edge computer, and a notification means for notifying the presence of a high fever person whose body temperature exceeds a predetermined value,
The edge computer
A first image recognizing unit, a second image recognizing unit, a person's body temperature determination unit, a detection time operation instructing unit that determines rules when a person with high fever is detected, and whether a mobile terminal or other external device is operated a detection time operation setting unit for setting whether or not, and a microservice for giving specific operation instructions to the mobile terminal and other external devices,
The first image recognition unit performs first image recognition from a first signal corresponding to each head of a plurality of persons obtained from a first optical sensor sensitive to infrared light,
The second image recognition unit recognizes a second image from a second signal corresponding to each head of the plurality of persons obtained from a second optical sensor including an image sensor having sensitivity to light of a wavelength other than infrared light. perform image recognition,
The person's body temperature determination unit detects the presence or absence of a hyperthermic person whose body temperature exceeds a predetermined value from the result of the first image recognition and the result of the second image recognition,
The detection-time operation instruction unit that responds to the detection of the presence or absence of a hyperthermic person whose body temperature exceeds the predetermined value outputs an instruction for notification according to a set rule, and outputs an instruction for notification to the detection-time operation setting unit also notifies that the presence or absence of the high fever person is detected,
The detection operation setting unit sets the microservice for giving specific operation instructions to the mobile terminal and other external devices, and in response to the notification of detection of the presence or absence of the high fever person, the A hyperthermia detection system for controlling the microservice corresponding to the mobile terminal and other external devices to output the specific operation instruction .

Images