JP7472238B1 - Information processing device and program - Google Patents

Abstract

[Problem] To provide an information leakage prevention system capable of evaluating the risk of information leakage due to peeking at information displayed on an image output device. [Solution] An information processing device includes a peripheral data acquisition unit that acquires image data of an image having one or more objects arranged around a display area in which an output image output by the image output device is displayed, or peripheral data that is three-dimensional point cloud data or distance data of the one or more objects, an object position determination unit that analyzes the peripheral data acquired by the peripheral data acquisition unit to determine the positional relationship between each of the one or more objects and the display area, and a viewability derivation unit that derives the degree of viewability of the output image by a third party other than the user of the image output device based on the positional relationship determined by the object position determination unit. [Selected Figure] Figure 1

Description

The present invention relates to an information processing device and a program.

Patent Document 1 discloses an information processing device that includes a display unit having a filter layer that can change the viewing angle of the display, and an anti-peek function that reduces visibility from any side other than the front of the display unit by using the filter layer.
[Prior Art Literature]
[Patent Documents]
[Patent Document 1] JP 2021-182069 A

In a first aspect of the present invention, an information processing device is provided. The information processing device includes, for example, a peripheral data acquisition unit that acquires image data of an image having one or more objects arranged around a display area in which an output image output by an image output device is displayed, or peripheral data that is three-dimensional point cloud data or distance data of the one or more objects. The information processing device includes, for example, an object position determination unit that analyzes the peripheral data acquired by the peripheral data acquisition unit and determines a positional relationship between each of the one or more objects and the display area. The information processing device includes, for example, a viewability derivation unit that derives a degree of viewability of the output image by a third party different from the user of the image output device based on the positional relationship determined by the object position determination unit.

In any of the above information processing devices, the viewability derivation unit may derive the degree of viewability such that (i) the degree of viewability is greater for objects located further away from the display area, and/or (ii) the degree of viewability is less for objects located closer to the front of the display area.

In any of the above information processing devices, the object position determination unit may include a point cloud data acquisition unit that acquires three-dimensional point cloud data of each of the one or more objects based on the peripheral data acquired by the peripheral data acquisition unit. In any of the above information processing devices, the object position determination unit may include a distance calculation unit that calculates the distance between a representative point of the display area and each of a plurality of points virtually arranged on the surface of each of the one or more objects represented by the three-dimensional point cloud data acquired by the point cloud data acquisition unit. In any of the above information processing devices, the viewability derivation unit may calculate, for each of the multiple points, a value of a suppression index that is an index indicating the degree to which each point suppresses viewing of the output image by a third party, based on the distance for each of the multiple points calculated by the distance calculation unit. The viewability derivation unit may derive the degree of viewability by summing up the values of the suppression index for each of the multiple points.

In any of the above information processing devices, the object position determination unit may have an angle calculation unit that calculates the absolute value of the angle between the normal direction at the representative point of the display area and the direction from the representative point of the display area toward each of a plurality of points arranged on the surface of each of the one or more objects. In any of the above information processing devices, the viewability derivation unit may calculate a suppression index value for each of the multiple points based on the distance for each of the multiple points calculated by the distance calculation unit and the absolute value of the angle for each of the multiple points calculated by the angle calculation unit. The viewability derivation unit may derive the degree of viewability by summing up the suppression index values for each of the multiple points.

In any of the above information processing devices, the object position determination unit may include a point cloud data acquisition unit that acquires three-dimensional point cloud data of each of the one or more objects based on the peripheral data acquired by the peripheral data acquisition unit. In any of the above information processing devices, the object position determination unit may include an angle calculation unit that calculates the absolute value of the angle between the normal direction at the representative point of the display area and the direction from the representative point of the display area to each of the multiple points arranged on the surface of each of the one or more objects. In any of the above information processing devices, the viewability derivation unit may calculate, for each of the multiple points, a value of a suppression index that is an index indicating the degree to which each point suppresses viewing of the output image by a third party, based on the absolute value of the angle for each of the multiple points calculated by the angle calculation unit. The viewability derivation unit may derive the degree of viewability by summing up the values of the suppression index for each of the multiple points.

In any of the above information processing devices, the image output device may include a housing. The image output device may include a display area arranged in the housing. The image output device may include a peripheral data generating device arranged in the housing and configured to generate peripheral data. In any of the above information processing devices, the peripheral data acquisition unit may acquire peripheral data generated by the peripheral data generating device. Any of the above information processing devices may include a two-dimensional image acquisition unit configured to acquire a two-dimensional image having one or more objects with one or more markers attached as a subject. Any of the above information processing devices may include a marker position determination unit configured to analyze the two-dimensional image acquired by the two-dimensional image acquisition unit and determine a positional relationship between at least a portion of the one or more markers and the display area. Any of the above information processing devices may include a point cloud generation unit configured to generate a point cloud around the position of the marker whose positional relationship with the display area has been determined by the marker position determination unit. In any of the above information processing devices, each of the one or more markers may include identification information for identifying each of the one or more markers, and/or a pattern representing information about the point cloud generated around the position of each marker.

In any of the above information processing devices, each of the one or more objects may be an object to which a set of markers indicating the end of each object in a substantially horizontal direction is attached. Any of the above information processing devices may include a two-dimensional image acquisition unit that acquires a two-dimensional image having one or more objects as a subject. Any of the above information processing devices may include an end detection unit that analyzes the two-dimensional image acquired by the two-dimensional image acquisition unit, detects a plurality of markers included in the two-dimensional image, and detects one end and the other end of each of the one or more objects in a substantially horizontal direction based on a combination of the plurality of markers. Any of the above information processing devices may include an occlusion angle calculation unit that calculates, for each of the one or more objects, an occlusion angle that is an angle between a direction from a representative point of the display area toward one end of each object and a direction from the representative point of the display area toward the other end of each object. In any of the above information processing devices, the viewability derivation unit may derive a degree of viewability based on each of the occlusion angles of the one or more objects calculated by the occlusion angle calculation unit.

Any of the above information processing devices may include an arrangement determination unit that determines the arrangement of the display area based on the degree of viewability derived by the viewability derivation unit. In any of the above information processing devices, the arrangement of the display area may include the position and direction in which the display area is placed. In any of the above information processing devices, the arrangement determination unit may determine the arrangement of the display area such that the degree of viewability is smaller than a predetermined degree.

In any of the above information processing devices, the predetermined degree may be different when the display area is located within an area having a predetermined characteristic and when it is located outside the area.

In any of the above information processing devices, the peripheral data acquisition unit may acquire peripheral data at multiple points in time during a period in which the user continuously uses the image output device. In any of the above information processing devices, the viewability derivation unit may derive the degree of viewability at each of the multiple points in time.

Any of the above information processing devices may include an output control unit that controls the output of an image by the image output device. In any of the above information processing devices, the output control unit may stop output of the output image that was displayed before the degree of viewability exceeded the predetermined degree when the degree of viewability derived by the viewability derivation unit exceeds a predetermined degree. In any of the above information processing devices, the output control unit may display an image different from the output image in the display area when the degree of viewability derived by the viewability derivation unit exceeds a predetermined degree.

In a second aspect of the present invention, an information processing method is provided. Each step of the above information processing method is executed by, for example, a computer. The above information processing method includes, for example, a peripheral data acquisition step of acquiring image data of an image having one or more objects arranged around a display area in which an output image output by an image output device is displayed, or peripheral data that is three-dimensional point cloud data or distance data of one or more objects. The above information processing method includes, for example, an object position determination step of analyzing the peripheral data acquired in the peripheral data acquisition step and determining a positional relationship between each of the one or more objects and the display area. The above information processing method includes, for example, a viewability derivation step of deriving a degree of viewability of the output image by a third party other than the user of the image output device based on the positional relationship determined by the object position determination unit.

In a third aspect of the present invention, a program is provided. The program may be a program for causing a computer to function as the information processing device according to the first embodiment. The program may be a program for causing a computer to execute the information processing method according to the second embodiment. A computer-readable medium for storing the program may be provided. The computer-readable medium may be a non-transitory computer-readable medium. The computer-readable medium may be a computer-readable recording medium.

Note that the above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions.

1 shows an example of a system configuration of an information leakage prevention system 100. The viewing angle of the display area 142 and the peripheral data generating device 152 are shown diagrammatically. 2 shows a schematic diagram of an example of the placement of user terminals 120. 13 illustrates an example of information processing in the control device 170. 13 illustrates an example of information processing in the control device 170. 2 shows an example of the internal configuration of the control device 170. 2 shows an example of an internal configuration of a structure information acquisition unit 612. 6 shows an example of the internal configuration of the virtual space construction unit 630. 13 illustrates an example of information processing in a first calculation unit 830. 13 illustrates an example of information processing in a second calculation unit 840. 2 shows an example of an internal configuration of the safety management unit 640. 13 illustrates an example of information processing in the safety management unit 640. 13 illustrates an example of information processing in the control device 170. 13A and 13B show an example of a generation process of the dynamic image data 1340. 13 shows an example of an internal configuration of the fluctuation index derivation unit 1374. 13 shows an example of an internal configuration of a safety management unit 1378. 13 illustrates an example of information processing in the safety management unit 1378. An example of a two-dimensional image 1800 is shown diagrammatically. An example of a panoramic image 1900 is shown diagrammatically. An example of the system configuration of a computer 3000 is shown in schematic form.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. In the drawings, the same reference numbers may be used to designate the same or similar parts, and duplicate explanations may be omitted.

The information processing terminal displays various information on an image output device such as a display, projector, or screen. If the information displayed on the image output device by the information processing terminal is viewed by a third party other than the user of the information processing terminal, the information may be leaked to the outside. The manner in which the information is viewed by the third party is not limited to visual inspection. For example, the third party may view the information displayed on the image output device by the information processing terminal using an imaging device arranged near the image output device. Examples of the imaging device include a surveillance camera and a hidden camera. For example, the third party may access the imaging device via a network to view the information displayed on the image output device by the information processing terminal.

In recent years, various technologies have been developed to prevent information leakage from image output devices, but the inventors thought that if the image output device could be installed in a location where it is relatively difficult for a third party to view it, the risk of information leakage could be significantly reduced. According to this embodiment, for example, an information leakage prevention system is provided that can evaluate the risk of information leakage caused by a third party peeking at information displayed on the image output device depending on the conditions around the image output device.

According to one embodiment, the information leakage prevention system evaluates the safety of an information processing terminal. More specifically, the information leakage prevention system evaluates the degree of information security (sometimes simply referred to as safety) at the planned installation location or installation location (sometimes referred to as the installation location of the information processing terminal) of an image output device that outputs images generated by the information processing terminal. This allows a user of an information processing terminal to start using the information processing terminal after installing the image output device of the information processing terminal in a safe location. As a result, the risk of information leakage is greatly reduced.

According to another embodiment, the information leakage prevention system monitors the safety of the information processing terminal, for example, while the user is using the information processing terminal. The information leakage prevention system may check the safety of the information processing terminal at all times, or may check the safety of the information processing terminal periodically or at any timing. This can reduce the risk of information leakage, for example, even if the degree of safety of the information processing terminal fluctuates due to changes in the surrounding environment over time.

The safety of the information processing terminal is evaluated, for example, based on the position of the image output device. The safety of the information processing terminal may also be evaluated based on the position of the image output device and the orientation of the image output device at that position (sometimes referred to as the installation direction of the information processing terminal). Note that in this specification, when it is written that an information processing terminal is installed, it may mean that the image output device of the information processing terminal is installed to the extent that no technical contradiction occurs.

Examples of the image output device include image display devices such as displays and screens, and image projection devices such as projectors. The screen outputs an image output by the image projection device, for example, by reflecting the image. In this way, the image output by the image projection device is displayed on the screen.

The image output by the image output device (sometimes referred to as the output image) may be displayed on a plane, a curved surface, or a sphere or a portion thereof. The output image may be a two-dimensional image or a three-dimensional image.

The installation direction of the information processing terminal is represented by, for example, at least one of rotation around the front-to-back axis of the image output device (sometimes referred to as roll), rotation around the left-to-right axis (sometimes referred to as pitch), and rotation around the up-to-down axis (sometimes referred to as yaw). The front-to-back axis of the image output device may be an axis that passes through the image output device from front to back. The up-to-down axis of the image output device may be an axis that passes through the image output device from top to bottom. The left-to-right axis of the image output device may be an axis that passes through the image output device from left to right. The above three axes may be mutually orthogonal. When the image output device is a curved surface, a space curve extended along the curved surface may be used as an axis.

(Safety assessment based on object movement or light fluctuation)
In one embodiment, it is possible to evaluate the safety of an information processing terminal based on the movement of objects or the fluctuation of light around the installation position of the information processing terminal. In this embodiment, the safety of the information processing terminal is evaluated based on the analysis result of a video obtained by capturing an image of the periphery of the installation position of the information processing terminal.

When an object located near the installation position of the information processing terminal moves, the pixel value of the pixel corresponding to the object fluctuates between the multiple frames that make up the video, even if the imaging device that captured the video is stationary. Also, even if the object located near the installation position of the information processing terminal and the imaging device that captured the video are stationary, the pixel value of a specific pixel fluctuates between the multiple frames that make up the video due to fluctuations in light reflected from the surface of the object, light that penetrates the object, or light emitted from the object.

For example, even if the average value of the light intensity is constant macroscopically, if there is a small deviation between the light intensity and the average value of the light intensity microscopically, light fluctuation is detected. Light fluctuation indicates, for example, that the intensity and/or color of the light fluctuates partially randomly while following a specific average. Light fluctuation observed around the installation position of the information processing terminal represents, for example, (i) the degree of movement of various other objects that exist or have existed between a light source and a subject that reflects or transmits light from the light source, and/or (ii) the degree of movement of other objects arranged between the subject and the imaging device. The other objects may be solid, gas, or liquid. Light fluctuation may (i) include changes with a high degree of periodicity to a certain extent, or (ii) include movements close to random.

An example of a method for evaluating the safety of an information processing terminal is to evaluate the degree to which object movement or light fluctuation is detected in the vicinity of the installation location of the information processing terminal. Examples of the degree include the magnitude of the detected movement or fluctuation, and the frequency with which the movement or fluctuation is detected.

When object movement is detected around the installation location of the information processing terminal, there is a high possibility that a third party is present around the installation location of the information processing terminal. Similarly, when relatively large light fluctuations are detected around the installation location of the information processing terminal, there is a high possibility that a third party is present around the installation location of the information processing terminal. Furthermore, when the pattern of light fluctuations around the installation location of the information processing terminal changes, there is a high possibility that a third party is present around the installation location of the information processing terminal. Therefore, the smaller the degree to which object movement or light fluctuations are detected, the greater the degree of safety of the information processing terminal can be evaluated (i.e., the smaller the possibility that the output image will be viewed by a third party).

When the light fluctuation pattern observed around the installation location of the information processing terminal resembles the natural light fluctuation pattern, there is a high possibility that a window or opening leading to the outdoors is present near the installation location, or that the installation location is outdoors. On the other hand, when the light fluctuation pattern observed around the installation location of the information processing terminal resembles the artificial light fluctuation pattern, there is a low possibility that a window or opening leading to the outdoors is present near the installation location, or that the installation location is outdoors. Therefore, the lower the degree to which light fluctuations similar to natural light are detected, the higher the degree of safety of the information processing terminal can be evaluated (i.e., the lower the possibility that the output image will be viewed by a third party).

In addition, if the pattern of light fluctuation observed around the installation position of the information processing terminal is similar to the pattern of light fluctuation when the information processing terminal is mounted on a predetermined type of moving object, there is a high possibility that the information processing terminal is mounted on the moving object. Examples of the predetermined type of moving object include vehicles, aircraft, and ships. The predetermined type of moving object may be public transportation such as trains, buses, passenger planes, and passenger ships. Therefore, if the detected light fluctuation has a predetermined first pattern, it may be evaluated that the degree of safety of the information processing terminal is low (i.e., there is a high possibility that the output image will be viewed by a third party). Similarly, if the detected light fluctuation has a predetermined second pattern, it may be evaluated that the degree of safety of the information processing terminal is high (i.e., there is a low possibility that the output image will be viewed by a third party). The pattern of light fluctuation may be an example of the degree to which light fluctuation is detected.

For example, consider a case where natural light entering a room is reflected by the surface of an object placed in the room, and the reflected light is received by a camera placed at the installation position of the information processing terminal. In this case, the fluctuation of the light received by the camera is affected by (i) the swaying of tree branches and leaves, (ii) the swaying of the water surface, (iii) the flow of dust, water droplets, water vapor, etc. suspended in the air, and (iv) fluctuations in the refractive index of the air, between the sun, which is the light source, and the object, and between the object and the camera. Therefore, when the fluctuation of light is subjected to a frequency analysis, the fluctuation includes a relatively wide range of frequencies. For example, the frequency distribution of the fluctuation of light has a relatively broad mountain-like or normal distribution shape.

Similarly, consider a case where artificial light emitted from a light source placed in a room is reflected off the surface of an object placed in the room, and the reflected light is received by a camera placed at the installation position of the information processing terminal. In this case, the distance between the light source and the above-mentioned object is shorter than when the light source is the sun. As a result, when the light fluctuation is subjected to a frequency analysis, the fluctuation contains a relatively narrow range of frequencies, or contains many specific frequency components. For example, the frequency distribution of the light fluctuation has a sharp peak shape or a normal distribution shape, or has a shape in which specific frequency components stand out, compared to when the light source is the sun.

The inventors have found that (i) natural light contains more light fluctuations in the frequency band of about several Hz to several tens of Hz than light from artificial light sources, (ii) the light fluctuations can be detected by analyzing video images captured by an imaging device at about 30 frames per second, and (iii) it is possible to estimate an event occurring at the location where the light is observed based on the pattern of the light fluctuations (e.g., a pattern of the frequency distribution of the fluctuations). The inventors have also come up with the idea of evaluating the safety of the location where the light is observed based on the observed light fluctuations.

(Evaluation of safety based on the presence or absence of shielding or the amount of shielding)
In another embodiment, it is possible to evaluate the safety of an information processing terminal based on the arrangement of structures (sometimes called shields) arranged around the installation position of the information processing terminal. The shields may be solid, liquid, gas, or a combination of these. Another method for evaluating the above safety is, for example, evaluating the arrangement of the shields.

For example, the greater the degree to which a third party's view of a display, screen, etc. is blocked by an obstruction, the greater the degree of safety of the information processing terminal is evaluated. The smaller the distance between the installation position of the information processing terminal and the obstruction, the smaller the possibility of a third party getting between the installation position and the obstruction. Thus, the smaller the distance between the installation position of the information processing terminal and the obstruction, the greater the degree of safety of the information processing terminal may be evaluated. Similarly, the greater the above distance, the less the degree of safety of the information processing terminal may be evaluated.

(Evaluation of safety based on the surrounding environment)
In yet another embodiment, it is possible to evaluate the safety of an information processing terminal based on the status of the environment (sometimes referred to as the surrounding environment) around the installation position of the information processing terminal. The environment around the installation position may include the environment of the installation position. Examples of the status of the surrounding environment include the temperature distribution status, brightness status, weather status, air status, and the flow of people or population status in the surrounding environment. The status of the surrounding environment may be advanced. An example of a method for evaluating the safety of an information processing terminal is to evaluate the status of the surrounding environment. The surrounding environment may be an example of the environment around the display area.

(Temperature distribution status)
The temperature of the surrounding environment can be measured using, for example, an infrared camera, a thermo camera, a thermal imaging camera, etc. The condition of the surrounding environment can be evaluated by comparing the temperature at the installation position of the information processing terminal with the temperature around the installation position of the information processing terminal. The infrared camera may be a mid-infrared camera or a far-infrared camera.

For example, people, tools used by people (such as electronic devices), and places exposed to natural light tend to become hotter than the surrounding environment. This tendency is particularly noticeable indoors. Therefore, within a particular space, areas that are hotter than other areas within the space can be evaluated as having a low level of security for information processing terminals (i.e., there is a high possibility that output images will be viewed by a third party).

For example, if a small camera is secretly installed in a room, the temperature at the location where the small camera is installed may be higher than the temperature in the vicinity of the installation location. In this case, the safety of information processing terminals in the vicinity of the installation location of the small camera may be reduced. As a result, the risk of information leakage due to the small camera may be reduced.

(Brightness situation)
Examples of indicators for evaluating brightness include (i) the current time, the time zone to which the current time belongs, and/or weather, and (ii) illuminance. The current time may or may not include information indicating a date. The outdoor brightness at a specific time may vary depending on the season. When the current time includes information indicating a date (sometimes referred to as the current date and time), the brightness may be evaluated more accurately.

For example, (i) the greater the brightness of the surrounding environment, or (ii) the greater the ratio of the area of the area set around the installation location where the illuminance is greater than a predetermined standard, the easier it is for the user of the information processing terminal to notice the presence of a third party in the vicinity, while also making it easier for the third party to view the information output by the information processing terminal. Thus, the greater the brightness or the greater the ratio, the greater the degree of safety of the information processing terminal can be evaluated. Also, the greater the brightness or the greater the ratio, the less safe the information processing terminal can be evaluated.

(Weather conditions)
Examples of meteorological conditions include weather, temperature, humidity, wind speed, and air pressure. Examples of meteorological conditions include type of weather, amount of solar radiation, amount of rainfall, amount of snowfall, and thickness of fog. For example, the greater the amount of rainfall, amount of snowfall, thickness of fog, and the like, the smaller the visibility. The smaller the visibility, the higher the degree of safety of the information processing terminal (i.e., the lower the possibility that an output image will be viewed by a third party).

Furthermore, for example, if the temperature is lower than a predetermined first threshold (e.g., if it is very cold) or if the temperature is lower than a predetermined second threshold (e.g., if it is very hot), the possibility of a third party being present in the vicinity of the installation location is reduced. Therefore, if the temperature is lower than the predetermined first threshold or the temperature is lower than the predetermined second threshold, the degree of security of the information processing terminal can be evaluated as being high (i.e., the possibility that the output image will be viewed by a third party is low).

For example, when the wind speed is high, the possibility of spying using a drone is reduced. Also, when the air pressure is low, the altitude may be high. If the altitude is high, the possibility of spying from the surroundings is reduced. Therefore, the degree of safety of an information processing terminal can be evaluated based on the wind speed, air pressure, altitude, etc.

(Air conditions)
Examples of air conditions include (i) the degree of air transparency, visibility, or the concentration of suspended or scattered substances, and (ii) the degree of odor or the concentration of odorous substances. For example, the lower the degree of air transparency or the higher the concentration of suspended or scattered substances, the lower the visibility. The lower the visibility, the higher the degree of safety of the information processing terminal (i.e., the lower the possibility that the output image will be viewed by a third party). The suspended or scattered substances may be solid, liquid, or gas. Examples of suspended or scattered substances include water droplets (e.g., mist or rain), chemical substances, allergens, sand dust, PM2.5, etc. An example of an allergen is pollen.

For example, the stronger the odor intensity in the surrounding environment, the less likely it is that a third party is present near the installation location. Therefore, the greater the degree of odor or the concentration of the odorous substance, the greater the degree of safety of the information processing terminal (i.e., the less likely it is that the output image will be viewed by a third party). The odorous substance may be a solid, liquid, or gas.

(Personnel flow situation)
Examples of the situation of people flow or population include the amount or flow of pedestrians and/or vehicles. The situation of people flow or population may be the amount or flow of pedestrians. The fewer the pedestrians and/or vehicles, the higher the degree of safety of the information processing terminal (i.e., the lower the possibility that the output image will be viewed by a third party) can be evaluated.

(Safety assessment based on object detection results)
In still another embodiment, it is possible to evaluate the safety of an information processing terminal based on the detection result of an object or event at the installation position of the information processing terminal and its surroundings. The safety of the information processing terminal may be evaluated based on the type and/or attribute of the detected object or event. The method of detecting the object or event is not particularly limited, but examples include image analysis and audio analysis. This method differs from the methods described in relation to the other embodiments described above in that the safety of the information processing terminal is evaluated based on the characteristics of an object disposed at the installation position or its surroundings, rather than the characteristics of the installation position or its surroundings.

(An example of image analysis)
For example, by analyzing an image obtained by capturing an image of the installation position or its surroundings (sometimes referred to as the installation position, etc.) of the information processing terminal, it is possible to detect an object arranged at the installation position, etc. or an event occurring at the installation position, etc., and to determine the type and/or attribute of the detected object or event. When a color value and/or a feature unique to a specific object is detected in the above image, the specific object can be detected and determined. Examples of specific objects include a human face, a TV, a picture frame, an AR marker, a window, a door, an imaging device, a drone, etc. An object may be a solid, a liquid, a gas, or a combination thereof. In this specification, even if it does not have a fixed shape, such as the sky, darkness, the sea, waves, or a flame, it can be treated as an object as long as it has a color or a refractive index.

If the above image is an image captured by a far-infrared camera, for example, a human being present behind a cloth, a wall, etc. may be detected. Therefore, for example, if a predetermined first type of object such as a human being, an imaging device, or a drone is detected, the degree of security of the information processing terminal may be evaluated as low (i.e., there is a high possibility that the output image may be viewed by a third party).

On the other hand, even if a human face is detected, if the detected human face is the user's face or a pre-registered human face, the risk of information leakage is relatively small. A human face may be an example of a human feature. Therefore, a process may be executed to compare the features of the detected object with the features of objects whose safety has been confirmed in advance. For example, the features of objects whose safety has been confirmed are recorded in a whitelist created in advance. If the features of the detected object do not match the features of objects whose safety has been confirmed in advance, the information processing terminal may be evaluated as having a low level of safety (i.e., there is a high possibility that the output image will be viewed by a third party).

Even if a human, imaging device, drone, etc. is detected, if the detection position is inside the TV screen or inside the frame, the risk of information leakage is relatively small. Therefore, when a predetermined second type of object is detected in an image, processing may be performed to reduce the impact on the safety evaluation of the information processing terminal of other objects located in the area in which the object in the image is detected. For example, processing is performed to adjust weights and/or thresholds related to the safety evaluation inside and outside the above-mentioned area.

Furthermore, when an encoded code such as an AR marker, a QR code (registered trademark), or a barcode is detected in an image, a process for evaluating the safety of the information processing terminal may be executed based on the information indicated by the encoded code. Examples of information indicated by the encoded code include that the encoded code may be evaluated as safe when detected, that the encoded code may be evaluated as unsafe when detected, that there is a low risk of information leakage at the position where the encoded code is placed, and that there is a high risk of information leakage at the position where the encoded code is placed.

(An example of voice analysis)
For example, by analyzing a sound recorded at the installation position, etc. of the information processing terminal, it is possible to detect an object placed at the installation position, etc. or an event that occurred at the installation position, etc., and to determine the type and/or attribute of the detected object or event. For example, if a dog's howl, human footsteps, etc. are detected and determined from the above-mentioned sound, there is a high possibility that a human being is present near the installation position, etc.

Therefore, when a particular type of sound is detected, the information processing terminal may be evaluated as having a low level of safety (i.e., there is a high possibility that the output image may be viewed by a third party). On the other hand, even when human footsteps are detected, the risk of information leakage differs greatly between when it is determined that the human is approaching the installation location or the like and when it is determined that the human is moving away from the installation location or the like. Therefore, processing may be performed to evaluate the safety of the information processing terminal based on the position of the sound source and/or a change in the position. For example, processing is performed to adjust weights and/or thresholds related to the evaluation of safety based on the position of the sound source and/or a change in the position.

(Other evaluation methods)
The method of evaluating the safety of the information processing terminal is not limited to the above-mentioned method. The safety of the information processing terminal may be evaluated by a combination of two or more of the above-mentioned methods. The information leakage prevention system 100 may determine a final evaluation of the safety of the information processing terminal by comprehensively judging the evaluations derived by each of the combinations of two or more of the above-mentioned methods. For example, the information leakage prevention system 100 determines a final evaluation of the safety of the information processing terminal based on the weights for each of the multiple methods and the evaluations derived by each method. The information leakage prevention system 100 may determine a final evaluation of the safety of the information processing terminal by combining one or more conditional branches and multiple methods. This allows the safety of the information processing terminal to be evaluated accurately. For example, even if a third party presents a photo of the user in front of camera 154 and attempts to impersonate the user, the information leakage prevention system 100 can prevent information leakage due to the above-mentioned impersonation act by taking into consideration the results of analysis such as infrared image analysis, analysis of object movement or light fluctuation, and making a comprehensive judgment on the safety of the information processing terminal.

(Information Leakage Prevention System 100)
According to this embodiment, the details of the information leakage prevention system 100 are described with reference to Figures 1 to 12. In addition, the details of a method for evaluating the degree to which object movement or light fluctuation is detected are described with reference to Figures 13 to 20. A person skilled in the art who has come into contact with the description of this specification can understand that technical matters described in one embodiment can also be applied to other embodiments to the extent that there is no technical contradiction.

(Overview of information leakage prevention system 100)
FIG. 1 is a schematic diagram showing an example of a system configuration of an information leakage prevention system 100. In this embodiment, the information leakage prevention system 100 evaluates the safety of the user terminal 120. More specifically, the information leakage prevention system 100 evaluates the safety of a third party viewing an output image of the user terminal 120. The safety of the user terminal 120 may represent the safety of the user terminal 120 installed at a specific position and direction, may represent the safety of the installation position of the user terminal 120, or may represent the safety of the installation position and installation direction of the user terminal 120. The installation position of the user terminal 120 may be an example of the installation position of the display area 142. The installation direction of the user terminal 120 may be an example of the installation direction of the display area 142.

For example, the information leakage prevention system 100 evaluates the safety of the user terminal 120 based on at least one of the following: (i) an index indicating the degree to which object movement or light fluctuation is detected around the installation location of the user terminal 120; (ii) an index indicating the arrangement of obstructions existing around the installation location of the user terminal 120; and (iii) an index indicating the surrounding environment of the installation location of the user terminal 120. Instead of or in addition to the at least one index described above, the information leakage prevention system 100 may evaluate the safety of the user terminal 120 based on the characteristics or type of objects detected around the installation location of the user terminal 120. Examples of the characteristics of the objects include color values unique to the objects. For example, a black object and a night sky have different color values. This allows a painting of the night sky to be distinguished from the night sky seen through a window.

In this embodiment, the information leakage prevention system 100 assists in the installation of the user terminal 120. In this embodiment, the information leakage prevention system 100 monitors the security of the user terminal 120 while the user terminal 120 is in use.

In this embodiment, the details of the information leakage prevention system 100 are described using an example in which the user terminal 120 executes the various information processes described above. However, the information leakage prevention system 100 is not limited to this embodiment. In other embodiments, the user terminal 120 may be configured to be able to communicate with other computers and may cooperate with the other computers to execute the various information processes described above.

(Overview of each part of the information leakage prevention system 100)
In this embodiment, the information leakage prevention system 100 includes a user terminal 120. In this embodiment, the user terminal 120 includes, for example, a housing 130, a display device 140, an environmental sensor 150, an input device 160, and a control device 170. In this embodiment, the display device 140 includes, for example, a display area 142. In this embodiment, the environmental sensor 150 includes a peripheral data generating device 152, a camera 154, and an illuminance sensor 156. In this embodiment, the installation position of the display area 142 may be referred to as the installation position of the user terminal 120. In addition, the installation direction of the display area 142 may be referred to as the installation direction of the user terminal 120.

In this embodiment, the user terminal 120 outputs information to the user. The user terminal 120 may output information by image or by sound. The user terminal 120 may accept information input from the user. The user terminal 120 executes various types of information processing.

In this embodiment, the user terminal 120 outputs an image. The image output by the user terminal 120 is displayed in the display area 142. A person (sometimes referred to as a viewer) present within the viewing angle range of the display area 142 can view the image displayed in the display area 142. If an imaging device is disposed within the viewing angle range of the display area 142, the imaging device can capture the image displayed in the display area 142. A person (sometimes referred to as a viewer) who has acquired image data captured by the imaging device can view the image displayed in the display area 142.

In one embodiment, the above image is output in response to a user's operation of the user terminal 120. For example, the user inputs instructions to the user terminal 120 via the input device 160 for the BIOS, operating system (sometimes referred to as OS), or application program (sometimes referred to collectively as program) running on the user terminal 120. The user terminal 120 executes information processing in response to the instructions, and displays an image showing the results of the information processing in the display area 142.

In another embodiment, the above image is output when the degree of security of the user terminal 120 does not meet a predetermined standard. In such a case, for example, a warning message, a warning screen, a lock screen, a screen in which information in some areas is concealed, or the like is displayed in the display area 142.

Examples of user terminals 120 include personal computers and mobile terminals. Examples of mobile terminals include mobile phones, smartphones, PDAs, tablets, notebook or laptop computers, and wearable computers.

In this embodiment, the housing 130 supports the display device 140, the peripheral data generating device 152, the input device 160, and the control device 170. Each of the display device 140, the peripheral data generating device 152, the input device 160, and the control device 170 may be at least partially housed inside the housing 130, or may be attached to the outside of the housing 130.

In this embodiment, the display device 140 outputs an image. The display device 140 may output an image or stop outputting an image according to an instruction from the control device 170. The display device 140 may adjust the brightness or contrast of the image according to an instruction from the control device 170. Examples of the display device 140 include a display and a projector.

In this embodiment, the image output by the display device 140 (sometimes referred to as an output image) is displayed in the display area 142. In this embodiment, the display area 142 is disposed in the housing 130. This causes the direction of the optical axis of the peripheral data generating device 152 and the direction of the normal vector of the display area 142 to approximately coincide.

In this embodiment, the environmental sensor 150 acquires various information related to the surrounding environment of the display area 142. The environmental sensor 150 may include one or more sensors for acquiring the above-mentioned various information. Examples of the above-mentioned sensors include a sensor for acquiring various images, a sensor for acquiring sound, a sensor for detecting brightness, a sensor for detecting temperature, a sensor for detecting distance or depth, and the like. The environmental sensor 150 may output the information acquired by the above-mentioned one or more sensors to the control device 170.

In this embodiment, the peripheral data generating device 152 generates information (sometimes referred to as peripheral data) about one or more objects (sometimes referred to as obstructions) arranged around the display area 142. The peripheral data generating device 152 outputs the generated peripheral data to the control device 170. Examples of the periphery of the display area 142 include positions within the range of the viewing angle of the display area 142, positions whose distance from the display area 142 is within a predetermined numerical range, and the like.

For example, the surrounding data generating device 152 generates information indicating the relative position of one or more obstructions arranged around the display area 142 and the display area 142. Examples of information indicating the relative position of each obstruction and the display area 142 include (i) the distance between the representative point of the display area 142 and the representative point of each obstruction, (ii) the direction from the representative point of the display area 142 to the representative point of each obstruction, and (iii) the magnitude and direction of one or more vectors pointing from the representative point of the display area 142 to the representative point of each obstruction. The surrounding data generating device 152 may generate information indicating at least one of the position, shape, and size of the obstruction.

The shielding object may be an object that satisfies a dimensional condition. An example of the dimensional condition is that at least one of the width, height, and depth is greater than a predetermined value. The shielding object may be an object that is large enough to block the viewer's line of sight when the object is placed between the viewer and the display area 142. The shielding object is preferably an object that suppresses the transmission of light. However, the degree to which the shielding object suppresses the transmission of light is not particularly limited. The shielding object may include a transparent object such as glass, and/or an object with through holes formed therein such as a lattice. The shielding object may be a stationary object, or may be an object that is configured to be movable within a predetermined range. Examples of the shielding object include furniture, ornaments, electrical appliances, floors, ceilings, walls, pillars, beams, doors, and windows.

In one embodiment, the peripheral data generating device 152 generates image data (sometimes referred to as image data) of one or more objects arranged around the display area 142 as the peripheral data. Examples of images of obstructing objects as subjects include two-dimensional images and stereoscopic images. In another embodiment, the peripheral data generating device 152 generates three-dimensional point cloud data or distance data (sometimes referred to as three-dimensional data) of one or more obstructing objects arranged around the display area 142 as the peripheral data.

The measurement accuracy of the three-dimensional data is not particularly limited. Furthermore, the three-dimensional data may be missing information about a portion of the obstruction, or the information about the portion of the obstruction may be inferred from other information. The distance data may include information indicating a relative distance, information indicating a distance ranking index, etc.

When the surrounding data generating device 152 generates three-dimensional data, a three-dimensional virtual space around the display area 142 can be constructed. This simplifies the calculation procedure for the degree to which a third party's line of sight is blocked by an obstruction. As a result, the amount of calculation required to derive an index indicating the positioning status of the obstruction described above can be reduced.

The peripheral data generating device 152 does not have to generate three-dimensional data. Even in this case, the degree to which the line of sight of a third party is blocked by an obstruction can be derived based on image data generated by the peripheral data generating device 152. In particular, when the position coordinates of the peripheral data generating device 152 and the display area 142 are approximately the same, and the direction of the optical axis of the peripheral data generating device 152 and the direction of the normal vector of the display area 142 are approximately parallel, the degree to which the line of sight of a third party is blocked by the obstruction can be derived with high accuracy.

Examples of the peripheral data generating device 152 include (i) a sensor that captures an image, and (ii) a sensor that measures the distance to an object using sound waves, ultrasonic waves, or electromagnetic waves. More specifically, examples of the peripheral data generating device 152 include an image sensor, a camera, a stereo camera, LiDAR, and a monocular camera depth distance measuring device (sometimes referred to as a monocular depth estimation device). Optical components such as a fisheye lens or an ultra-wide-angle lens may be attached to the above-mentioned camera. This increases the angle of view of the camera.

The viewing angle (sometimes referred to as the viewing angle) of the peripheral data generating device 152 may be larger than the viewing angle of the display area 142. In this case, the information leakage prevention system 100 can evaluate the safety of the display area 142 for multiple cases in which the direction in which the display area 142 is installed (sometimes referred to as the installation direction) is different, for example, based on a single image data or a single three-dimensional data generated by the peripheral data generating device 152. This allows the safety of the display area 142 to be evaluated with high accuracy. Examples of the safety of the display area 142 include the safety of the display area 142 installed at a specific position and direction, the safety of the installation position of the display area 142, and the safety of the installation position and installation direction of the display area 142.

The installation direction of the display area 142 may be an example of the orientation of the information processing terminal described above. The installation direction of the display area 142 may be represented by the direction of a normal vector of the display area 142, or may be represented by rotation around at least one of the front-back axis, the up-down axis, and the left-right axis of the display area 142.

When the display area 142 is housed in the housing 130, the installation position of the display area 142 approximately coincides with the installation position of the user terminal 120. In this case, the installation direction of the display area 142 approximately coincides with the installation direction of the user terminal 120.

In this embodiment, camera 154 generates video data (sometimes referred to as video data) including multiple still images (sometimes referred to as frames). For example, camera 154 captures an image of the periphery of the planned installation location or installation location (as described above, sometimes simply referred to as the installation position) of display area 142, and generates video data. Camera 154 outputs the generated video data to control device 170. Camera 154 may be an image sensor for capturing two-dimensional images, or an infrared camera for capturing infrared images (sometimes referred to as thermal images).

In this embodiment, the camera 154 is disposed in the housing 130. In this embodiment, the display area 142 is also disposed in the housing 130, so the position where the camera 154 captures the video (sometimes referred to as the capture position) and the installation position of the display area 142 or the user terminal 120 are approximately the same. In addition, the direction of the optical axis of the camera 154 and the direction of the normal vector of the display area 142 are approximately the same. Therefore, the direction where the camera 154 captures the video (sometimes referred to as the capture direction) and the installation direction of the display area 142 or the user terminal 120 are approximately the same.

Note that if the peripheral data generating device 152 includes an imaging element, the user terminal 120 does not need to include the camera 154. In this case, the imaging element may have the same function as the camera 154 according to this embodiment.

In this embodiment, the illuminance sensor 156 measures the illuminance around the installation position of the display area 142. The illuminance sensor 156 outputs information indicating the measurement result to the control device 170.

In this embodiment, the input device 160 accepts input from a user of the user terminal 120. The input device 160 outputs information input by the user to the control device 170. Examples of the input device 160 include a keyboard, a pointing device, a touch panel, and a microphone.

In this embodiment, the control device 170 controls the user terminal 120. The control device 170 executes various information processing in the user terminal 120. For example, the control device 170 evaluates the safety of the installation position and/or installation direction of the user terminal 120 based on the data acquired by the peripheral data generating device 152. The control device 170 may input the data acquired by the peripheral data generating device 152 to multiple application programs. For example, the control device 170 includes a virtual camera driver for inputting the data acquired by the peripheral data generating device 152 to multiple application programs. Details of the control device 170 will be described later.

The information processing method executed by the control device 170 includes, for example, peripheral state acquisition, which acquires peripheral state information indicating the peripheral state of the display area in which the output image output by the image output device is displayed. The above information processing method includes, for example, a viewability derivation step, which derives the degree of viewability of the output image by a third party other than the user of the image output device, based on the peripheral state of the display area indicated by the peripheral state information acquired in the peripheral state acquisition step.

The information processing method executed by the control device 170 may be an image processing method. The above image processing method, for example, has a video acquisition step of acquiring data of a video including a plurality of frames. The above image processing method, for example, has a variation index derivation step of (a) dividing each frame of the video into a plurality of predetermined regions, and (b) deriving, for each of the plurality of regions, a value of a variation index that is an index showing the variation of color and/or brightness of each region among a plurality of frames, based on the pixel values of the pixels included in each region. In the above image processing method, each of the plurality of regions includes, for example, one or more pixels.

The information leakage prevention system 100 may be an example of an information processing device. The user terminal 120 may be an example of an information processing device or an image output device. The display device 140 may be an example of an image output device. The camera 154 may be an example of an imaging device. The imaging device may be an example of a surrounding state acquisition unit. The imaging device may be an example of a device that generates surrounding state information and outputs the surrounding state information to the surrounding state acquisition unit. The illuminance sensor 156 may be an example of an environmental information acquisition unit. The illuminance sensor 156 may be an example of a device that generates environmental information and outputs the environmental information to the environmental information acquisition unit. The control device 170 may be an example of an information processing device, an image output device, an imaging device, an environmental information acquisition unit, or an output control unit.

The image output by the display device 140 may be an example of an output image. The degree of safety of the user terminal 120 may be an example of the degree of viewability of the output image. The degree of safety of the display area 142 may be an example of the degree of viewability of the output image. Information about obstructions generated by the surrounding data generating device 152 may be an example of surrounding data. Image data may be an example of surrounding data. Three-dimensional data may be an example of surrounding data.

(An example of another embodiment)
In the present embodiment, an example of the information leakage prevention system 100 has been described by taking as an example a case where the display area 142 is a part of the display device 140. However, the information leakage prevention system 100 is not limited to this embodiment. In other embodiments, the display area 142 is disposed at a position physically separated from the display device 140. For example, when the display device 140 is a projector, a screen is used as the display area 142.

2 shows a schematic diagram of the viewing angles of the display area 142 and the peripheral data generating device 152. In the embodiment described in relation to FIG. 2, the relationship between the viewing angles of the display area 142 and the peripheral data generating device 152 is described using an example in which the representative point 240 of the display area 142 and the representative point 250 of the peripheral data generating device 152 are approximately coincident. As described in relation to FIG. 1, in this embodiment, the display area 142 is part of the display device 140, and the display device 140 and the peripheral data generating device 152 are housed in the same housing 130. Therefore, as shown in FIG. 2, the direction of the normal vector 242 of the display area 142 and the direction of the optical axis 252 of the peripheral data generating device 152 are approximately coincident.

In this embodiment, the viewing angle (sometimes referred to as viewing angle θ _D or angle of view) of the display region 142 is derived as the sum of the absolute value of the angle θ _DR between the normal vector 242 and the vector 244 and the absolute value of the angle θ _DL between the normal vector 242 and the vector 246. Similarly, in this embodiment, the viewing angle (sometimes referred to as viewing angle θ _C or angle of view) of the peripheral data generating device 152 is derived as the sum of the absolute value of the angle θ _CR between the optical axis 252 and the vector 254 and the absolute value of the angle θ _CL between the optical axis 252 and the vector 256.

(An example of another embodiment)
In this embodiment, an example of the information leakage suppression system 100 has been described using a case where the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generating device 152 as an example. For example, if a film for preventing a third party from peeking is attached to the surface of the display area 142, the viewing angle of the display area 142 may be smaller than the viewing angle of the peripheral data generating device 152. However, the information leakage suppression system 100 is not limited to this embodiment. In other embodiments, the viewing angle of the display area 142 may be larger than the viewing angle of the peripheral data generating device 152.

Figure 3 shows a schematic diagram of an example of the arrangement of the user terminal 120. In this embodiment, an example of the arrangement of the user terminal 120 is described using as an example a case where the user terminal 120 is arranged at position A, position B, or position C inside the work room 300.

In this embodiment, the work room 300 includes a wall 312, a wall 314, a wall 316, a wall 318, and a floor 320. In this embodiment, a shield 330 is arranged inside the work room 300. Markers 342, 344, and 346 are arranged on the surface of the wall 316. A glass window 322 is arranged on the wall 318.

(Safety at Position A)
3, when a user 22 of the user terminal 120 places the user terminal 120 at position A so that a normal vector 242 of the display area 142 faces downward in the figure, the space in which the image displayed in the display area 142 can be viewed is the space surrounded by vectors 244 and 246, a wall 314, and a floor 320. In this case, a third party 32 and a third party 34 different from the user 22 cannot view the image displayed in the display area 142.

Therefore, even if third parties 32 and 34 do not have the proper authority to view the information contained in the output image of user terminal 120, the possibility of the information being leaked is small, and the safety of user terminal 120 at position A can be evaluated as high. In this way, control device 170 can evaluate the safety of user terminal 120, for example, by evaluating the placement of obstructions around the installation position of user terminal 120.

(Safety at Position B)
When a user 22 of the user terminal 120 places the user terminal 120 at position B so that a normal vector 242 of the display area 142 faces rightward in the figure, the space in which the image output to the display area 142 can be viewed is the space surrounded by vector 244, vector 246, wall 316, obstruction 330, and floor 320. In this case, a third party 32 can view the image displayed in the display area 142. On the other hand, a third party 34 cannot view the image displayed in the display area 142.

In this embodiment, the distance Lb between position B and wall 318 is greater than the distance La between position A and wall 314. Therefore, when user terminal 120 is installed at position B, the size of the space in which the above images can be viewed becomes larger than when user terminal 120 is installed at position A, and the possibility that a third party 32 will enter the space also becomes greater (i.e., safety is reduced).

Therefore, it is preferable that the control device 170 evaluates the safety of the user terminal 120 at position B, taking into consideration the presence or absence of a third party within the above-mentioned space, and the presence or absence of a third party intruding into the above-mentioned space. As described above, the control device 170 can evaluate, for example, the degree to which object movement or light fluctuation is detected in the vicinity of the installation position of the user terminal 120. This allows the control device 170 to accurately evaluate the safety of the user terminal 120.

In addition, according to this embodiment, a glass window 322 is provided in the wall 318 located behind the user 22. Therefore, it is preferable that the control device 170 evaluates the safety of the user terminal 120 at position B, taking into account the possibility that a third party outside the glass window 322 may view the output image.

In particular, when the outside of the glass window 322 is darker than the inside of the work room 300, it is more difficult to detect a third party outside the glass window 322 based on the information acquired by the peripheral data generating device 152 or the camera 154, compared to when the outside of the glass window 322 is brighter than the inside of the work room 300. Therefore, when the glass window 322 is present in the above-mentioned space and/or when the outside of the glass window 322 is darker than the inside of the work room 300, the safety of the user terminal 120 at position B is reduced.

As described above, the control device 170 can, for example, evaluate the brightness around the installation location of the user terminal 120. The control device 170 can evaluate the brightness outside the glass window 322 by evaluating the brightness of the glass window 322. This allows the control device 170 to accurately evaluate the safety of the user terminal 120.

(Safety at Position C)
When the user 22 of the user terminal 120 places the user terminal 120 at position C so that the normal vector 242 of the display area 142 faces rightward in the figure, the space in which the image output to the display area 142 can be viewed is the space surrounded by the vector 244, the vector 246, the wall 318, and the floor 320. The third parties 32 and 34 cannot view the image displayed in the display area 142.

According to this embodiment, markers 342, 344, and 346 are arranged on the wall 316 located behind the user 22. In this embodiment, markers 342 and 344 are arranged inside the above-mentioned space. On the other hand, marker 346 is arranged outside the above-mentioned space. While the user 22 is using the user terminal 120, the control device 170 analyzes the data acquired by the environmental sensor 150 to detect markers 342 and 344. If the control device 170 is no longer able to detect at least one of markers 342 and 344, the control device 170 determines, for example, that the safety of the user terminal 120 has decreased due to some cause. Examples of the above-mentioned cause include a third party entering between the user terminal 120 and the wall 316, or at least one of the installation position and installation direction of the user terminal 120 being changed.

In this way, the control device 170 can use various markers placed around the user terminal 120 to accurately evaluate the safety of the user terminal 120. This can highly effectively prevent information leaks.

Various pieces of information may be attached to markers 342, 344, and 346. The control device 170 can use the information attached to each marker to evaluate the safety of the user terminal 120. This further improves the accuracy of the safety determination. Therefore, the safety of the user terminal 120 at position C is further improved. Details of each marker will be described later.

(Overview of information processing in the control device 170)
As shown in Fig. 3, the degree of possibility (sometimes referred to as viewability) that the output image will be viewed by a third party 32 or a third party 34 other than the user 22 of the user terminal 120 varies greatly depending on the installation position and installation direction of the user terminal 120. However, it is difficult for the user 22 to objectively grasp the degree of viewability. Therefore, even if the user 22 installs the user terminal 120 in a place that the user 22 intuitively considers to be safe, the degree of viewability may be objectively high depending on the arrangement of obstructions around the installation position. Note that the higher the degree of viewability, the higher the risk of information leakage due to a third party peeking, and the lower the safety.

Therefore, in this embodiment, the control device 170 derives the degree of viewability based on various types of peripheral data generated by the peripheral data generating device 152. This makes it possible to objectively ensure the safety of the installation position and installation direction of the user terminal 120. The procedure for deriving the degree of viewability will be described in detail later.

(Marker Overview)
Each of the markers 342, 344, and 346 may include various types of information. Each of the markers 342, 344, and 346 may include identification information that allows the control device 170 to identify each marker. Each of the markers 342, 344, and 346 may be an encoded code such as an AR (Augmented Reality) marker, a QR code (registered trademark), or a barcode.

When each of markers 342, 344, and 346 is used as an AR marker, each marker includes, for example, information regarding a three-dimensional model of the object corresponding to each marker. This allows the control device 170 to place a three-dimensional model corresponding to the object indicated by each marker at the position of each marker in the three-dimensional virtual space corresponding to the work room 300.

Marker 342 includes, for example, information for generating a three-dimensional model of obstruction 330 or a point cloud simulating obstruction 330 to the right of the detection position of marker 342. This allows control device 170 to construct a three-dimensional model corresponding to obstruction 330 in the three-dimensional virtual space. Marker 342 may also include information for generating a three-dimensional model or point cloud having a predetermined width and height to the left of the detection position of marker 342. This allows control device 170 to construct a three-dimensional model corresponding to a portion of wall 316 in the three-dimensional virtual space.

The marker 344 includes, for example, information for generating a three-dimensional model or point cloud having a predetermined width and height to the right of the detection position of the marker 344, and information for generating a three-dimensional model or point cloud having a predetermined width and height to the left of the detection position of the marker 344. This allows the control device 170 to construct a three-dimensional model of a portion of the wall 316 in the three-dimensional virtual space.

The marker 346 includes information for generating a three-dimensional model or point cloud having a predetermined width and height, for example, to the right of the detection position of the marker 346. This allows the control device 170 to construct a three-dimensional model corresponding to a portion of the wall 316 in the three-dimensional virtual space.

Examples of AR markers include ArUco and chameleon codes. However, the AR marker is not limited to these. According to another embodiment, for example, a QR code (registered trademark) and a barcode are used as the AR marker. In still another embodiment, a subject in an image may be used as an AR marker. For example, by using a technology (for example, various deep learning technologies) that detects a known object from among the subjects in an image by image analysis, the known object can be used as an AR marker. According to the above technology, first, a classifier is made to learn the characteristics of known objects such as road signs, vending machines, microwave ovens, books, and chairs in advance. Next, the learned classifier is used to infer the orientation of the known object among the subjects in the image, the distance between the object and the imaging device, and the like. This allows the three-dimensional coordinates of the object to be derived. As described later, if the shape and size of the object are known, a three-dimensional model of the object can be constructed in a three-dimensional virtual space.

For example, when marker 342, marker 344, or marker 346 is a QR code (registered trademark) or a barcode, or when a known object is used as an AR marker, control device 170 may execute information processing when the marker is detected by referring to a database that stores the identification information of the marker in association with the content of information processing when the marker is detected. As described above, the above information processing may be information processing about a three-dimensional virtual space in which workroom 300 or a part of it is electronically reproduced.

The information processing related to the three-dimensional virtual space includes, for example, a step of identifying a position in the above-mentioned three-dimensional virtual space that corresponds to a marker detection position in the real world, and a step of generating a point cloud or a three-dimensional model that is pre-associated with the marker at the identified position in the three-dimensional virtual space. In this case, the above-mentioned database stores, for example, identification information of the marker and information indicating the shape and size of the point cloud or the three-dimensional model in association with each other.

By attaching a marker to the subject of the image, the information processing load on the control device 170 can be significantly reduced. For example, if (i) the dimensions of the marker and (ii) the relative positional relationship between the display area 142 and the peripheral data generating device 152 are known, the control device 170 can determine the relative positional relationship between the display area 142 and the object to which the marker is attached, based on a two-dimensional image in which the marker appears. The control device 170 may also determine the direction of the normal vector of the marker (sometimes referred to as the attitude of the marker) based on the two-dimensional image in which the marker appears.

For example, if the dimensions of the marker are known, the relative positional relationship between the peripheral data generating device 152 and the marker can be determined based on a two-dimensional image in which the marker appears. Also, if the relative positional relationship between the display area 142 and the peripheral data generating device 152 is known, the relative positional relationship between the display area 142 and the marker can be determined based on the relative positional relationship between the peripheral data generating device 152 and the marker.

The procedure for determining the relative positional relationship between the peripheral data generating device 152 and the marker based on a two-dimensional image in which the marker appears includes, for example, a procedure for analyzing the two-dimensional image and determining the camera coordinates of the marker. The process for determining the camera coordinates of the marker can be realized, for example, by using a publicly known library. An example of the publicly known library mentioned above is the OpenCV library. In the OpenCV library, one or more items related to the camera can be set. Examples of the above items include resolution, FPS, focal length, vertical distortion, and horizontal distortion.

The process for determining the camera coordinates of the marker includes, for example, a camera calibration process and a three-dimensional reconstruction process. The three-dimensional reconstruction process is realized, for example, by calculating a coordinate transformation matrix using the camera coordinate system, the image coordinate system, and the marker coordinate system. The coordinate transformation matrix may include a rotation matrix and a translation matrix.

If the error in the coordinate system calculation process is large, the control device 170 may calculate the coordinates of the marker by averaging the coordinate information of the marker in multiple frames captured by the peripheral data generation device 152. The number of frames may be four or more, or eight or more. This may reduce the error. The average may be calculated before the instance is identified. Note that if the variation in the coordinates of the marker in each of the multiple frames exceeds a predetermined level, the control device 170 may determine that the peripheral data generation device 152 has moved and interrupt the averaging process.

As described above, according to this embodiment, the control device 170 can generate a point cloud or a three-dimensional model specified by the marker at the position where the marker is detected. This, for example, makes it unnecessary to perform a process for identifying the contour of the object, or makes it possible to replace the object with a point cloud or a model of a simple shape. As a result, the amount of calculation required for the process to determine the relative positional relationship between the display area 142 and the object to which the marker is attached is significantly reduced. In particular, when a set of markers (for example, a pair of markers, although the number of markers per set is not limited) is attached to the edge of an obstruction, the amount of calculation required for the process to identify the location of the obstruction is significantly reduced.

As described above, each of markers 342, 344, and 346 includes, for example, identification information for identifying each of the one or more markers, and/or a pattern representing information about a point cloud or a three-dimensional model generated around the position of each marker. The information about the point cloud may include information indicating dimensions in the world coordinate system of the generated point cloud or three-dimensional model. The information about the dimensions may include information indicating the shape and information indicating the size.

Each of markers 342, 344, and 346 may include information indicating the dimensions of the marker, information regarding the object to which the marker is attached, and information regarding the degree of viewability of the location to which the marker is attached. The information regarding the object to which the marker is attached may include information indicating that the object is one that easily transmits light. Examples of objects that easily transmit light include glass, lattices, and punched metal.

In addition, according to this embodiment, an object that does not have an AR marker arranged in the real space is not reflected in the three-dimensional space. For example, if six AR markers indicating a wall having a width of 1 m and a height of 3 m are arranged horizontally every 1 m on a wall having a width of 6 m and a height of 3 m in the real space, a wall having a width of 6 m and a height of 3 m will appear in the three-dimensional space. On the other hand, if three AR markers indicating a wall having a width of 1 m and a height of 3 m are arranged every 2 m on the wall in the real space, three walls having a width of 1 m and a height of 3 m will appear in the three-dimensional virtual space, with a gap of 1 m between each wall. In this way, when the degree of viewability is calculated using a three-dimensional virtual space, the arrangement of the AR markers in the real space may affect the calculation result of the degree of viewability.

Wall 312 may be an example of one or more objects or obstructions. Wall 314 may be an example of one or more objects or obstructions. Wall 316 may be an example of one or more objects or obstructions. Wall 318 may be an example of one or more objects or obstructions. Floor 320 may be an example of one or more objects or obstructions. Glass window 322 may be an example of one or more objects or obstructions. Obstruction 330 may be an example of one or more objects or obstructions.

Marker 342 may be an example of one or more markers. Marker 344 may be an example of one or more markers. Marker 346 may be an example of one or more markers. Wall 316 may be an example of one or more objects or subjects having one or more markers attached thereto.

An example of information processing in the control device 170 will be described using Figures 4 and 5. Figure 4 shows an example of processing by the control device 170 to evaluate the safety of the user terminal 120. Figure 5 shows an example of processing in step 416.

As shown in FIG. 4, according to this embodiment, first, in step 412 (step may be abbreviated as S), the control device 170 accepts an instruction from the user 22 to the user terminal 120, which is an instruction to start the user terminal 120 or a program running on the user terminal 120 (sometimes referred to as a start instruction). The above program may be a BIOS, an OS, or an application program. The application program may be a pre-specified application program, or may be an application program for evaluating the security of the user terminal 120.

When the control device 170 receives a startup instruction from the user 22, the control device 170 starts executing the startup process corresponding to the instruction from the user 22 and starts executing an application program (sometimes called a safety evaluation program) for evaluating the safety of the user terminal 120.

When the safety evaluation program is executed, the control device 170 acquires various information related to the surroundings of the user terminal 120 from the environmental sensor 150. Immediately after the above startup process is executed, a sufficient amount of data may not be collected to accurately evaluate the safety of the user terminal 120. In such a case, it is possible to suspend execution of the program indicated by the startup process until the safety of the user terminal 120 is confirmed. However, the above method degrades the user experience of the user 22. In addition, immediately after starting the user terminal 120 or the program, it is highly likely that the user 22 is checking the safety of the surroundings.

Therefore, according to this embodiment, in S414, the control device 170 determines settings related to the evaluation of the safety of the user terminal 120. For example, the control device 170 has (i) a first setting for evaluating the safety of the user terminal 120 based on a small amount of data, and (ii) a second setting for accurately evaluating the safety of the user terminal 120 based on a sufficient amount of data. Each of the above first setting and second setting may include at least one of a setting for deriving values of various indices for evaluating safety (sometimes referred to as an index setting), and a setting for deriving an evaluation value related to safety based on the values of various indices (sometimes referred to as an evaluation setting).

As described below, in the process of deriving the values of various indices, the amount of data collected by the environmental sensor 150 also increases. Therefore, in S414, the control device 170 determines the index setting, but does not need to determine the evaluation setting. This reduces the amount of calculation.

Examples of index settings include settings indicating the type of index used in the process for evaluating the safety of the user terminal 120 out of one or more indexes related to the safety of the user terminal 120, settings related to a function or algorithm for deriving the value of the index, settings related to parameters included in the function or algorithm, etc. Examples of evaluation settings include settings related to a function or algorithm for deriving an evaluation value based on the value of the index, settings related to parameters included in the function or algorithm, etc.

For example, if the time that has elapsed since the start-up instruction was accepted satisfies a predetermined condition, the control device 170 decides to evaluate the safety of the user terminal 120 based on the first setting. On the other hand, if the time that has elapsed since the start-up instruction was accepted does not satisfy the predetermined condition, the control device 170 decides to evaluate the safety of the user terminal 120 based on the second setting. An example of a predetermined condition is that the length of the elapsed time is less than a predetermined first threshold value.

As a result, even if a sufficient amount of data is not collected to accurately evaluate the safety of the user terminal 120, such as immediately after the above-mentioned startup process is executed, the control device 170 can continue the startup process of the user terminal 120 or the program while ensuring a certain level of safety. This improves the user experience of the user 22.

Next, in S416, the control device 170 executes processing to evaluate the safety of the usage environment of the user terminal 120. The control device 170 derives the values of one or more indicators (sometimes referred to as index values) relating to the safety of the user terminal 120 based on various data acquired by the environmental sensor 150 in accordance with the index settings determined in S414. The control device 170 evaluates the safety of the user terminal 120 based on the values of the above indicators in accordance with the evaluation settings determined in S414. Specifically, the control device 170 derives an evaluation value indicating the degree of safety of the user terminal 120 based on the values of the above indicators.

In one embodiment, the control device 170 derives an evaluation value for the current safety of the user terminal 120 based on the current installation position and current installation orientation of the display area 142. In another embodiment, the control device 170 determines a combination (sometimes referred to as a pattern) of the installation position and installation orientation of the display area 142 such that the evaluation of the safety of the user terminal 120 satisfies a predetermined criterion. The installation position of the display area 142 may be represented by a numerical value or a numerical range. The installation orientation of the display area 142 may be represented by a numerical value or a numerical range.

For example, the control device 170 determines a pattern in which the degree of viewability described above is smaller than a predetermined value. As described above, when the display area 142 is housed in the housing 130, the installation position and installation direction of the display area 142 approximately coincide with the installation position and installation direction of the user terminal 120. The procedure for deriving the index value and the procedure for deriving the evaluation value will be described in detail later.

Next, in S420, the control device 170 determines whether the safety evaluation value of the user terminal 120 is within a predetermined numerical range (sometimes called an acceptable range). The above numerical range may have either an upper limit or a lower limit, or may have both an upper limit and a lower limit.

The control device 170 may adjust the sensitivity or responsiveness of the above judgment to fluctuations in the evaluation value of the safety of the user terminal 120, taking into account the user experience and system stability. As described below, if the evaluation value of the safety of the user terminal 120 is outside the acceptable range, processing for restricting the use of the user terminal 120 (sometimes referred to as a usage restriction process) may be executed in S432. Furthermore, if the evaluation value of the safety of the user terminal 120 returns to within the acceptable range, the above usage restriction may be lifted. If the sensitivity or responsiveness of the above judgment is too high, the restriction on the use of the user terminal 120 and the lifting of the restriction are frequently repeated. As a result, the screen display becomes unstable, degrading the user experience.

According to this embodiment, the sensitivity or responsiveness of the above-mentioned judgment is adjusted so that the frequency with which the above-mentioned usage restriction and restriction release are switched is less than a predetermined value. This stabilizes the system and improves the user experience. The method for adjusting the sensitivity or responsiveness of the above-mentioned judgment is not particularly limited, and any known method may be adopted.

In one embodiment, the control device 170 may determine that the evaluation value is outside the allowable range based on the integral of the amount of change in the evaluation value over a small amount of time. The control device 170 may determine whether the evaluation value is outside the allowable range based on whether the degree according to the integral of the amount of change in the evaluation value over a small amount of time satisfies a predetermined criterion. The length of the small amount of time may be between 0.1 seconds and 1 second, or may be around 0.5 seconds. This adjusts the balance between the responsiveness of the usage restriction and the stability of the system. As a result, the user's usage experience is improved.

The control device 170 may limit the use of the display area 142 of the user terminal in proportion to the degree according to the integral value of the change in the evaluation value within the above-mentioned short time. If the integral value is greater than a predetermined value (sometimes referred to as an upper limit value), the control device 170 may change the color, pattern, or brightness inside at least a part of one or more windows displayed in the display area 142 to reduce the readability of the information displayed in the window. The control device 170 may change the color, pattern, or brightness inside the largest window among the one or more windows. The control device 170 may change the color inside the above-mentioned window to gray, white, or black. The control device 170 may change the color inside the above-mentioned window to gray (sometimes referred to as graying). The window may be a display area assigned to each program running on the user terminal 120.

If the integral value is smaller than a predetermined value (sometimes referred to as a lower limit value), the control device 170, for example, executes 100% source image output to the display area 142. This achieves maximum legibility. If the integral value is equal to or greater than the lower limit value and equal to or less than the upper limit value, the control device 170 may determine the above-mentioned change in color, pattern, or brightness according to the integral value. For example, the control device 170 determines the degree of graying (semi-transparency) according to the integral value. This can limit the degree of legibility according to the integral value.

The control device 170 may temporarily release the above-mentioned usage restriction process based on instructions from the user or other authorized person. The control device 170 may relax the various thresholds described above (e.g., the above-mentioned upper limit and/or lower limit) based on instructions from the user or other authorized person. The time during which the usage restriction process can be released or the time during which the thresholds can be relaxed is determined in advance, for example, by the user or other authorized person. Examples of the above-mentioned authorized person include the user's superior, a system administrator, etc.

For example, if a window assigned to a program for a remote conference becomes increasingly grayed out while a user is speaking in the remote conference, the user experience will be degraded if the graying process is not stopped. According to the above embodiment, even in such a case, the graying process can be temporarily stopped or the graying can be canceled. This prevents the user experience from degrading.

In another embodiment, the control device 170 may determine that the evaluation value is outside the acceptable range when the duration of the state in which the evaluation value is outside the acceptable range becomes greater than a predetermined value. This prevents the evaluation value from being determined to be outside the acceptable range when the evaluation value suddenly becomes outside the acceptable range. As a result, the user's usage experience is improved.

The control device 170 may adjust the sensitivity or responsiveness of the above judgment based on the evaluation result regarding the safety of the usage environment of the user terminal 120 obtained in S416. For example, when the safety of the usage environment satisfies a predetermined standard, the control device 170 adjusts the sensitivity or responsiveness of the above judgment so that the sensitivity or responsiveness of the judgment is lower than when the safety of the usage environment does not satisfy the standard. For example, the control device 170 adjusts the sensitivity or responsiveness of the above judgment so that the sensitivity or responsiveness of the judgment is lower the greater the degree of safety of the usage environment. This improves the user's usage experience.

If the safety evaluation value of the user terminal 120 is within the acceptable range (Yes in S420), the process proceeds to S440. On the other hand, if the safety evaluation value of the user terminal 120 is outside the acceptable range (No in S420), in S430, the control device 170 determines whether the length of time that has elapsed since the user terminal 120 or the program was started based on a start-up instruction from the user 22 is greater than a predetermined second threshold. The above-mentioned elapsed time may be the time that has elapsed since the start-up instruction was accepted, may be the time that has elapsed since the start-up process of the user terminal 120 or the program was completed, or may be the time that has elapsed since the control device 170 detected the start-up of the user terminal 120 or the program.

In S430, for example, if the elapsed time is greater than the second threshold (Yes in S430), in S432, the control device 170 executes the use restriction process described above. Examples of use restriction processes include a process for restricting the acceptance of instructions from the user 22, a process for locking the screen displayed in the display area 142, a process for displaying a predetermined image in the display area 142, a process for displaying a warning in the display area 142, a process for adjusting the brightness of the display area 142 to darken the screen, and a process for narrowing the viewing angle of the display device 140.

In the process for displaying a predetermined image in display area 142, the image may be displayed over the entire screen or over a portion of the screen. In the process for displaying a predetermined image in display area 142, the image may be a semi-transparent image. The process for displaying a predetermined image in display area 142 may be a process for hiding at least a portion of the screen that would have been displayed if the usage restriction process had not been executed, or may be a process for superimposing a semi-transparent image over at least a portion of the screen that would have been displayed if the usage restriction process had not been executed.

The control device 170 may determine the type and/or level of usage restriction processing based on the evaluation result regarding the safety of the usage environment of the user terminal 120 obtained in S416. Examples of the evaluation result include (i) at least one index value of one or more indexes regarding the safety of the user terminal 120, (ii) a type of index value whose magnitude satisfies a predetermined condition among one or more indexes regarding the safety of the user terminal 120, and (iii) an evaluation value indicating the degree of safety of the user terminal 120. Examples of the predetermined condition include a condition that the index value is greater than a predetermined value, a condition that the index value is equal to or greater than a predetermined value, a condition that the index value is smaller than a predetermined value, and a condition that the index value is equal to or less than a predetermined value.

The control device 170 may determine the type and/or degree of the usage restriction processing so that the type and/or degree of the usage restriction processing differs between a case where the index value of a specific index satisfies the above-mentioned predetermined condition and a case where the index value of the index does not satisfy the condition. The control device 170 may determine the type and/or degree of the usage restriction processing so that the type and/or degree of the usage restriction processing differs between a case where the combination of indexes that satisfies the predetermined condition is a specific combination and a case where it does not.

For example, even if the safety evaluation value is outside the allowable range, if the safety of the usage environment satisfies a predetermined standard, the control device 170 determines, as the usage restriction process, a process other than at least one of the process for restricting the reception of instructions from the user 22, the process for locking the screen displayed in the display area 142, and the process for displaying a warning in the display area 142. In this case, the control device 170 may determine, as the usage restriction process, at least one of the process for displaying a predetermined image in the display area 142, the process for adjusting the brightness of the display area 142 to darken the screen, and the process for narrowing the viewing angle of the display device 140. This allows the user to continue using the user terminal 120, for example, when the usage environment is relatively safe. As a result, the user's usage experience is improved.

Examples of the degree of usage restriction processing include the amount of change in brightness, the amount of change in the viewing angle, the size of the image to be superimposed, and the transparency of the image to be superimposed. The control device 170 may determine the degree of usage restriction processing so that the safer the usage environment, the less the degree of usage restriction. The control device 170 may determine the degree of usage restriction processing so that the safer the usage environment, the less the degree of usage restriction. This allows the user to continue using the user terminal 120, for example, when the usage environment is relatively safe. As a result, the user's usage experience is improved.

While the use of the user terminal 120 is restricted, the control device 170 may or may not execute the process described in relation to S416. If the control device 170 executes the process described in relation to S416 during the period during which the use of the user terminal 120 is restricted, the safety evaluation value may return to within the acceptable range during that period.

In this case, the control device 170 may cancel or suspend the execution of the process described in relation to S432 or the process described in relation to S434, and remove the above-mentioned restriction. For example, if the degree of security of the user terminal 120 is greater than a predetermined degree (i.e., if the degree of possibility that the output image will be viewed by a third party is less than a predetermined degree), the control device 170 permits the output of the original output image.

The control device 170 may adjust the sensitivity or responsiveness of the determination as to whether or not to lift the restriction, in the same manner as when the control device 170 determines whether or not to restrict the use of the user terminal 120. This suppresses switching between restricting use and lifting the restriction. As a result, the user's usage experience is improved.

Next, in S434, the control device 170 executes a process (sometimes referred to as a guidance process) for guiding the user 22 to adjust the installation position and installation direction of the user terminal 120 so as to improve the safety of the user terminal 120 or so that the safety of the user terminal 120 satisfies a predetermined standard. In this way, the control device 170 can assist the user 22 in adjusting the installation position and/or installation direction of the user terminal 120.

The guidance process may be a process for guiding the user 22 to an appropriate installation position and/or installation direction (sometimes referred to as a guide process). In one embodiment, the control device 170 guides the user 22 by outputting a message such as "Turn the display area 142 to the right by about 10 degrees." The control device 170 may output the above message in the display area 142, or may output the above message by voice. In another embodiment, the control device 170 may superimpose a current image of the interior of the workroom 300 captured by the peripheral data generating device 152 and/or the camera 154 on an image that serves as a hint for an installation position and installation direction that satisfies the safety standard in the display area 142. Examples of the hint image include an image for highlighting an area where safety is insufficient, an image for highlighting an area where safety is sufficiently ensured, and an image for indicating the adjustment direction of the display area 142. The hint image may have a transparent or semi-transparent area.

As described above, according to one example of this embodiment, the sensitivity or responsiveness in determining whether the safety evaluation value of the user terminal 120 is within an acceptable range is adjusted. This allows the introduction of (i) a function for dynamically changing one or more thresholds or parameters that are the basis for determining safety depending on the usage environment or the display content of the display area 142, and (ii) a delay function for delaying the time until a penalty such as a screen lock or usage restriction is imposed after the safety evaluation value exceeds a threshold or the evaluation value falls outside the acceptable range.

For example, an office is considered safe because there is little chance of intrusion by outsiders, and the threshold value used as the standard for determining safety can be lowered. The environment and display content can be automatically determined and the threshold value can be changed dynamically, or the threshold value can be changed by an operation by a superior. In addition, when automatically determining whether it is an office, it can be determined that it is an office based on the presence or absence of a specific SID using the identification ID of the wireless connection (commonly called a SID), or it can be confirmed and determined based on a wired connection or its attributes, or the presence of a specific server or interface.

If the contents displayed in the display area 142 are confidential information, the aforementioned threshold may be raised to strictly evaluate the security judgment. The confidentiality of the displayed contents may be automatically evaluated based on the type of application used or the frequency of occurrence of words that are perceived as confidential information. The exceptional function described above may, for example, be implemented by introducing a boost button that lowers the judgment, and dynamically relaxing the threshold by pressing the boost button when desired or approved by the user. The boost state may be released over time, or automatically released when a large change in the state around the display area 142 is detected (when a large environmental change such as a movement of the terminal location or a change in orientation, or a change in brightness is detected in the state around the terminal).

Furthermore, the control device 170 may store some or all of the images and information acquired by the environmental sensor 150 during processing to lock the screen displayed in the aforementioned display area 142, or in exceptional circumstances such as when the level of safety is low or when the safety assessment is explicitly relaxed or strengthened. Furthermore, the time range of the stored information may be a sufficient time that includes the occurrence time of the exceptional situation, or it may be a partial time, it may be discrete, or it may target only slightly related information.

(Example of detailed guide process)
In the present embodiment, the imaging direction of the peripheral data generating device 152 and/or the camera 154 substantially coincides with the normal direction of the display area 142. In this case, the control device 170 can assist the user 22 in adjusting the installation position and/or installation direction of the user terminal 120, for example, by the following procedure.

In one embodiment, the control device 170 controls the display device 140 to display in the display area 142 an image (sometimes referred to as an adjustment image) in which (a) a current image of the interior of the workroom 300 captured by the peripheral data generating device 152 and/or the camera 154 and (b) (i) an image of the interior of the workroom 300 (sometimes referred to as a guide image) or (ii) an icon indicating the position of a characteristic point in the interior of the workroom 300 (sometimes referred to as a guide icon) in the case where the display area 142 is installed according to the combination of the installation position and installation direction determined in S416 are superimposed. The current image and the guide image or guide icon may be superimposed by a known method.

The control device 170 may control the display device 140 to display an icon (sometimes referred to as an adjustment icon) indicating the adjustment direction of the display area 142 in the display area 142. The control device 170 may control the display device 140 to display an image for highlighting an area where safety is insufficient, an image for highlighting an area where safety is sufficiently ensured, and the like in the display area 142.

The current image of the interior of the work room 300 is generated, for example, based on the current two-dimensional image or three-dimensional data captured in S434. On the other hand, the guide image or guide icon is generated, for example, based on the past two-dimensional image or three-dimensional data captured in S416.

In S416, multiple two-dimensional images or three-dimensional data with different imaging positions and/or imaging directions can be acquired. Based on these two-dimensional images or three-dimensional data, the control device 170 generates two-dimensional images or three-dimensional data that would be captured by the surrounding data generating device 152 and/or the camera 154 if the display area 142 were installed according to the combination of installation position and installation direction determined in S416. The control device 170 generates a guide image or guide icon based on the above two-dimensional images or three-dimensional data.

For example, the control device 170 may determine to use a single two-dimensional image or a portion thereof captured by the peripheral data generating device 152 and/or the camera 154 as a guide image. The control device 170 may determine the display position of the guide icon based on the single two-dimensional image. The control device 170 may determine to generate a combined image by synthesizing multiple two-dimensional images captured by a single or multiple peripheral data generating devices 152 and/or cameras 154, and to use the combined image or a portion thereof as a guide image. The control device 170 may determine the display position of the guide icon based on the combined image.

In another example, the control device 170 may generate a guide image or a guide icon using the three-dimensional virtual space described above. For example, the control device 170 first generates a three-dimensional model of the structures constituting the work room 300 and/or the structures arranged inside the work room 300 based on the past two-dimensional images or three-dimensional data captured in S416. The control device 170 may generate the above three-dimensional model based on multiple two-dimensional images or three-dimensional data captured at different imaging positions and/or imaging directions.

This generates a three-dimensional virtual space of the workroom 300. For example, three-dimensional models of walls, pillars, doors, windows, furniture, etc. are generated based on multiple AR markers captured in multiple two-dimensional images. In addition, the generated three-dimensional models are placed in appropriate positions in the three-dimensional space corresponding to the workroom 300 based on the positions and orientations of the AR markers obtained by analyzing the two-dimensional images.

Next, the control device 170 determines the position and orientation (sometimes referred to as posture) of the peripheral data generating device 152 and/or the camera 154 when the display area 142 is installed according to the combination of the installation position and installation orientation determined in S416. The control device 170 generates an image of the interior of the workroom 300 captured by the peripheral data generating device 152 and/or the camera 154, assuming that the peripheral data generating device 152 and/or the camera 154 are placed at the above-mentioned positions and in the above-mentioned postures in the three-dimensional virtual space of the workroom 300. This enables the control device 170 to generate a guide image or a guide icon.

By displaying the adjustment image or adjustment icon in the display area 142, the user 22 can adjust the installation position and installation direction of the user terminal 120 while viewing the screen output in the display area 142. For example, the user 22 adjusts the installation position and/or installation direction of the display area 142 so that the current image of the interior of the work room 300 and the guide image approximately match. For example, the user 22 adjusts the installation position and/or installation direction of the display area 142 so that the position of the feature point in the current image of the interior of the work room 300 approximately matches the position of the guide icon in the adjustment image.

According to this embodiment, even if the viewing angle of the display area 142 is larger than the viewing angle of the peripheral data generating device 152 and/or the camera 154, the installation position and installation direction of the user terminal 120 can be adjusted. Note that the adjustment method according to this embodiment can also be applied when the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generating device 152 and/or the camera 154.

When the peripheral data generating device 152 and/or the camera 154 are mounted on the user terminal 120 as in the user terminal 120 according to the present embodiment, depending on the type of the peripheral data generating device 152 and/or the camera 154, when the user 22 moves the user terminal 120 to adjust the installation position and installation direction of the user terminal 120, a strong afterimage may appear in the image captured by the peripheral data generating device 152 and/or the camera 154. For example, when the peripheral data generating device 152 is a LiDAR using a laser, even if the user 22 moves the user terminal 120 relatively quickly, the amount of afterimage is extremely small. On the other hand, when the peripheral data generating device 152 and/or the camera 154 are an image sensor such as a CMOS, when the user 22 moves the user terminal 120 relatively quickly, the peripheral data generating device 152 and/or the camera 154 cannot obtain a clear image. Furthermore, when the peripheral data generating device 152 and/or the camera 154 capture the above-mentioned markers, if the degree of movement of the peripheral data generating device 152 and/or the camera 154 becomes large, the coordinates of the markers cannot be calculated with high accuracy in the process for determining the camera coordinates of the markers.

The control device 170 may therefore output information to guide the user on how to move the user terminal 120. For example, the control device 170 may output instructions indicating the moving speed of the user terminal 120, instructions indicating whether the moving speed of the user terminal 120 is appropriate or inappropriate, instructions to stop the user terminal 120, and the like. The above instructions may be displayed in the display area 142, or may be output as audio.

While the guide process is being executed, the control device 170 repeatedly evaluates the safety of the user terminal 120. For example, the control device 170 determines whether the evaluation value of the safety of the user terminal 120 is within an acceptable range. The control device 170 may determine whether the installation position and installation direction of the display area 142 are approximately the same as the installation position and installation direction determined in S416.

If it is determined that the safety evaluation value of the user terminal 120 is within the acceptable range, or if it is determined that the installation position and installation direction of the display area 142 are approximately the same as the installation position and installation direction determined in S416, the control device 170 outputs a message indicating that the user terminal 120 is installed in a safe position. For example, the control device 170 controls the display device 140 to display the above message in the display area 142. The control device 170 may output the above message by voice. This allows the user 22 to easily adjust the installation position and installation direction of the user terminal 120.

(An example of another embodiment of the induction process)
In this embodiment, the control device 170 repeatedly evaluates the safety of the user terminal 120 at all times, periodically, or at any timing. Therefore, even if guide information on how to move the user terminal 120 is not output, the user 22 can install the user terminal 120 in a position and direction that ensures safety. More specifically, the user 22 arbitrarily changes the installation position and/or installation direction of the user terminal 120. Each time the installation position and/or installation direction of the user terminal 120 is changed, the control device 170 evaluates the safety of the user terminal 120 at the installation position and installation direction, and presents the result of the evaluation to the user 22. This allows the user 22 to install the user terminal 120 in a position and direction that ensures safety.

When the guidance process ends in S434, the control device 170 repeats the process from S416 onwards. This allows the control device 170 to continue monitoring the safety of the user terminal 120.

On the other hand, in S430, for example, if the elapsed time is less than the second threshold (No in S430), in S440, the control device 170 inquires of the user 22, for example, whether or not the user wishes to be suggested a more appropriate usage environment. For example, if the user 22 inputs information indicating that the user wishes to be suggested to the user terminal 120 (Yes in S440), the control device 170 accepts the input and executes the guide process described in relation to S434.

On the other hand, in S440, for example, if the user 22 does not input information indicating that the above proposal is desired into the user terminal 120, or if the user 22 inputs information indicating that the above proposal is not desired into the user terminal 120 (No in S440), in S440, the control device 170 determines, for example, whether to terminate the user terminal 120 or the program started in S412. Specifically, the control device 170 determines whether an instruction (sometimes referred to as an instruction to terminate) the user terminal 120 or the above program has been received.

If the termination instruction has been received (Yes in S450), the control device 170 terminates the user terminal 120 or the program. This ends the process. On the other hand, if the termination instruction has not been received (No in S450), the control device 170 repeats the process from S416 onwards. This allows the control device 170 to continue monitoring the safety of the user terminal 120.

(An example of a process for evaluating the safety of the usage environment)
As described above, FIG. 5 shows an example of the process in step 416. As shown in FIG. 5, in this embodiment, the control device 170 comprehensively judges the values of a plurality of indexes including a first index, a second index, a third index, and a fourth index to evaluate the safety of the user terminal 120, and the details of the process (sometimes referred to as an evaluation process) for evaluating the safety of the user terminal 120 are described as an example. Note that even when the control device 170 comprehensively judges the values of a plurality of indexes to evaluate the safety of the user terminal 120, the safety of the user terminal 120 may ultimately be evaluated based on the value of a single index.

As described above, the evaluation process described in relation to S416 is periodically repeated while the user terminal 120 or the program is running. When the first evaluation process is executed, the types of the first index, second index, third index, and fourth index are determined, for example, based on the index settings determined in S414. In the first evaluation process, at least one of the first index, second index, third index, and fourth index may not be derived.

According to this embodiment, first, in S512, the control device 170 derives the value of a first index included in the multiple indexes. The control device 170 derives the value of the first index, for example, based on the index setting determined in S414. The first index may be an index indicating the degree to which object movement or light fluctuation is detected in the vicinity of the installation position of the user terminal 120.

In S514, the control device 170 derives the value of a second index included in the multiple indexes. The control device 170 derives the value of the second index based on, for example, the index setting determined in S414. The second index may be an index indicating the arrangement of obstructions present around the installation position of the user terminal 120. The second index may be an index indicating the degree to which the obstruction blocks a third party's line of sight to the display area 142.

In S516, the control device 170 derives the value of a third index included in the multiple indexes. The control device 170 derives the value of the third index based on, for example, the index setting determined in S414. The third index may be an index indicating the condition of the surrounding environment of the installation location of the user terminal 120.

The third index may be an index indicating the illuminance around the installation location of the user terminal 120 or the distribution or variation of the illuminance. The third index may be an index indicating the outdoor brightness in the area where the installation location of the user terminal 120 belongs. The third index may be an index indicating the degree of temperature distribution around the installation location of the user terminal 120. The third index may be an index indicating the degree of visibility around the installation location of the user terminal 120. The third index may be an index indicating the concentration of suspended matter, flying matter, or odorous matter around the installation location of the user terminal 120 or the distribution or variation of the concentration.

In S518, the control device 170 derives the value of a fourth index included in the multiple indexes. The control device 170 derives the value of the fourth index based on, for example, an object or event observed in the vicinity of the installation position of the user terminal 120. The fourth index may be an index indicating the risk of information leakage according to the type of object or event. For example, the control device 170 refers to a database in which, for each of one or more objects or events, the type and/or attributes of the object or event are associated with a value indicating the degree of possibility that a third party will view the output image from the position where the object or event was observed, and determines the value of the fourth index based on the type and/or attributes of the object or event actually observed.

In one embodiment, at least two steps included in S512, S514, S516, and S518 are performed in parallel. In another embodiment, at least two steps included in S512, S514, S516, and S518 are performed sequentially.

Next, in S520, the control device 170 determines a procedure for deriving an evaluation value indicating the degree of safety of the user terminal 120 based on the values of one or more indicators. In this embodiment, the control device 170 determines the above-mentioned evaluation setting based on various data acquired by the environmental sensor 150 during the execution periods of S512, S514, S516, and S518.

For example, the control device 170 determines the type of one or more indicators to be used to derive the evaluation value from among the multiple indicators based on (i) the information acquired by the environmental sensor 150 and/or (ii) the current time or the time zone to which the current time belongs. The control device 170 may determine the type of one or more indicators to be used to derive the evaluation value from among the multiple indicators based on (i) the information acquired by the environmental sensor 150, (ii) the current time or the time zone to which the current time belongs, and (iii) the latitude and/or longitude of the installation location of the user terminal 120.

For example, the control device 170 determines a weighting value to be set for each of one or more indicators used to derive an evaluation value based on (i) the information acquired by the environmental sensor 150 and/or (ii) the current time or the time zone to which the current time belongs. The control device 170 may determine a weighting value to be set for each of one or more indicators used to derive an evaluation value among the multiple indicators based on (i) the information acquired by the environmental sensor 150, (ii) the current time or the time zone to which the current time belongs, and (iii) the latitude and/or longitude of the installation location of the user terminal 120.

Next, in S530, the control device 170 derives an evaluation value indicating the degree of safety of the user terminal 120 based on the values of one or more indicators. The control device 170 derives the above evaluation value based on the evaluation settings determined in S520. For example, the control device 170 derives the above evaluation value based on the values of each of the multiple indicators derived in S512, S514, and S516 and the weighting values set for each of the multiple indicators. This allows the control device 170 to evaluate the safety of the user terminal 120 or the degree of viewability described above.

The process for evaluating the security of the user terminal 120 may be an example of a process for deriving viewability.

(An example of another embodiment)
In the present embodiment, the details of the information leakage prevention system 100 have been described by taking as an example a case in which the control device 170 evaluates the safety of the user terminal 120 based on a plurality of indexes. However, the information leakage prevention system 100 is not limited to the present embodiment.

In another embodiment, the control device 170 may evaluate the safety of the user terminal 120 based on at least one of the first index, the second index, the third index, and the fourth index. For example, the control device 170 may evaluate the safety of the user terminal 120 by comprehensively judging the values of one or more indexes. In yet another embodiment, the control device 170 may evaluate the safety of the user terminal 120 based on the value of a single predetermined index.

In this embodiment, the details of the information leakage prevention system 100 are described using as an example a case in which the use restriction process is executed in S432 and then the guidance process is executed in S434. However, the information leakage prevention system 100 is not limited to this embodiment.

In other embodiments, step S434 may be omitted. For example, the control device 170 may omit step S434 in accordance with a user instruction or setting.

Figure 6 shows an example of the internal configuration of the control device 170. In this embodiment, the control device 170 includes, for example, a surrounding state acquisition unit 610, a start-up detection unit 622, a timing unit 624, a virtual space construction unit 630, a safety management unit 640, a placement determination unit 652, a placement support unit 654, an output control unit 660, and a storage unit 670. In this embodiment, the surrounding state acquisition unit 610 includes, for example, a structural information acquisition unit 612, a physical property information acquisition unit 614, an environmental information acquisition unit 616, and an attribute information acquisition unit 618. Each element constituting the control device 170 is configured to be able to send and receive information to each other.

In this embodiment, the surrounding state acquisition unit 610 acquires information (sometimes referred to as surrounding state information) indicating the state of the surroundings of the display area 142. The surrounding state acquisition unit 610 acquires, for example, surrounding state information generated by the environmental sensor 150. The surrounding state acquisition unit 610 may output the surrounding state information to the safety management unit 640. In this embodiment, the surroundings of the display area 142 may include the installation position of the display area 142 and the vicinity of the installation position.

In this embodiment, the structural information acquisition unit 612 acquires information (sometimes referred to as surrounding data) indicating the shape, size, and/or arrangement of one or more objects arranged around the display area 142. The structural information acquisition unit 612 outputs the surrounding data to, for example, the safety management unit 640. The surrounding data may be an example of surrounding state information.

The structural information acquisition unit 612 acquires, for example, image data of an image having as its subject one or more objects arranged around the display area 142. The structural information acquisition unit 612 acquires the above image data from, for example, the peripheral data generation device 152. The above image may be a still image or a moving image. The above image may be a stereo image. The image data may be an example of peripheral data.

The structural information acquisition unit 612 acquires, for example, three-dimensional point cloud data or distance data of one or more objects arranged around the display area 142. The structural information acquisition unit 612 acquires the above three-dimensional point cloud data or distance data from, for example, the surrounding data generation device 152. The distance data may include information indicating a relative distance from a reference position and information indicating a distance ranking index. The reference position may be a predetermined specific position or a specific object. The three-dimensional point cloud data and/or distance data may be an example of surrounding data.

Examples of distance ranking indices include indices derived using monocular camera depth estimation technology such as MiDaSnet. In many cases, the above indices are not proportional to absolute distance and can be used as reference indices for ranking. For example, the learned person information can be used to infer the absolute scale of the estimation target by referring to the scale learning scale from its similarity, and the distance to the target can be calculated from this.

In this embodiment, the physical property information acquisition unit 614 acquires information indicating the physical properties of one or more objects arranged around the display area 142 (sometimes referred to as physical property information). The physical property information acquisition unit 614 outputs the physical property information to, for example, the safety management unit 640. The physical property information may be an example of surrounding state information.

The physical property information includes, for example, information indicating the optical properties of the surface of the object. The physical property information includes, for example, information indicating the characteristics of the electromagnetic waves traveling from the surface of the object toward the display area 142. The characteristics of the electromagnetic waves may be characteristics related to optical properties. The electromagnetic waves may be electromagnetic waves reflected from the surface of the object, electromagnetic waves transmitted through the object, or electromagnetic waves emitted by the object. Examples of optical properties include wavelength, amplitude, intensity, and brightness. Examples of light intensity and brightness include irradiance, photon density, and illuminance.

The physical property information acquisition unit 614 acquires, for example, image data of an image having one or more objects arranged around the display area 142 as its subject. The physical property information acquisition unit 614 acquires, for example, image data output by the camera 154. The above image data may be still image data or moving image data. As will be described later, in this embodiment, the above-mentioned first index is derived based on the fluctuation in light intensity. For this reason, it is preferable that the above image data is moving image data.

This allows the physical property information acquisition unit 614 to acquire color information of one or more objects arranged around the display area 142. Color information may be an example of physical property information. Physical property information of a transparent or translucent object may include information indicating the characteristics of electromagnetic waves that have passed through the object. For example, color information of a window glass may include color information of an object that is visible through the window glass.

The physical property information acquisition unit 614 may acquire multiple pieces of video data captured in different imaging directions, the multiple pieces of video data being video images of the periphery of a specific position in real space. The multiple pieces of video data may be multiple pieces of video data captured in multiple separate sessions, or multiple pieces of video data cut out from multiple positions in a single video.

For example, if the user 22 wishes to install the user terminal 120 at a specific location, the user 22 rotates the user terminal 120 around the desired location and captures an image of the surrounding area of the location using the camera 154 of the user terminal 120. This allows multiple video data captured in different directions to be acquired.

The user 22 may capture the surroundings of the above-mentioned position without the user 22 himself appearing in the image. For example, the user 22 captures the surroundings of the user terminal 120 while positioned behind the camera 154. This omits the process of removing the image of the user 22 from the image captured by the camera 154.

In this embodiment, the environmental information acquisition unit 616 acquires, for example, information indicating the status of the surrounding environment of the display area 142 (sometimes referred to as environmental information). The environmental information acquisition unit 616 acquires, for example, environmental information generated by the environmental sensor 150. The environmental information acquisition unit 616 may output the environmental information to the safety management unit 640. The environmental information may be an example of surrounding state information.

For example, the environmental information acquisition unit 616 acquires information indicating the illuminance around the display area 142 (sometimes referred to as illuminance information). For example, the environmental information acquisition unit 616 acquires information from the illuminance sensor 156 indicating the results of the measurement of the illuminance by the illuminance sensor 156.

For example, the environmental information acquisition unit 616 acquires information (sometimes referred to as temperature information) indicating the temperature around the display area 142. For example, if the camera 154 is configured to be able to generate infrared images, the environmental information acquisition unit 616 acquires image data of the thermal image generated by the camera 154.

For example, the environmental information acquisition unit 616 may acquire various information provided by other information providing systems via a communication network. For example, the environmental information acquisition unit 616 acquires weather information, time information, etc. from other information providing systems.

In this embodiment, the attribute information acquisition unit 618 acquires, for example, information indicating the type or attributes of an object or event detected in the vicinity of the display area 142 (sometimes referred to as attribute information). The environmental information acquisition unit 616 may output the attribute information to the safety management unit 640. The attribute information may be an example of surrounding state information.

More specifically, the environmental information acquisition unit 616 acquires, for example, image data and/or audio data output by the environmental sensor 150. The attribute information acquisition unit 618 analyzes the image data and/or audio data to detect an object or event. The attribute information acquisition unit 618 determines the type or attribute of the detected object or event, for example, by referring to an appropriate database. In this way, the attribute information acquisition unit 618 can acquire attribute information.

For example, the attribute information acquisition unit 618 analyzes image data of the periphery of the display area 142 to detect one or more objects disposed in the periphery of the display area 142. The attribute information acquisition unit 618 may refer to a database that stores the characteristics and/or attributes of various objects to identify the detected object or determine the attributes of the object. The method of detecting an object in an image and identifying the detected object is not particularly limited, and any known method may be applied.

For example, the attribute information acquisition unit 618 may analyze audio data that records audio around the display area 142 to detect events that have occurred around the display area 142. The attribute information acquisition unit 618 may refer to a database that stores the characteristics and/or attributes of various events to distinguish the detected event and determine the attributes of the event.

For example, if a dog's bark or growl is detected, it is determined that there is a person nearby. Also, if a person's voice or footsteps are detected, it is possible to determine that there is a person nearby. If the footsteps of an approaching person are detected, it is possible to determine that there is an increase in the number of people nearby. If the footsteps of a person moving away are detected, it is possible to determine that there is a decrease in the number of people nearby.

In this embodiment, the startup detection unit 622 detects the startup of the user terminal 120, or the startup of the BIOS, OS, or application program running on the user terminal 120. When the startup of the user terminal 120, or the startup of the BIOS, OS, or application program running on the user terminal 120 is detected, the startup detection unit 622 may output information indicating that the above startup has been detected to the timer unit 624 and/or the safety management unit 640.

In this embodiment, the timing unit 624 measures the elapsed time since the activation detection unit 622 detected the above-mentioned activation. The timing unit 624 may output information indicating the measurement result to the safety management unit 640.

In this embodiment, the virtual space construction unit 630 constructs a three-dimensional virtual space corresponding to the periphery of the user terminal 120. The virtual space construction unit 630 constructs the above-mentioned three-dimensional virtual space based on, for example, information acquired by the structural information acquisition unit 612. Details of the virtual space construction unit 630 will be described later.

In this embodiment, the safety management unit 640 derives the degree of viewability of the output image by a third party other than the user 22 based on the state of the vicinity of the display area 142 indicated by the surrounding state information acquired by the surrounding state acquisition unit 610. The safety management unit 640 may evaluate the safety of the display area 142 based on the state of the vicinity of the display area 142.

For example, the safety management unit 640 determines that the safety of the display area 142 is higher when the risk of information displayed in the display area 142 being leaked is lower (i.e., the degree of viewability is lower). Examples of the safety of the display area 142 include the safety of the display area 142 installed in a specific position and direction, the safety of the installation position of the display area 142, and the safety of the installation position and installation direction of the display area 142. When the display area 142 is housed in the housing 130, the safety of the display area 142 may mean the safety of the user terminal 120.

The safety management unit 640, for example, comprehensively evaluates various pieces of information collected by the structural information acquisition unit 612 to derive the degree of viewability when the display area 142 is installed at a specific position and direction. More specifically, the safety management unit 640, for example, calculates the values of one or more types of indices indicating the degree of viewability. The safety management unit 640 derives an evaluation value indicating the degree of viewability or the degree of safety using a learning model or function for comprehensively evaluating the values of the one or more indices. The safety management unit 640 may derive an evaluation value for each of multiple patterns having different arrangements of the display area 142. Details of the safety management unit 640 will be described later.

In one embodiment, the safety management unit 640 evaluates the safety of the display area 142 when the process of determining the layout of the display area 142 is executed. The process of determining the layout of the display area 142 is executed, for example, at least one of the following times: when the user 22 starts using the user terminal 120, when the user 22 starts using a specific application program, and when the lock or sleep mode is released. In another embodiment, the safety management unit 640 evaluates the safety of the display area 142 periodically or at any time during the period when the user 22 is using the user terminal 120.

In this embodiment, the placement determination unit 652 determines the placement of the display area 142. The placement determination unit 652 may determine the placement of the display area 142 based on the evaluation result of the safety management unit 640. An example of the evaluation result of the safety management unit 640 is the degree of viewability. As described above, the degree of safety of the user terminal 120 is an example of the degree of viewability.

Examples of the arrangement of the display area 142 include the position of the display area 142 (sometimes referred to as the installation position) and/or the direction of the normal vector 242 of the display area 142 (sometimes referred to as the installation direction). Examples of the direction of the normal vector 242 include at least one of the direction of the vector in the horizontal direction and the direction of the vector in the vertical direction. Note that when the display area 142 is housed in the housing 130, the arrangement of the display area 142 and the arrangement of the user terminal 120 approximately match.

In one embodiment, the placement determination unit 652 acquires information indicating the placement of the display area 142 and information indicating the degree of viewability in the placement from the safety management unit 640. Examples of information indicating the placement of the display area 142 include information indicating the installation position of the display area 142, and information indicating the installation position and installation direction of the display area 142.

If the degree of viewability satisfies a predetermined criterion, the layout determination unit 652 determines the above layout as the layout of the display area 142. An example of the predetermined criterion is that the degree of viewability is smaller than a predetermined degree.

The above-mentioned predetermined degree may be different when the display area 142 is located within an area having a predetermined characteristic and when the display area 142 is located outside the area. The above-mentioned area may define the installation position of the display area 142, or may define the installation position and installation direction of the display area 142.

According to one embodiment of the information leakage prevention system 100, even in an area where the degree of viewability is relatively high, if the use of the user terminal 120 in that area has been approved in advance by a person (sometimes called an approver) who has the authority to permit the user 22 to use the user terminal 120, the user 22 can use the user terminal 120. In this case, the standard regarding the degree of viewability that is applied when it is confirmed that the display area 142 is located within the range of the area is different from the standard regarding the degree of viewability that is applied when it is not confirmed that the display area 142 is located within the range of the area. Note that the above approval may specify a start time when the approval becomes effective, and may specify an expiration date for the approval.

The degree of viewability that is applied when it is confirmed that the display area 142 is located within the range of the area may be greater than the degree of viewability that is applied when it is not confirmed that the display area 142 is located within the range of the area. When it is confirmed that the display area 142 is located within the range of the area, the determination regarding the degree of viewability may be omitted.

In this case, examples of the predetermined features include features of the peripheral data acquired by the peripheral data generating device 152 of the user terminal 120 located in the area where the approver has approved the use of the user terminal 120. More specifically, examples include one or more feature points included in a two-dimensional image captured by the peripheral data generating device 152, and one or more feature points included in the point cloud data acquired by the peripheral data generating device 152. The predetermined features may be two-dimensional images or three-dimensional data captured by the peripheral data generating device 152 when the approver approves the use of the user terminal 120. Note that the above approval may specify a start time when the approval becomes effective, and may specify an expiration date for the approval.

According to this embodiment, information indicating the spatial structure at the time when the approver approves the use of the user terminal 120 is recorded in the user terminal 120. The user terminal 120 may be provided with a database that stores information indicating the spatial structure at the time when the approver approves the use of the user terminal 120 and information indicating the degree of viewability.

This allows, for example, the control device 170 of the user terminal 120 to compare the data acquired by the surrounding data generating device 152 at the current position and orientation of the user terminal 120 with the data of the recorded spatial structure. If the two match, the control device 170, for example, refers to the above-mentioned database and extracts information indicating the degree of viewability stored in correspondence with the recorded spatial structure. The control device 170 may evaluate the safety of the user terminal 120 based on the extracted information indicating the degree of viewability. Note that, if the two match, the control device 170 may determine that the degree of viewability is less than a predetermined degree and that the installation position and/or orientation of the user terminal 120 is safe.

According to this embodiment, if the user terminal 120 or the display area 142 is installed in a similar manner to a space in which an approving person has previously approved the use of the user terminal 120, the user 22 can use the user terminal 120. For example, if an approving person has approved the use of the user terminal 120 in a seat having specific characteristics on a Shinkansen train, the user 22 can use the user terminal 120 without having to apply for approval every time he or she uses the Shinkansen.

The above approval may be set for each user 22, or for each space or feature of a space. When the above approval is set for each space or feature of a space, for example, if user A performs an approval operation for a seat with specific features on a Shinkansen train and approval is obtained for that seat, when another user B uses a seat with similar features on another Shinkansen train, user B can use the user terminal 120 without applying for approval.

According to one embodiment of the information leakage prevention system 100, even in an area where the degree of viewability is relatively high, for example, when the user 22 uses the user terminal 120 in a room where only persons with confidentiality obligations can enter, the user 22 can use the user terminal 120. In this case, the standard regarding the degree of viewability that is applied when it is confirmed that the display area 142 is located within the range of the area is different from the standard regarding the degree of viewability that is applied when it is not confirmed that the display area 142 is located within the range of the area.

The degree of viewability that is applied when it is confirmed that the display area 142 is located within the range of the area may be greater than the degree of viewability that is applied when it is not confirmed that the display area 142 is located within the range of the area. When it is confirmed that the display area 142 is located within the range of the area, the determination regarding the degree of viewability may be omitted. Note that when it is confirmed that the display area 142 is located within the range of the area, a function or system for determining the degree of viewability may not be activated.

In this case, examples of the predetermined characteristic include a wireless signal or sound wave that can be received in the area, a presence or connection check to a specific OBJ via a wired LAN connection in the area, and the like. Similarly, when the user terminal 120 acquires information indicating that the user 22 is located within the range of the area from an attendance management system or the like, the determination regarding the degree of viewability may be omitted. Also, when the user terminal 120 acquires (i) information indicating that the approver or the person himself/herself has approved that the display area 142 is located within the range of the area, or (ii) information indicating that the approver or the person himself/herself has approved the use of the user terminal 120, the determination regarding the degree of viewability may be omitted.

On the other hand, if the degree of viewability does not satisfy the predetermined criteria, the placement determination unit 652 determines to output a message to the user 22 to prompt the user 22 to change at least one of the installation position and installation direction of the display area 142. The placement determination unit 652 controls the display device 140 to display the message in the display area 142, for example. The message may be output by voice.

In another embodiment, the placement determination unit 652 acquires from the safety management unit 640 information indicating the placement of the display area 142 and information indicating the degree of viewability in that placement for each of a plurality of cases in which the placement of the display area 142 is different. Examples of information indicating the placement of the display area 142 include information indicating the installation position of the display area 142, and information indicating the installation position and installation direction of the display area 142.

The arrangement determination unit 652 compares the degree of viewability of each of the multiple cases with a predetermined criterion, and extracts cases in which the degree of viewability satisfies the predetermined criterion. The arrangement determination unit 652 determines one of the arrangements indicated by the extracted cases as the arrangement of the display area 142.

For example, the placement determination unit 652 determines the placement indicated by the case with the best degree of viewability as the placement of the display area 142. For example, the placement determination unit 652 extracts a number of cases with an especially good degree of viewability. The placement determination unit 652 may extract the above cases based on the degree of viewability, or may extract the above cases such that the number of extracted cases is equal to or less than a predetermined number. The placement determination unit 652 presents information indicating the placement of each of the extracted cases to the user 22, and determines the case selected by the user 22 as the placement of the display area 142.

In this embodiment, the placement support unit 654 supports the user 22 in setting up the display area 142 or the user terminal 120. The placement support unit 654 may support the user 22 in setting up the display area 142 or the user terminal 120 by executing the guidance process or guide process in S434 described in relation to FIG. 4.

In this embodiment, the output control unit 660 controls the output of images by the display device 140. The output control unit 660 may execute various usage restriction processes in S432 described in relation to FIG. 4. The output control unit 660 may execute a process for removing usage restrictions (sometimes referred to as a restriction removal process).

For example, if the degree of viewability derived by the safety management unit 640 is less than a predetermined degree, the output control unit 660 permits the output of the output image. For example, if the evaluation value indicating the degree of viewability or the degree of safety described above is within the range of the above-mentioned tolerance, the output control unit 660 determines that the degree of viewability is less than a predetermined degree.

For example, if the degree of viewability derived by the safety management unit 640 exceeds a predetermined degree, the output control unit 660 decides to execute the above-mentioned usage restriction process. For example, if the evaluation value indicating the above-mentioned degree of viewability or degree of safety is outside the above-mentioned allowable range, the output control unit 660 determines that the degree of viewability exceeds the predetermined degree. The output control unit 660 may stop outputting the output image that was displayed before the degree of viewability exceeded the predetermined degree, may adjust the brightness or contrast of the output image, or may display an image different from the above-mentioned output image in the display area 142.

The output control unit 660 can adjust the brightness or contrast of the output image to white out or black out the screen displayed in the display area 142. This interferes with the output of the original output image. The output control unit 660 causes an image different from the original output image to be displayed in the display area 142, thereby interfering with the output of part or all of the original output image. The image that is output to interfere with the output of the original output image may be a translucent image or an opaque image.

For example, when the degree of viewability derived by the safety management unit 640 becomes smaller than a predetermined degree while the use of the user terminal 120 is restricted, the output control unit 660 permits the output of the original output image. For example, the output control unit 660 decides to execute a restriction release process. When the restriction release process is executed, the output of the original output image is resumed. When the evaluation value indicating the degree of viewability or the degree of safety described above is within the allowable range described above, the output control unit 660 may determine that the degree of viewability is smaller than the predetermined degree.

Examples of images that are different from the above output image include images for outputting a screen that may be viewed by a third party, images for outputting a warning, and images for preventing the viewing of the original output image. These images may have the same configuration as the various images described in relation to the usage restriction process.

If the output control unit 660 starts to execute the usage restriction process the moment the degree of viewability exceeds a predetermined degree, or if the output control unit 660 starts to execute the restriction removal process the moment the degree of viewability falls below a predetermined degree, there is a possibility that the user terminal 120 may frequently switch between a state in which the use of the user terminal 120 is restricted and a state in which the use of the user terminal 120 is not restricted. Frequent switching between a state in which the use of the user terminal 120 is restricted and a state in which the use of the user terminal 120 is not restricted results in a degraded user experience.

Therefore, the output control unit 660 may adjust the sensitivity or responsiveness of the above-mentioned judgment regarding the degree of viewability to fluctuations in the evaluation value of the safety of the user terminal 120, taking into account the user experience and the stability of the system. As described above, any known method may be adopted as a method for adjusting the sensitivity or responsiveness. This adjusts, for example, the time until a fluctuation in the evaluation value of the safety of the user terminal 120 affects the judgment result regarding the degree of viewability and the magnitude relationship of the predetermined degree. This can reduce the frequency of switching between a state in which the use of the user terminal 120 is restricted and a state in which the use of the user terminal 120 is not restricted, even if the evaluation value of the safety of the user terminal 120 suddenly fluctuates.

The output control unit 660 may adjust the sensitivity or responsiveness so that the length of time is greater than a predetermined value. The sensitivity or responsiveness of the judgment may be adjusted based on the safety evaluation value of the user terminal 120.

In the above embodiment, the details of the usage restriction by the output control unit 660 are described using as an example a case where the output control unit 660 restricts the use of the user terminal 120 by stopping the output of the original output image or preventing the viewing of the original output image. However, the usage restriction by the output control unit 660 is not limited to the above embodiment.

The output control unit 660 may restrict the acceptance of operational input from the user 22 by controlling the output of images by the display device 140. Restricting the acceptance of operational input may be an example of a usage restriction.

For example, when a process for deriving the degree of viewability or a process for evaluating the safety of the installation position and/or installation direction of the display area 142 is executed, the output control unit 660 displays a transparent screen in front of the desktop screen output by the display device 140. At this stage, the user 22 can input various instructions by mouse operation, keyboard operation, touch operation, etc. On the other hand, if the degree of viewability derived by the safety management unit 640 exceeds a predetermined level, a screen that may be viewed by a third party, a warning screen, etc., different from the output image described above is displayed.

Most of these screens are not transparent, and even if the user 22 operates a mouse on these screens, for example, the mouse operation cannot penetrate these screens and operate icons on the desktop located below these screens. This limits the acceptance of operational inputs from the user 22.

The state of some pixels of the screen that may be viewed by a third party may be specified as a transparent color. This allows instructions entered by user 22 to be accepted in the area where those pixels are arranged. The screen that may be viewed by a third party and the transparent screen may be smaller than the desktop screen. This allows instructions from user 22 to be accepted in a portion of the desktop screen.

In this embodiment, the storage unit 670 stores various types of information. In one embodiment, the storage unit 670 stores various types of information used in information processing executed in the user terminal 120. In another embodiment, the storage unit 670 stores various types of information generated by information processing executed in the user terminal 120. The user terminal 120 may transmit at least a portion of the information stored in the storage unit 670 to an external server periodically or when a predetermined event is detected.

The safety management unit 640 may be an example of a viewability derivation unit. The safety management unit 640 may be an example of a peripheral data acquisition unit or an information processing device.

Figure 7 shows an example of the internal configuration of the structural information acquisition unit 612. In this embodiment, the structural information acquisition unit 612 includes a three-dimensional point cloud acquisition unit 712, a distance image acquisition unit 714, a stereo image acquisition unit 716, and a two-dimensional image acquisition unit 718.

In this embodiment, the three-dimensional point cloud acquisition unit 712 acquires three-dimensional point cloud data of one or more objects arranged around the display area 142 from the surrounding data generation device 152. The three-dimensional point cloud data has information indicating the three-dimensional coordinate values of each of the one or more points. The three-dimensional point cloud data may include information indicating the three-dimensional coordinate values of each of the one or more points and information indicating the color of each point.

The distance image acquisition unit 714 acquires data (sometimes referred to as distance data or depth data) of a distance image (sometimes referred to as a depth image, depth map, etc.) of one or more objects arranged around the display area 142 from the surrounding data generation device 152. A distance image is a planar image that represents the depth or the distance to an object, and the value of each pixel in the planar image indicates the distance to the object corresponding to that pixel.

The stereo image acquisition unit 716 acquires stereo image data having as its subject one or more objects arranged around the periphery of the display area 142 from the peripheral data generation device 152. The stereo image acquisition unit 716 may generate a distance image of one or more objects arranged around the periphery of the display area 142 based on the stereo image.

The two-dimensional image acquisition unit 718 acquires, from the peripheral data generation device 152, data of a two-dimensional image having as its subject one or more objects arranged around the display area 142 (acquiring image data may be simply referred to as acquiring an image). The two-dimensional image acquisition unit 718 may acquire a two-dimensional image having as its subject one or more objects to which one or more markers are attached.

The three-dimensional point cloud acquisition unit 712 may be an example of a peripheral data acquisition unit. The distance image acquisition unit 714 may be an example of a peripheral data acquisition unit. The stereo image acquisition unit 716 may be an example of a peripheral data acquisition unit. The two-dimensional image acquisition unit 718 may be an example of a peripheral data acquisition unit.

Figure 8 shows an example of the internal configuration of the safety management unit 640. In this embodiment, the safety management unit 640 includes a data conversion unit 810 and an object position determination unit 820. In this embodiment, the data conversion unit 810 includes a marker position determination unit 812 and a point cloud generation unit 814. In this embodiment, the object position determination unit 820 includes a first calculation unit 830 and a second calculation unit 840. In this embodiment, the first calculation unit 830 includes a point cloud data acquisition unit 832, a distance calculation unit 834, and an angle calculation unit 836. The second calculation unit 840 includes an edge detection unit 842 and a shielding angle calculation unit 844.

In this embodiment, the data conversion unit 810 converts two-dimensional data into three-dimensional data. The data conversion unit 810 generates three-dimensional data of at least a portion of one or more objects arranged around the display area 142, for example, based on the stereo image acquired by the stereo image acquisition unit 716. The data conversion unit 810 generates three-dimensional data of at least a portion of one or more objects arranged around the display area 142, for example, based on the two-dimensional image acquired by the two-dimensional image acquisition unit 718.

The data conversion unit 810 outputs the generated three-dimensional data to the object position determination unit 820. This allows the object position determination unit 820 to construct a virtual space including a three-dimensional model of at least a part of one or more objects arranged around the display area 142. Furthermore, the object position determination unit 820 can determine the relative positional relationship between the display area 142 and the one or more objects in the virtual space.

In this embodiment, the details of the data conversion unit 810 are described using as an example a case where the data conversion unit 810 generates three-dimensional data of at least a part of one or more objects arranged around the display area 142 based on a two-dimensional image acquired by the two-dimensional image acquisition unit 718 when a predetermined type of marker is captured in the two-dimensional image. Examples of the predetermined type of marker include the above-mentioned AR marker, QR code (registered trademark), and barcode. In addition, for the purpose of simplifying the description, in this embodiment, an example of the data conversion process in the data conversion unit 810 is described using as an example a case where the display area 142 and the peripheral data generation device 152 are housed in the same housing 130, and the direction of the normal vector 242 of the display area 142 and the direction of the optical axis 252 of the peripheral data generation device 152 are approximately the same.

In this embodiment, the marker position determination unit 812 analyzes a two-dimensional image having as its subject one or more objects to which one or more markers are attached, and determines the positional relationship between at least a portion of the one or more markers and the display area 142. For example, the marker position determination unit 812 first analyzes the two-dimensional image acquired by the two-dimensional image acquisition unit 718, and executes processing to detect a predetermined type of marker. If one or more markers are not detected, the marker position determination unit 812 ends the processing. This ends, for example, the data conversion processing in the data conversion unit 810.

For example, the marker position determination unit 812 calculates three-dimensional coordinates in the camera coordinate system from the four vertices of each marker. For example, if the camera coordinate system moves, the distribution coordinates of the AR markers in the camera coordinate system at a past reference point in time may differ from the distribution coordinates of the AR markers in the camera coordinate system at the current point in time. Therefore, the marker position determination unit 812 may determine whether the camera coordinate system has moved. For example, the marker position determination unit 812 executes a process to find pairs of markers (sometimes called pair markers) that are the same instance between the camera coordinate system at a past reference point in time and the camera coordinate system at the current point in time through a brute force search. In addition, the marker position determination unit 812 determines whether the camera coordinate system has moved based on the position of the pair marker.

Specifically, the marker position determination unit 812 first calculates the transformation matrix between the two coordinate systems in a brute-force manner at each of the multiple time points at which the above determination is performed. This derives the transformation matrix that changes the marker coordinates. Next, the marker position determination unit 812 uses the derived transformation matrix to transform the actual current marker position into a past reference point, and evaluates whether it matches the displayed marker image and whether there is a large difference using Euclidean distance or the like. The marker position determination unit 812 repeatedly attempts the above process to determine the pair group with the smallest difference, or the pair group with the absolute value of the difference smaller than a predetermined value. This allows the marker position determination unit 812 to find the pair marker.

On the other hand, if one or more markers are detected by the image analysis described above, the marker position determination unit 812 extracts, for example, from the one or more detected markers, a marker whose distance from the display area 142 can be calculated. For example, if the dimensions of the marker are known, or if it is known that two or more markers of the same type are placed on approximately the same plane, the distance between the marker and the display area 142 can be calculated by analyzing a two-dimensional image in which the marker appears.

Next, the marker position determination unit 812 determines the positional relationship between the marker and the display area 142 for each or at least some of the markers whose distance from the display area 142 can be calculated. The positional relationship between the marker and the display area 142 may be the relative positional relationship between the marker and the display area 142 in the real world. The marker position determination unit 812 may calculate (i) the distance between the display area 142 and the representative point of each marker, and (ii) the angle between the normal vector 242 of the display area 142 and the direction from the representative point of the display area 142 toward the representative point of each marker.

In this embodiment, the point cloud generation unit 814 generates a point cloud around the position of the marker whose positional relationship with the display area 142 has been determined by the marker position determination unit 812. If information about the point cloud to be generated is embedded in the marker, the point cloud generation unit 814 generates a point cloud according to the information embedded in the marker. As described above, examples of the information about the point cloud include the dimensions of the generated point cloud in the world coordinate system.

If information about the point cloud to be generated is not embedded in the marker, the point cloud generation unit 814 may generate a point cloud in a predetermined range around the marker, for example. If information about the point cloud to be generated is not embedded in the marker and the identification information of the marker is embedded in the marker, the point cloud generation unit 814 may refer to a database that stores the identification information of the marker in association with information about the point cloud to be generated around the marker, and generate a point cloud according to the information stored in the database.

In this embodiment, the object position determination unit 820 analyzes the surrounding data acquired by the structural information acquisition unit 612 to determine the positional relationship between each of the one or more objects and the display area 142. The above positional relationship may be a relative positional relationship in the real world. Examples of procedures for determining the above relative positional relationship include a procedure using information processing in the first calculation unit 830 and a procedure using information processing in the second calculation unit 840.

In this embodiment, the first calculation unit 830 uses three-dimensional data of one or more objects arranged around the display area 142 to determine the relative positional relationship between the display area 142 and the one or more objects. The first calculation unit 830 may construct a virtual space including a three-dimensional model of at least a part of the one or more objects arranged around the display area 142. The first calculation unit 830 may determine the relative positional relationship between the display area 142 and the one or more objects in the virtual space.

In this embodiment, the point cloud data acquisition unit 832 acquires three-dimensional point cloud data for each of the one or more objects based on the surrounding data acquired by the structural information acquisition unit 612. In one embodiment, the point cloud data acquisition unit 832 acquires point cloud data acquired by the three-dimensional point cloud acquisition unit 712 or the distance image acquisition unit 714. In another embodiment, the point cloud data acquisition unit 832 acquires point cloud data generated by the data conversion unit 810.

In this embodiment, the distance calculation unit 834 calculates the distance between the representative point of the display area 142 and each of a plurality of points virtually arranged on the surface of each of the one or more objects represented by the three-dimensional point cloud data acquired by the point cloud data acquisition unit 832. An example of the representative point of the display area 142 is the representative point 240 described in relation to FIG. 2. Note that the representative point of the display area 142 may be any point on the surface of the display area 142 on the side where the output image is displayed, and is not limited to the representative point 240.

In this embodiment, the angle calculation unit 836 calculates the absolute value of the angle between the normal direction at the representative point of the display area 142 and the direction from the representative point of the display area 142 to each of a plurality of points arranged on the surface of each of the one or more objects. An example of the normal direction at the representative point of the display area 142 is the direction of the normal vector 242 at the representative point 240.

According to this embodiment, the relative distance between each of the one or more objects and the display area 142 is determined by information processing in the distance calculation unit 834. Furthermore, the relative direction between each of the one or more objects and the display area 142 is determined by information processing in the angle calculation unit 836. This determines the positional relationship between each of the one or more objects and the display area 142.

In this embodiment, the second calculation unit 840 determines the positional relationship between each of the one or more objects and the display area 142 without considering the distance between the object detected around the display area 142 by analyzing the surrounding data and the display area 142. Note that, in the process of detecting the above-mentioned objects, information indicating the distance between the object and the display area 142 may be used.

In this embodiment, for the purpose of simplifying the explanation, for example, a set of markers indicating the ends of each object in the approximately horizontal direction is attached to each of the one or more objects, and an example of a procedure in which the second calculation unit 840 analyzes the two-dimensional image acquired by the two-dimensional image acquisition unit 718 and determines the relative positional relationship between the display area 142 and one or more objects is explained using a case in which the markers are reflected in the two-dimensional image acquired by the two-dimensional image acquisition unit 718. Note that a person skilled in the art who has come into contact with the description of this specification can understand that the relative positional relationship between the display area 142 and one or more objects can be determined even if the markers are not reflected in the two-dimensional image, as long as the positions of both ends of the one or more objects in the approximately horizontal direction can be determined.

In this embodiment, the end detection unit 842 analyzes the two-dimensional image acquired by the two-dimensional image acquisition unit 718 and detects multiple markers included in the two-dimensional image. The end detection unit 842 detects one end and the other end of each of one or more objects in the approximately horizontal direction based on a combination of the detected multiple markers.

In this embodiment, the occlusion angle calculation unit 844 calculates, for each of the one or more objects, an occlusion angle, which is the angle between the direction from the representative point of the display area 142 toward one end of each object and the direction from the representative point of the display area 142 toward the other end of each object. The occlusion angle calculation unit 844 calculates the proportion of the viewing angle of the display area 142 that is occluded by the one or more objects based on the occlusion angles of each of the one or more objects. When the occlusion angle of the first object and the occlusion angle of the second object overlap, the occlusion angle calculation unit 844 calculates the proportion that is occluded by the one or more objects without counting the overlap.

The marker position determination unit 812 may be an example of a two-dimensional image acquisition unit or a marker position determination unit.

Figure 9 shows an example of information processing in the first calculation unit 830. The distance calculation unit 834 and the angle calculation unit 836, for example, refer to the point cloud data or distance data to obtain information about the vectors pointing from the representative point 250 to each of the n points P1 to Pn included in the point cloud 910 representing a single object.

In FIG. 9, vector 924 indicates a vector directed from representative point 250 to point P1. The magnitude of vector 924 is dp1, and the angle between vector 924 and optical axis 252 is θP1. Similarly, vector 926 indicates a vector directed from representative point 250 to point Pn. The magnitude of vector 924 is dpn, and the angle between vector 926 and optical axis 252 is θPn.

Figure 10 shows an example of information processing in the second calculation unit 840. In this embodiment, a marker 1022 is attached to one end of the shielding object 1020, and a marker 1024 is attached to the other end of the shielding object 1020. Similarly, a marker 1042 is attached to one end of the shielding object 1040, and a marker 1044 is attached to the other end of the shielding object 1040.

The shielding angle calculation unit 844 can calculate the shielding angle θA of the shielding object 1020 using the markers 1022 and 1024. The shielding angle calculation unit 844 can calculate the shielding angle θB of the shielding object 1040 using the markers 1042 and 1044.

Figure 11 shows an example of the internal configuration of the safety management unit 640. In this embodiment, the safety management unit 640 includes, for example, a derivation procedure determination unit 1130, an index value derivation unit 1140, and an evaluation unit 1150. In this embodiment, the derivation procedure determination unit 1130 includes, for example, an index setting unit 1132 and an evaluation setting unit 1134. In this embodiment, the index value derivation unit 1140 includes, for example, a first index derivation unit 1142, a second index derivation unit 1144, a third index derivation unit 1146, and a fourth index derivation unit 1148.

In this embodiment, the derivation procedure determination unit 1130 determines a derivation procedure, which is a procedure for the safety management unit 640 to derive the degree of viewability or the degree of safety. An example of the derivation procedure is the setting related to the evaluation of the safety of the user terminal 120 described above. The derivation procedure determination unit 1130 may determine the derivation procedure so that the derivation procedure is different between a case where the elapsed time since the start-up detection unit 622 detected the start-up satisfies a predetermined condition and a case where the elapsed time does not satisfy the predetermined condition. As described above, an example of the predetermined condition is a condition that the length of the elapsed time is smaller than a predetermined first threshold value.

In this embodiment, the indicator setting unit 1132 determines the content of the indicator setting described above. The indicator setting unit 1132 may determine the content of the indicator setting in the first setting described above, or may determine the content of the indicator setting in the second setting described above. Examples of the content of the indicator setting include the type of indicator used in the evaluation process, a function or learning model for deriving the value of the indicator, the type of explanatory variable of the function or learning model, and the parameter value of the function or learning model. Any known evaluation function may be used as the above function. The above function may be a weighted linear sum.

The indicator setting unit 1132 determines the type of one or more indicators to be used in the evaluation process, for example, based on (i) the state of the surroundings of the display area 142 indicated by the surrounding state information acquired by the surrounding state acquisition unit 610, and/or (ii) the current time or the time zone to which the current time belongs. The indicator setting unit 1132 may determine a function or learning model for deriving the value of the indicator depending on the type of indicator. The indicator setting unit 1132 may determine the parameter value of the function or learning model depending on the type of function or learning model.

In one embodiment, the index setting unit 1132 analyzes the image data output by the camera 154 to determine the type of index to be used in the evaluation process. The index setting unit 1132 may analyze the image data output by the camera 154 to determine whether or not to correct the index value or the evaluation value in the evaluation process. The index setting unit 1132 may determine the manner of the above correction or the degree of correction based on the analysis result.

For example, the index setting unit 1132 acquires image data output by the camera 154 from the structural information acquisition unit 612 or the physical property information acquisition unit 614. For example, when sunlight is directly incident on the imaging element of the camera 154, the camera 154 outputs an image in which at least a portion is whited out. However, it is difficult for the safety management unit 640 to accurately judge the surrounding situation based only on the whited out image. Even in such a case, the safety management unit 640 derives an evaluation value based on the surrounding data acquired by the surrounding data generation device 152 and/or various environmental information acquired by the environmental information acquisition unit 616, thereby improving the accuracy of deriving the evaluation value. In addition, the safety management unit 640 corrects the above-mentioned evaluation value in the whited out area, thereby improving the accuracy of deriving the evaluation value.

More specifically, the index setting unit 1132 determines whether or not a whiteout region is included in the image data output by the camera 154. If it is determined that a whiteout region is included in the image data output by the camera 154, the index setting unit 1132 may decide to calculate the evaluation value using an index that can be calculated without using the image data output by the camera 154, out of one or more indexes used in the evaluation process. The index setting unit 1132 may correct the index value so that the degree of viewability in the whiteout region is increased.

The index setting unit 1132 may distinguish between a whiteout caused by sunlight entering the imaging element and a whiteout caused by strong light being irradiated onto the surface of a white object. For example, the index setting unit 1132 calculates the positional relationship between the camera 154 and the sun from the GPS information and/or the date and time when the camera 154 outputs the above image data. The index setting unit 1132 may determine whether or not the whiteout in the image data is a whiteout caused by sunlight entering the imaging element from the positional relationship between the camera 154 and the sun.

In another embodiment, the index setting unit 1132 determines the type of index to be used in the evaluation process based on whether the surroundings of the installation position are dark or not. The index setting unit 1132 may determine whether to correct the index value or the evaluation value in the evaluation process based on whether the surroundings of the installation position are dark or not. The index setting unit 1132 may determine the manner of the above correction, or may determine the degree of correction.

For example, the index setting unit 1132 determines whether the surroundings of the installation position are dark based on the illuminance data output by the illuminance sensor 156. The index setting unit 1132 may determine whether the surroundings of the installation position are dark by considering environmental information other than illuminance, such as latitude and longitude, date and time, instead of or in addition to illuminance.

The correction modes described in relation to these embodiments are not particularly limited. For example, the degree of erroneous recognition can be improved by using the average value of multiple depth estimation results. Examples of multiple depth estimation results include a result of depth estimation using a front image (e.g., an image output by the camera 154) and a result of depth estimation using an image that has been inverted upside down or inverted left-right. In an ideal system, the depth estimated using the front image, the depth estimated using the image that has been inverted left-right, and the depth estimated using the image that has been inverted upside down are expected to be the same value. However, if an abnormality occurs in the image output by the camera 154, the depth estimated using the front image, the depth estimated using the image that has been inverted left-right, and the depth estimated using the image that has been inverted upside down are different. Therefore, by estimating the depth using the statistics of these values (e.g., the average value), the estimation accuracy is improved and the stability of the estimation process is improved.

In this embodiment, the evaluation setting unit 1134 determines the contents of the evaluation setting described above. The evaluation setting unit 1134 may determine the contents of the evaluation setting in the first setting described above, or may determine the contents of the evaluation setting in the second setting described above. Examples of the contents of the evaluation setting include the type of index used as an explanatory variable for deriving the evaluation value, a function or learning model for deriving the evaluation value, and the parameter values of the function or learning model. Any publicly known evaluation function can be used as the above function. The evaluation function may be a function that uses the index values of one or more indexes as explanatory variables and the evaluation value as a response variable. The above function may be a weighted linear sum.

The evaluation setting unit 1134 determines a function or learning model for deriving an evaluation value according to, for example, the type of index set by the index setting unit 1132. The evaluation setting unit 1134 determines a function or learning model for deriving an evaluation value based on, for example, (a) the type of index set by the index setting unit 1132 and (b) (i) the state of the surroundings of the display area 142 indicated by the surrounding state information acquired by the surrounding state acquisition unit 610, and/or (ii) the current time or the time zone to which the current time belongs. The above function may be a weighted linear sum. The evaluation setting unit 1134 determines a weight value to be set for each of one or more indexes used in the evaluation process based on, for example, (i) the state of the surroundings of the display area 142 indicated by the surrounding state information acquired by the surrounding state acquisition unit 610, and/or (ii) the current time or the time zone to which the current time belongs.

The evaluation setting unit 1134 may determine the weighting values to be set for one or more indicators used in the evaluation process based on the environmental information acquired by the environmental information acquisition unit 616. This allows the control device 170 to appropriately set the weighting values according to the surrounding environment of the user terminal 120.

For example, the evaluation setting unit 1134 determines a weighting value to be set for each of one or more indicators used in the evaluation process based on (i) the illuminance indicated by the illuminance information acquired by the environmental information acquisition unit 616, and/or (ii) the current time or the time zone to which the current time belongs. As described above, the current time or the time zone to which the current time belongs can be used as an indicator of the brightness around the user terminal 120. This allows the control device 170 to derive the above evaluation value based on the illuminance around the user terminal 120.

For example, there may be a large difference in the exposure quality of the image captured by the optical camera in a daytime environment and a nighttime environment. As a result, it is possible that the accuracy of deriving the evaluation value may decrease in certain environments. For example, in extremely dark places, the accuracy of deriving the evaluation value may decrease. Therefore, when there is a relatively high possibility that the accuracy of deriving the evaluation value may decrease, the risk of information leakage can be further reduced by deriving the evaluation value so that the above-mentioned usage restriction process is executed. Examples of cases where there is a relatively high possibility that the accuracy of deriving the evaluation value may decrease include when the illuminance indicated by the illuminance information is smaller than a predetermined value, and when the illuminance predicted based on the current time or the time zone to which the current time belongs is smaller than a predetermined value.

In this embodiment, the index value derivation unit 1140 derives the values (sometimes referred to as index values) of one or more indicators related to the degree of viewability described above. The index value derivation unit 1140 derives the index value of each indicator according to the content of the index setting determined by the index setting unit 1132. For example, the index value derivation unit 1140 derives index values for the types of indicators determined by the index setting unit 1132.

(Index value of the first index)
In this embodiment, the first index derivation unit 1142 derives the value of the first index described above. As described above, the first index indicates the degree to which object movement or light fluctuation is detected around the installation position of the display area 142. For example, in video data captured around the installation position of the display area 142, if the light intensity of a pixel corresponding to a specific area in the real world fluctuates significantly, there is a high possibility that an object has moved, deformed, or deteriorated in the area. In addition, by analyzing the microscopic light intensity fluctuation in the pixel corresponding to the specific area, it can be determined, for example, whether natural light is irradiated to the area, whether natural light is transmitted through the area, etc.

In this embodiment, the first index derivation unit 1142, for example, analyzes multiple frames included in the video data acquired by the physical property information acquisition unit 614, and derives the variation in light intensity between frames. The first index derivation unit 1142 derives the value of the first index based on the variation in the light intensity. The value of the first index is derived, for example, based on statistics of the variation in the light intensity.

The first index derivation unit 1142 may derive the first index by changing the conditions for deriving the first index for the same video data. This allows multiple types of first indexes to be derived from the same video data.

Examples of conditions for deriving the first index include (i) the shape and/or size of the calculation unit when each frame of the video is divided into multiple predetermined regions (sometimes referred to as calculation units), (ii) the positions in the video data of the multiple frames used to derive the first index and the number of such frames, and (iii) the elements used to derive the first index when the color of each pixel is specified by multiple elements. When each of the multiple frames constituting the video data is a color image, the color of each pixel constituting the color image is specified by the respective values of multiple elements for specifying the color. For example, when each frame is a color image in RGB format, the color of each pixel in each frame is specified by an R value, which is the value of the R element, a G value, which is the value of the G element, and a B value, which is the value of the B element.

The first index derivation unit 1142 outputs, for example, data (sometimes referred to as variation data) in which identification information for identifying each of one or more calculation units is associated with the value of the first index of each calculation unit. The first index derivation unit 1142 may output data (sometimes referred to as variation image data) in which the identification information of each of the multiple calculation units and the value of the first index of each calculation unit are stored in an image format. In this case, the identification information of each calculation unit indicates, for example, the x and y coordinates in the image. The variation image data may be an example of variation data.

In one embodiment, the first index derivation unit 1142 outputs multiple pieces of variation data having different shapes and/or sizes of calculation units. For example, the first index derivation unit 1142 uses the same frame group extracted from the same video data to output (i) variation image data generated based on the value of the first index derived for each square calculation unit consisting of 9 pixels, and (ii) variation image data generated based on the value of the first index derived for each square calculation unit consisting of 1024 pixels. Note that the shape of the calculation unit and the number of pixels constituting the calculation unit are not limited to the above example.

This allows the evaluation unit 1150 to apply different evaluation criteria to each of the multiple dynamic image data to evaluate the degree of safety of the display area 142. As a result, the degree of safety of the display area 142 can be evaluated with greater accuracy.

For example, in a case where the value of the first index is determined such that the greater the risk of information leakage, the greater the value of the first index, if either the evaluation criterion "if the value of the first index derived for each square calculation unit consisting of 9 pixels is greater than a threshold, the evaluation of the safety of the display area 142 is reduced" or the evaluation criterion "if the value of the first index derived for each square calculation unit consisting of 1024 pixels is greater than a threshold, the evaluation of the safety of the display area 142 is reduced" is applied, it is difficult to set the threshold. As a result, it is also difficult to improve the accuracy of the evaluation.

In contrast to this, when the evaluation criterion that "if the value of the first index derived for each square calculation unit consisting of 9 pixels is greater than the first threshold value, or if the value of the first index derived for each square calculation unit consisting of 1024 pixels is greater than the second threshold value, the evaluation of the safety of the display area 142 is reduced" is applied, it becomes easier to set the threshold value compared to the case where either one of the above evaluation criteria is applied. In addition, by appropriately setting the threshold value, the accuracy of the evaluation is improved.

In another embodiment, the first index derivation unit 1142 outputs a plurality of variation data in which the positions and/or the number of frames used to derive the first index are different. For example, the first index derivation unit 1142 uses the same frame group extracted from the same video data to output (i) one variation image data generated based on the value of the first index derived using all frames included in the frame group, and (ii) one or more variation image data generated based on the value of the first index derived using some of the frames included in the frame group.

For example, the first index derivation unit 1142 extracts the 150th to 399th frames included in the video data as a frame group used to derive the first index. The first index derivation unit 1142 derives the value of the first index for each calculation unit using the 150th to 399th frames, and outputs basic variation image data. The first index derivation unit 1142 also derives the value of the first index for each calculation unit using the 150th to 199th frames, and outputs another set of variation image data. Similarly, the first index derivation unit 1142 derives the value of the first index for each calculation unit using the 200th to 249th frames, the 250th to 299th frames, the 300th to 349th frames, and the 350th to 399th frames, and outputs another set of nine variation image data.

The smaller the number of frames (sometimes referred to as target frames) used in the calculation, the greater the impact that a sudden event that occurs during the imaging period of the target frames will have on the value of the first index derived using the target frames. As described above, the frequency distribution of artificial light fluctuations often has a sharper shape than the frequency distribution of natural light fluctuations. The frequency distribution of the above fluctuations corresponds to the frequency distribution of light fluctuations described above.

As a result, the evaluation unit 1150 can determine the proportion of natural light contained in the light detected in each pixel or each calculation unit by, for example, comparing the frequency distribution of the fluctuation when the number of target frames is small with the frequency distribution of the fluctuation when the number of target frames is large. For example, if (i) the absolute value of the difference between the variance, standard deviation, or coefficient of variation of the above frequency distribution is greater than a predetermined value, or (ii) the ratio of the variance, standard deviation, or coefficient of variation of the frequency distribution of the fluctuation when the number of target frames is small to the variance, standard deviation, or coefficient of variation of the frequency distribution of the fluctuation when the number of target frames is large is less than a predetermined value, the evaluation unit 1150 determines that the proportion of natural light is high.

In yet another embodiment, the first index derivation unit 1142 outputs multiple pieces of variation data in which the color elements used to derive the first index are different. For example, when each frame is a color image in RGB format, the first index derivation unit 1142 uses the same frame group extracted from the same video data to output (a) variation image data generated based on the value of the first index derived using at least two of the R value, the G value, and the B value, and (b) (i) another variation image data generated based on the value of the first index derived using a combination different from the above combination, or (ii) any one of the R value, the G value, and the B value.

The inventors have found that the frequency at which light fluctuations are emphasized varies depending on the type of subject or the condition of the subject's surface. For example, the variation in the R value of light reflected from the surface of human skin is greater than the variation in the G or B value of that light. Also, the variation in the R value of light reflected from the surface of human skin is greater than the variation in the R value of light reflected from the surface of a plant. Furthermore, since the sensitivity characteristics of human vision and the sensitivity characteristics of an imaging device differ, the display device 140 outputs an image that has been color calibrated according to the human visual characteristics. Therefore, the balance of the R, G, and B values is different between light from an object displayed on a display device such as a display or screen and light from an object that actually exists.

As a result, the first index derivation unit 1142 derives the first index based on, for example, one or two of the R value, G value, and B value, and the evaluation unit 1150 can improve the accuracy of detecting a specific type of object (for example, a human). In addition, the evaluation unit 1150 can improve the detection accuracy of a specific type of object (for example, a human) by comparing, for example, a first index derived based on a first combination of the R element, the G element, and the B element with a first index derived based on a second combination of the R element, the G element, and the B element. The first combination and the second combination have different RGB elements that constitute each combination. The first combination may be composed of one of the R element, the G element, and the B element, and the second combination may be composed of one of the R element, the G element, and the B element.

The first index derivation unit 1142 may derive the first index by changing at least two of the conditions for deriving the first index described above. The first index derivation unit 1142 may perform processing for correcting the movement of the camera 154 itself on the video data acquired by the physical property information acquisition unit 614, and then execute processing for deriving the first index. Details of the information processing in the first index derivation unit 1142 will be described later.

(Index value of the second index)
In this embodiment, the second index derivation unit 1144 derives an index value of the above-described second index. As described above, the second index indicates the arrangement of obstructions present around the installation position of the display area 142. For example, the second index indicates the degree to which the obstructions block the line of sight of a third party to the display area 142. For example, the second index derivation unit 1144 analyzes the surrounding data acquired by the structural information acquisition unit 612 to derive the degree to which each of the obstructions arranged around the display area 142 blocks the line of sight of a third party.

The second index derivation unit 1144 may derive the degree to which each of the obstructions arranged around the display area 142 obstructs the line of sight of a third party based on the point cloud data or distance data acquired by the structural information acquisition unit 612. The second index derivation unit 1144 may calculate the distance between the subject and the camera 154 from the image data acquired by the structural information acquisition unit 612 to generate distance data, and use the distance data to derive the degree to which each of the obstructions arranged around the display area 142 obstructs the line of sight of a third party. The second index derivation unit 1144 may use the three-dimensional virtual space constructed by the object position determination unit 820 to derive the degree to which each of the obstructions arranged around the display area 142 obstructs the line of sight of a third party.

The second index derivation unit 1144 derives the value of the second index based on, for example, the distance between each of the multiple points calculated by the distance calculation unit 834 and the representative point of the display area 142. The second index derivation unit 1144 may (i) derive the value of the second index for each of the multiple points, or (ii) after identifying each of one or more obstructions, derive the value of the second index for each obstruction based on the distance between multiple points on each obstruction or a representative point of each obstruction and a representative point of the display area 142.

The second index derivation unit 1144 derives the value of the second index based on, for example, the absolute value of the angle for each of the multiple points calculated by the angle calculation unit 836. The second index derivation unit 1144 may (i) derive the value of the second index for each of the multiple points, or (ii) identify each of the one or more obstructions and then derive the value of the second index for each obstruction.

The second index derivation unit 1144 derives the value of the second index based on, for example, the distance for each of the multiple points calculated by the distance calculation unit 834 and the absolute value of the angle for each of the multiple points calculated by the angle calculation unit 836. The second index derivation unit 1144 may (i) derive the value of the second index for each of the multiple points, or (ii) identify each of one or more obstructions and then derive the value of the second index for each obstruction.

(Index value of the third index)
In this embodiment, the third index derivation unit 1146 derives the index value of the above-described third index. As described above, the third index indicates the status of the surrounding environment of the installation position of the display area 142. The third index derivation unit 1146 derives the value of the third index based on, for example, the status of the surrounding environment indicated by the environmental information acquired by the environmental information acquisition unit 616.

In one embodiment, the third index derivation unit 1146 derives the value of the third index based on the illuminance indicated by the illuminance information acquired by the environmental information acquisition unit 616. The third index derivation unit 1146 may derive the value of the third index such that the value of the third index increases as the illuminance increases.

In another embodiment, the third index derivation unit 1146 may determine the visibility at the installation position of the display area 142 based on the state of the surrounding environment indicated by the environmental information acquired by the environmental information acquisition unit 616. For example, the third index derivation unit 1146 refers to a database that stores the state of the surrounding environment and the visibility in association with each other, and determines the visibility based on the environmental information acquired by the environmental information acquisition unit 616. The third index derivation unit 1146 may derive the value of the third index based on the visibility.

For example, the greater the visibility, the easier it is for the user of the information processing terminal to notice the presence of a third party in the vicinity. On the other hand, the greater the visibility, the easier it is for a third party to view information output by the information processing terminal. Therefore, the third index derivation unit 1146 may derive the value of the third index, for example, so that the greater the visibility, the greater the value of the third index. The third index derivation unit 1146 may derive the value of the third index so that the greater the visibility, the smaller the value of the third index.

(Index value of the fourth index)
In this embodiment, the fourth index derivation unit 1148 derives the index value of the above-mentioned fourth index. As described above, the fourth index indicates the risk of information leakage according to the type of object or event. The fourth index derivation unit 1148 derives the value of the fourth index based on, for example, the attribute information acquired by the attribute information acquisition unit 618. The fourth index derivation unit 1148 determines the value of the above-mentioned fourth index based on the type and/or attribute of the object or event actually observed, for example, by referring to a database in which the type and/or attribute of the object or event is associated with a value indicating the degree of possibility that a third party will view the output image from the position where the object or event is observed, for each of one or more objects or events.

As described above, the attribute information acquisition unit 618 detects an object by analyzing the image data output by the environmental sensor 150. For example, the attribute information acquisition unit 618 detects the contour of an object from within the image and determines the type and/or attributes of the detected object. However, when detecting an object using contour detection technology, it is difficult to determine whether the detected object is displayed on a display device such as a display or screen, or whether the object actually exists in real space.

In one embodiment, the fourth index derivation unit 1148 may determine or change the type and/or attribute of the detected object based on the position where the object is detected in the image. For example, when a first object and a second object are detected, and the position of the second object is inside the contour of the first object, and the type or attribute of the first object is a display device such as a display or screen, the fourth index derivation unit 1148 determines that the type and/or attribute of the second object is an image displayed on the display device. Furthermore, the fourth index derivation unit 1148 determines the value of the fourth index based on the determination result that the type and/or attribute of the second object is an image displayed on the display device. This allows the fourth index derivation unit 1148 to appropriately determine the value of the fourth index.

In another embodiment, the fourth index derivation unit 1148 may determine or change the type and/or attribute of the detected object based on the color of the detected object. Since the sensitivity characteristics of human vision and the sensitivity characteristics of the imaging device are different, the display device outputs an image that has been color calibrated according to the human visual characteristics. Therefore, the color detected by the camera 154 differs between when an object is displayed on the display device and when the object actually exists in real space. This allows the fourth index derivation unit 1148 to determine or change the type or attribute of the detected object based on the color of the detected object. Furthermore, the fourth index derivation unit 1148 determines the value of the fourth index based on the determined or changed type or attribute of the object. This allows the fourth index derivation unit 1148 to appropriately determine the value of the fourth index.

In this case, the fourth index derivation unit 1148 may cooperate with the first index derivation unit 1142 to determine or change the type and/or attribute of the detected object. As described above, the first index derivation unit 1142 can derive a first index based on any combination of the R element, the G element, and the B element. In addition, the first index derivation unit 1142 can output a first index derived based on a first combination of the R element, the G element, and the B element, and a first index derived based on a second combination of the R element, the G element, and the B element. The fourth index derivation unit 1148, for example, refers to a database in which (a) the type and/or attribute of the object or event and (b) (i) information indicating the value of the first index derived by a specific combination of the R element, the G element, and the B element, or (ii) information indicating the comparison result between the value of the first index derived based on a first combination of the R element, the G element, and the B element and the value of the first index derived based on a second combination of the R element, the G element, and the B element are associated with each other, and determines the type and/or attribute of the detected object based on the first index output by the first index derivation unit 1142. Based on the determined type or attribute of the object, the fourth index derivation unit 1148 determines the value of the fourth index. This allows the fourth index derivation unit 1148 to appropriately determine the value of the fourth index.

(Evaluation of the degree of accessibility)
In this embodiment, the evaluation unit 1150 evaluates the degree of viewability based on the value of one or more indexes derived by the index value derivation unit 1140. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability.

(comprehensive evaluation)
In this embodiment, the evaluation unit 1150 derives the above evaluation value in accordance with the contents of the evaluation setting determined by the evaluation setting unit 1134. The evaluation unit 1150 may derive the above evaluation value by inputting index values of one or more types of indexes determined by the index setting unit 1132 into an arbitrary function or learning model.

For example, the evaluation unit 1150 derives a weighted linear sum of the index values of one or more types of indexes determined by the index setting unit 1132, based on the index values of the one or more types of indexes and the weight values set for each of the one or more indexes. The evaluation unit 1150 outputs the weighted linear sum as the evaluation value. This allows the evaluation unit 1150 to comprehensively evaluate the degree of viewability.

The method of comprehensively evaluating the degree of viewability is not limited to the above embodiment. In another embodiment, the evaluation unit 1150 may use a script including weighted logic calculation, conditional branching, and repetitive processing to comprehensively evaluate the degree of viewability. In yet another embodiment, the evaluation unit 1150 may use neural network technology to comprehensively evaluate the degree of viewability. For example, the evaluation unit 1150 derives one or more digest values that express the characteristics of the various index values based on the various index values output by the index value derivation unit 1140. The evaluation unit 1150 may derive the digest value using a general-purpose script including various function processes, convolution processes, and the like. The evaluation unit 1150 prepares learning data that receives one or more digest values as input and outputs the result of the comprehensive evaluation, and executes a learning process for an arbitrary learning model. The evaluation unit 1150 may use a learned learning model to comprehensively evaluate the degree of viewability.

(Evaluation based on the first indicator)
In this embodiment, the evaluation unit 1150 evaluates the degree of viewability based on the value of the first index. For example, the evaluation unit 1150 evaluates the safety of the user terminal 120 by evaluating the degree to which object movement or light flicker is detected around the installation position of the user terminal 120. For example, the evaluation unit 1150 evaluates the safety of the user terminal 120 higher the smaller the degree to which object movement or light flicker is detected around the installation position of the user terminal 120. Details of the procedure for evaluating the degree of viewability based on the value of the first index will be described later.

(Evaluation based on the second indicator)
In the present embodiment, the evaluation unit 1150 evaluates the degree of viewability based on the value of the second index. For example, the evaluation unit 1150 derives the evaluation value based on the positional relationship determined by the object position determination unit 820.

As described above, the degree of viewability of the output image is used as an index of the degree of safety against information leakage. The greater the degree of viewability of the output image, the more difficult it is to block a third party's line of sight with an obstruction. In other words, the greater the degree of viewability of the output image, the lower the degree of safety against information leakage.

In one embodiment, when comparing a case where the distance between the display area 142 and the obstruction is large with a case where the distance between the display area 142 and the obstruction is small, it is easier for a third party to get in between the display area 142 and the obstruction when the distance between the display area 142 and the obstruction is large. Also, when the distance between the display area 142 and the obstruction is large, it becomes difficult for the user 22 of the user terminal 120 to discover that a third party is peeking at the display area 142. In other words, it can be considered that the greater the distance between the display area 142 and the obstruction, the greater the degree of viewability.

On the other hand, if we assume that a third party is present between the display area 142 and an obstruction and is peeking at the output image, the greater the distance between the third party and the display area 142, the more difficult it will be for the third party to understand the contents of the output image. In other words, the greater the distance between the display area 142 and the obstruction, the less likely it is that the image will be viewable.

According to another embodiment, when comparing a case where a third party views the display area 142 from an angle with a case where the third party views the display area 142 from the front, the third party can more easily understand the contents of the output image when the third party views the display area 142 from the front. On the other hand, the user 22 is present in front of the display area 142. Therefore, it is considered to be relatively difficult for a third party to peek at the display area 142 from in front of the display area 142 and behind the user 22.

Therefore, the method of deriving the degree of viewability can be appropriately set depending on the purpose and the situation. In one embodiment, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability such that (i) the degree of viewability is greater for objects arranged in positions farther from the display area 142, and/or (ii) the degree of viewability is smaller for objects arranged in positions closer to the front of the display area 142. In another embodiment, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability such that (i) the degree of viewability is greater for objects arranged in positions closer to the display area 142, and/or (ii) the degree of viewability is greater for objects arranged in positions closer to the front of the display area 142.

(Embodiment in which the degree of viewability is derived based on the first calculation unit 830)
In one embodiment, the second index derivation unit 1144 calculates a value of a suppression index for each of the multiple points, the suppression index being an index indicating the degree to which each point suppresses the viewing of the output image by a third party, based on the distance for each of the multiple points calculated by the distance calculation unit 834. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability by summing up the suppression index values for each of the multiple points. The suppression index may be an example of the second index.

In another embodiment, the second index derivation unit 1144 calculates a suppression index value for each of the multiple points, which is an index indicating the degree to which each point suppresses viewing of the output image by a third party, based on the absolute value of the angle for each of the multiple points calculated by the angle calculation unit 836. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability by summing up the suppression index values for each of the multiple points. The suppression index may be an example of the second index.

In yet another embodiment, the second index derivation unit 1144 calculates a suppression index value for each of the multiple points based on the distance for each of the multiple points calculated by the distance calculation unit 834 and the absolute value of the angle for each of the multiple points calculated by the angle calculation unit 836. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability by summing up the suppression index values for each of the multiple points. The suppression index may be an example of the second index.

(An embodiment in which the degree of viewability is derived based on the calculation result of the second calculation unit 840)
The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability based on the occlusion angles of each of the one or more objects calculated by the occlusion angle calculation unit 844. For example, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability based on the proportion of the viewing angle of the display area 142 that is occluded by one or more objects. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability such that the greater the occlusion proportion, the smaller the degree of viewability.

(Evaluation based on the third indicator)
In this embodiment, the evaluation unit 1150 evaluates the degree of viewability based on the value of the third index. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability based on, for example, the value of the third index. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability based on the state of the surrounding environment indicated by the environmental information acquired by the environmental information acquisition unit 616. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability such that the greater the visibility, the smaller the degree of viewability. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability such that the smaller the visibility, the smaller the degree of viewability.

For example, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability such that the greater the illuminance, the smaller the degree of viewability. The evaluation unit 1150 may also derive an evaluation value indicating the degree of viewability such that the lower the illuminance, the smaller the degree of viewability.

(Evaluation based on the fourth indicator)
In this embodiment, the evaluation unit 1150 evaluates the degree of viewability based on the value of the fourth index. The evaluation unit 1150 derives an evaluation value indicating the degree of viewability based on, for example, the value of the fourth index. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability based on the type or attribute of an object or event indicated by the attribute information acquired by the attribute information acquisition unit 618. The evaluation unit 1150 may output the value of the fourth index as an evaluation value indicating the degree of viewability.

Figure 12 shows an example of information processing in the safety management unit 640. In this embodiment, an example of processing for evaluating the safety of the user terminal 120 is described, taking as an example a case where the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generation device 152 or the camera 154. More specifically, an example of processing for the safety management unit 640 to evaluate the safety of the user terminal 120 using an image 1200 is described. Image 1200 shows an example of an image captured by the peripheral data generation device 152 or the camera 154 of the user terminal 120 placed at position A in the work room 300.

As described above, in this embodiment, the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generating device 152 or the camera 154. Therefore, the size of the area equivalent to the viewing angle of the display area 142 (represented as unit image 1240 in the figure) is smaller than the size of the image 1200. The safety management unit 640 may evaluate safety while shifting the center 1242 of the image (represented as unit image 1240 in the figure) of a size equivalent to the viewing angle of the display area 142 in the left-right direction or up-down direction along a predetermined dotted line 1250. This allows the safety management unit 640 to evaluate safety for multiple installation directions using a single image 1200.

The position of the center 1242 of the unit image 1240 in the image 1200 corresponds to the installation direction of the display area 142. In one embodiment, the shape and size of the unit image 1240 are adjusted according to the position of the unit image 1240 in the image 1200. For example, the shape and size of the unit image 1240 are adjusted so that the larger the distance between the position of the center 1242 of the unit image 1240 and the position of the center of the image 1200, the larger the size of the unit image 1240 becomes and the greater the degree of deformation of the shape. This allows the safety management unit 640 to evaluate the safety of the user terminal 120 in multiple patterns with different installation directions of the display area 142 based on a single image data or a single three-dimensional data generated by the surrounding data generation device 152 or the camera 154.

In another embodiment, the safety management unit 640 may evaluate the safety of the user terminal 120 in multiple patterns in which the installation direction of the display area 142 is different, assuming that the shape and size of the area corresponding to the viewing angle of the display area 142 are the same regardless of the installation direction of the display area 142. This reduces the amount of calculation.

(An example of another embodiment)
In the present embodiment, an example of the image 1200 and the safety management unit 640 has been described by taking as an example a case where the viewing angle of the display area 142 is smaller than the viewing angle of the peripheral data generating device 152 or the camera 154. However, as described above, in other embodiments, the viewing angle of the display area 142 may be larger than the viewing angle of the peripheral data generating device 152 or the camera 154.

For example, the safety management unit 640 can acquire information beyond the viewing angle of the peripheral data generation device 152 or the camera 154 by combining multiple pieces of information related to the surrounding environment of the installation position of the display area 142 acquired by the peripheral data generation device 152 or the camera 154. For example, the safety management unit 640 acquires multiple images of the periphery of the installation position of the display area 142, where at least one of the imaging positions and the imaging direction is different. The multiple images may be images captured by a single peripheral data generation device 152, or may be images captured by multiple peripheral data generation devices 152. The safety management unit 640 synthesizes the multiple images to generate a single image (sometimes referred to as a combined image). As a result, a combined image in which multiple images are synthesized may be generated as another example of the image 1200.

The imaging direction may be expressed in two dimensions or three dimensions. The safety management unit 640 may analyze the combined image in a similar manner to the image 1200 in the above embodiment, thereby calculating the position, shape, and size of the area corresponding to the viewing angle of the display area 142 when the installation mode of the display area 142 is changed. Examples of the installation mode include the installation position, installation direction, and roll direction.

The specific procedure for synthesizing multiple images to generate a combined image is not particularly limited. The combined image may be, for example, a panoramic image in which multiple images are panorama-synthesized.

The safety management unit 640 may generate a three-dimensional virtual space, described below, based on multiple two-dimensional images or a combined image. However, if the overlapping portion of the two images to be combined is less than about 20-30% of each image, the number of features required to accurately combine the two images is small, making it difficult to accurately combine the two images. In particular, when a three-dimensional virtual space is generated based on multiple two-dimensional images or a combined image, a shift of a few pixels on the two-dimensional image may lead to a large shift in the three-dimensional coordinates. For example, the above-mentioned shift may occur in the two-dimensional image if the image is captured out of focus, in a dark place, or in backlight.

Therefore, when the safety management unit 640 generates a three-dimensional virtual space based on a plurality of two-dimensional images or a combined image, it is preferable that the control device 170 executes a process for reducing or correcting the combination error. Examples of the process for reducing or correcting the combination error include (a) a process for estimating the amount of movement of the peripheral data generation device 152 or the camera 154 while the peripheral data generation device 152 or the camera 154 is continuously capturing a plurality of images (for example, while capturing a moving image), and sequentially combining the images while taking into account the amount of movement, and (b) (i) a process for calculating the two-dimensional coordinates of each feature point when the three-dimensional model or three-dimensional point cloud is displayed two-dimensionally after combining the coordinate systems of the spatial structure in the three-dimensional virtual space, (ii) a process for determining whether the two-dimensional coordinates of each feature point displayed two-dimensionally match the position of each feature point in the two-dimensional image before combination, and (iii) a process for correcting the combination position so that the error between each feature point is reduced when it is determined that all or a part of the above feature points do not match.

The specific procedure for determining whether the two-dimensional coordinates of each feature point displayed in two dimensions match the position of each feature point in the two-dimensional image before merging is not particularly limited. For example, for structural data collected in two coordinate systems, a three-dimensional transformation matrix is calculated that matches the coordinates of the two transformation matrices. The collected information is then merged into one coordinate system using the three-dimensional transformation matrix. To calculate this transformation matrix, it is necessary to find multiple identical feature points in the two coordinate systems. Therefore, for example, the correct answer is inferred inductively by brute force for the pairing of the feature points. For example, a probabilistic method is used for the above inference.

As described below, when multiple markers are arranged in real space, the above pairing can be made easier by making the value of each marker unique. Even if the values of multiple markers are the same, the above brute force inference process can be performed by finding unique characteristics from the correlation between the roll directions and/or positions of paired markers.

In this embodiment, the details of the information leakage prevention system 100 have been described using as an example a case in which the presence or absence of a third party is determined around the installation position of the display area 142 by detecting light fluctuations. However, the information leakage prevention system 100 is not limited to this embodiment.

In another embodiment, the control device 170 may derive an index value of an index indicating the degree of clutter of objects in the vicinity by using the degree of light fluctuation detected as a precaution against hidden cameras. The control device 170 may determine safety based on the degree of clutter of objects. An example of a case where the degree of clutter is high is a state in which a state in which a large amount of light fluctuation is seen around. When the degree of clutter is high, for example, it is difficult to reveal that a hidden camera is installed. Therefore, for example, the control device 170 may calculate an index value such that the degree of safety decreases as the degree of clutter increases. By introducing a judgment based on the degree of clutter, preventive judgment is possible, and the number of places with a high degree of safety increases. On the other hand, for example, the location where the terminal can be used may be restricted simply by the presence of books, miscellaneous goods, etc. scattered at the installation position of the display area 142. Therefore, when an evaluation value of the safety of the user terminal 120 is derived based on the index value indicating the degree of clutter, the above-mentioned function of adjusting the sensitivity and/or responsiveness of the judgment may be used.

Figure 13 shows an example of information processing in the control device 170. Using Figure 13, an example of a procedure in which the evaluation unit 1150 evaluates the degree of viewability based on the value of the first index is described. As described above, in this embodiment, the evaluation unit 1150 evaluates the degree to which object movement or light fluctuation is detected in the vicinity of the installation position of the user terminal 120, thereby evaluating the safety of the user terminal 120. In this embodiment, an example of a process in which the control device 170 acquires the imaging condition data 1310 and the video data 1320 from the camera 154 and generates the dynamic image data 1340 is described.

In this embodiment, the control device 170 acquires video data generated by the camera 154. The control device 170 analyzes the pixel values of each pixel constituting each of the multiple frames included in the video data to derive the fluctuation in the intensity of light reaching the camera 154 during a specific period of time.

The variation in the intensity of light from the subject of the camera 154 reflects (i) the variation in the refractive index of the space between the light source and the subject, (ii) the variation of an object disposed in the space between the light source and the subject, (iii) the variation in the refractive index of the space between the subject and the camera 154, and/or (iv) the variation of an object disposed in the space between the subject and the camera 154. The light from the subject may be reflected light reflected from the surface of the subject, transmitted light transmitted through the subject, or emitted light emitted by the subject. Examples of the variation of the object include the variation of at least one of the position, size, shape, and optical characteristics of the object. Examples of the optical characteristics include the refractive index, transmittance, and reflectance.

Therefore, the magnitude of the above-mentioned light intensity fluctuations can be used as an index of the degree to which object movement or light fluctuations are detected around the camera 154. The magnitude of the above-mentioned light intensity fluctuations can also be used as an index of the degree of viewability described above. In other words, when the light intensity fluctuations in a particular area in the frame are small, this indicates that a small object that moves, transmits light, reflects light, etc. is located at a position corresponding to that area in the real world.

For example, the control device 170 (a) divides each frame of the video into a number of predetermined regions (which may be referred to as calculation units), and (b) for each of the multiple calculation units, derives the value of an index (which may be referred to as a variation index) that indicates the variation in color and/or brightness of the region between multiple frames based on the pixel values of the pixels included in each calculation unit. Examples of the variation index include at least one of the variance, standard deviation, and coefficient of variation, as well as values obtained by dividing these by a reference value. The variation index may be an example of a first index.

Each of the multiple calculation units includes one or more pixels. The greater the number of pixels included in each calculation unit, the less the load for calculating the light intensity fluctuations described above, and the faster the calculation speed. On the other hand, the smaller the number of pixels included in each calculation unit, the greater the accuracy or resolution.

At least two computational units included in a single frame may differ in at least one of size and shape. For example, the size of a computational unit displayed near the center of the screen is different from the size of a computational unit displayed on the periphery of the screen. For example, the shape of a computational unit displayed near the center of the screen is different from the shape of a computational unit displayed on the periphery of the screen.

(Configuration of imaging condition data 1310)
In the present embodiment, the image capturing condition data 1310 includes information indicating an image capturing position. The image capturing condition data 1310 may further include information indicating at least one of an image capturing direction and a focal length of the camera 154.

If the focal length f of the camera 154 is known, the control device 170 can calculate the angle between the direction of the optical axis of the camera 154 and the direction from the representative point 250 of the camera 154 toward the position Q corresponding to the position q in the real world, based on the positional relationship between the specific position q in the image and the position o of the optical axis in the image. In the camera coordinate system of the camera 154, the coordinates of the position o are expressed as (0, 0, f), for example, the coordinates of the position q are expressed as (x, y, f), and the coordinates of the position Q are expressed as (X, Y, Z), for example.

Note that, if the installation position and installation direction of the display area 142 have been determined, the imaging condition data 1310 may not include information indicating the imaging position and information indicating the imaging direction. For example, if the user 22 installs the user terminal 120 in a specific position and direction, and the degree of viewability at that position and direction is verified, the imaging condition data 1310 may not include information indicating the imaging position and information indicating the imaging direction. Also, if the focal length f of the camera 154 is known, the imaging condition data 1310 may not include information indicating the focal length of the camera 154.

(Configuration of video data 1320)
In this embodiment, the video data 1320 has a plurality of frames including a frame 1322, a frame 1324, a frame 1332, and a frame 1334. Each frame included in the video data 1320 may be a two-dimensional image.

The data structure of a frame constituting the video data 1320 includes, for example, identification information of each pixel constituting the frame, information indicating the position of each pixel in the frame, and information indicating the brightness of each pixel. For example, data of the i-th pixel NP _i included in the frame includes, for example, xi indicating the position in the x direction on the image plane of the frame, yi indicating the position in the y direction on the image plane, and a pixel value indicating brightness. The brightness may be represented by luminance or lightness. When the frame is a color image expressed in RGB format, the pixel value indicating brightness includes, for example, information indicating the luminance value of each of RGB. In other words, the color of each pixel can be specified by the information indicating the brightness of each pixel.

(Configuration of the dynamic image data 1340)
In this embodiment, the dynamic image data 1340 has a plurality of frames including a frame 1342 and a frame 1344. Each of the plurality of frames constituting the dynamic image data 1340 is generated based on the data of a plurality of frames included in the moving image data 1320. As described above, in this embodiment, each frame of the moving image is divided into a plurality of predetermined calculation units, and a value of a dynamic index is derived for each calculation unit. Therefore, each calculation unit of the frames constituting the moving image data 1320 corresponds to each pixel of the frames constituting the dynamic image data 1340.

The data structure of a frame constituting the variation image data 1340 includes, for example, identification information of each pixel constituting the frame, information indicating the position of each pixel in the frame, and information indicating the degree of brightness variation of each pixel. For example, data of the i-th pixel _VPi included in a frame includes xi indicating the position in the x direction on the image plane of the frame, yi indicating the position in the y direction on the image plane, and a pixel value indicating the degree of brightness variation of the pixel.

The above-mentioned pixel value may be the value of the variation index. If the frame is a color image represented in RGB format, three values indicating the degree of variation in the luminance of each of the RGB may be stored as the pixel value, or a single value indicating the degree of variation in the sum of the luminance of each of the RGB may be stored.

In one embodiment, when a general camera 154 captures an image in an extremely dark place, the evaluation accuracy is improved by using the B luminance data of the RGB luminances, or by setting the weight of the B luminance data to be greater than the weight of the R and G luminance data. In another embodiment, when the subject of the camera 154 is human skin, the evaluation accuracy is improved by using the R luminance data, or by setting the weight of the R luminance data to be greater than the weight of the B and G luminance data. In yet another embodiment, when the subject of the camera 154 is a plant, the evaluation accuracy is improved by using the G luminance data, or by setting the weight of the G luminance data to be greater than the weight of the B and R luminance data.

(Configuration of control device 170)
In this embodiment, the control device 170 includes a physical property information acquisition unit 614, a variation index derivation unit 1374, an image generation unit 1376, and a safety management unit 1378. The variation index derivation unit 1374 is disposed in the first index derivation unit 1142. The safety management unit 1378 may be another example of the safety management unit 640. Details of the safety management unit 1378 will be described later.

The fluctuation index derivation unit 1374 derives the value of the fluctuation index described above. This makes it possible to derive the degree of fluctuation in the intensity of light reaching the camera 154 during a specific period of time. The fluctuation index derivation unit 1374 derives the value of the fluctuation index, for example, by the following procedure.

According to this embodiment, the variation index derivation unit 1374 first extracts multiple frames from the video data acquired by the physical property information acquisition unit 614 in accordance with a predetermined rule. The variation index derivation unit 1374 may extract multiple frames from the video data acquired by the physical property information acquisition unit 614 that was captured while the camera 154 was stationary.

Examples of the predetermined rule include (i) a rule of extracting a predetermined number of consecutive frames, (ii) a rule of extracting a predetermined number of frames from among frames corresponding to a period having a predetermined length (sometimes referred to as a unit period), and (iii) a rule of extracting a predetermined second number of frames (the second number being less than the first number) from among a predetermined first number of consecutive frames. For example, according to rule (i), 10 consecutive frames are extracted. On the other hand, according to rule (ii) or (iii), for example, one frame is extracted every three frames, for a total of 10 frames are extracted.

Next, the fluctuation index derivation unit 1374 divides each of the extracted frames into a plurality of calculation units. The fluctuation index derivation unit 1374 may divide each of the extracted frames into a plurality of calculation units according to a predetermined rule. The shapes of the plurality of calculation units may all be the same, some may differ, or all may be different. The sizes of the plurality of calculation units may all be the same, some may differ, or all may be different.

Next, the variation index derivation unit 1374 derives a variation index value for each of the multiple calculation units. The variation index value for each calculation unit is determined, for example, based on the pixel values of the pixels included in each calculation unit.

For example, if the camera 154 is stationary during the period in which the extracted frames are captured, the first calculation unit of the first frame and the first calculation unit of the second frame correspond to the same position in the real world. In this case, the first calculation unit of the first frame and the first calculation unit of the second frame are sometimes said to correspond to each other. This relationship is also sometimes called the corresponding relationship of the calculation units between frames.

The variation index derivation unit 1374 first sums up the pixel values of the pixels included in each calculation unit in each frame. This provides a total value of the pixel values for each calculation unit in each of the multiple frames. Next, the variation index derivation unit 1374 calculates an index indicating the variation of the above total value between multiple frames for each calculation unit (sometimes referred to as performing statistical processing). As described above, examples of the index indicating the variation include at least one of the variance, standard deviation, and coefficient of variation, and values obtained by dividing these by a reference value. The variation index derivation unit 1374 outputs information indicating the value of the variation index in each calculation unit to the image generation unit 1376 in association with the identification information of each calculation unit.

The variation index derivation unit 1374 may analyze the extracted multiple frames to determine whether the camera 154 moved during the period in which the multiple frames were captured. If it is determined that the camera 154 moved during the above period, the variation index derivation unit 1374 first determines the correspondence relationship of the calculation units between the multiple frames.

For example, the variation index derivation unit 1374 first determines a first frame of the multiple frames as a reference frame. Next, the variation index derivation unit 1374 identifies an area corresponding to the first area of the first frame from among the multiple areas of the second frame. The variation index derivation unit 1374 repeats the same process for the other frames. The variation index derivation unit 1374 also repeats the same process for the second area of the first frame. In this way, the correspondence between the calculation units between frames is determined. Next, the variation index derivation unit 1374 performs the statistical process described above between corresponding calculation units, taking into account the correspondence between the calculation units between frames. In this way, the value of the variation index for each calculation unit is derived.

In this embodiment, the image generating unit 1376 generates each frame (sometimes referred to as a variation image) of the variation image data 1340. The image generating unit 1376 may generate each frame of the variation image data 1340 based on the values of the variation indices of the multiple calculation units derived by the variation index derivation unit 1374 and a predetermined number of gradations.

In one embodiment, the image generation unit 1376 generates a variation image such that each of the multiple calculation units constitutes a pixel of the variation image. For example, the image generation unit 1376 first determines the position of each calculation unit in the variation image based on the positional relationship of the multiple calculation units in a single frame (sometimes referred to as a normal image) of the video data 1320. The variation index derivation unit 1374 may determine the position of each calculation unit in the variation image such that the positional relationship of the multiple calculation units in the normal image is reflected.

Next, the variation index derivation unit 1374 generates each frame of the variation image data 1340 by associating identification information of each calculation unit, information indicating the position of each calculation unit in the variation image, and information indicating the value of the variation index for each calculation unit. The information indicating the value of the variation index may be information indicating a continuous numerical value, or may be information indicating a gradational division. When the number of gradations of the variation image is determined in advance, the variation index derivation unit 1374 may determine the pixel value of each pixel based on the value of the variation index for each pixel of the variation image and the number of gradations.

The variation index derivation unit 1374, for example, directly derives a variation index from various degrees of variation (sometimes referred to as primary derivation). The variation index derivation unit 1374 may generate a new variation index using a narrowing filter operation according to the characteristics and purpose based on several derived variation indexes (sometimes referred to as secondary derivation). Examples of the above filter operation include (i) logical calculations such as sum, product, and logical OR, and (ii) conditional branching. The content of the above filter operation is determined, for example, based on the basis for extracting each of the multiple variation indexes. Examples of the basis for extracting the multiple variation indexes include the type of sensor used to calculate the variation index, the aggregation conditions used to calculate the variation index, and the RGB elements or frequencies used to calculate the variation index. A secondary derivation may be performed based on the result of the primary derivation already calculated, or a secondary derivation may be further performed based on the result of the secondary derivation already calculated.

As an example of secondary derivation, in one embodiment, when the camera 154 captures an image in an extremely dark place, for example, the extremely dark area in the image can be detected by extracting pixels that satisfy the condition 130>B>G>R. The extremely dark area may be the entire image or a part of the image. Since the fluctuation index tends to be calculated to be small in the extremely dark area, a filter calculation process is performed to correct the value of the fluctuation index in the extremely dark area. This can improve the accuracy of derivation of the fluctuation index.

In another embodiment, when the camera 154 captures the light source and the sun, pixels in the exposed area that are whited out can be detected by extracting pixels that satisfy the conditions, for example, color = ALL 255 AND deviation = 0. In this case, accuracy can be improved by substituting the variation index with another variation index by weighting it.

In yet another embodiment, for example, when slight vibrations occur in the user terminal 120 due to typing on the keyboard, the vibrations of the camera 154 also appear in the above standard deviation. In this case, for example, as an example of the above-mentioned filter calculation, a fluctuation index is generated by calculating the product or AND of the fluctuation indexes of the two deviations. Examples of the above two deviations are 0.5 seconds and 0.1 seconds. This can suppress a decrease in evaluation accuracy due to unnecessary vibrations.

In yet another embodiment, the image generation unit 1376 generates the variation image such that each of the multiple pixels included in each calculation unit constitutes a pixel of the variation image. For example, the image generation unit 1376 first determines the pixel value of each of the one or more pixels included in each of the multiple calculation units. The image generation unit 1376 may determine the value of the variation index in each calculation unit as the value of the variation index for each pixel included in the calculation unit.

Next, the variation index derivation unit 1374 generates each frame of the variation image data 1340 by associating the identification information of each pixel with information indicating the position of each pixel in the variation image and information indicating the value of the variation index for each pixel. The information indicating the value of the variation index may be information indicating a continuous numerical value or may be information indicating a gradational division. When the number of gradations of the variation image is determined in advance, the variation index derivation unit 1374 may determine the pixel value of each pixel based on the value of the variation index for each pixel of the variation image and the number of gradations.

As described above, the physical property information acquisition unit 614 may acquire multiple pieces of video data 1320 with different imaging directions. In this case, the variation index derivation unit 1374 may generate variation image data 1340 corresponding to each of the multiple pieces of video data 1320.

In this embodiment, the safety management unit 1378 evaluates the safety of the display area 142. The safety management unit 1378 may evaluate the safety of the display area 142 by deriving the degree of viewability of the output image by a third party other than the user 22.

The safety management unit 1378 evaluates the safety of the display area 142, for example, when the display area 142 is arranged at the imaging position of the camera 154. The safety management unit 1378 may evaluate the safety of the display area 142 when the display area 142 is arranged at the imaging position of the camera 154 so that the installation direction of the display area 142 is approximately the same as the imaging direction of the camera 154. The safety management unit 1378 determines that the safety of the display area 142 is higher, for example, as the risk of information displayed in the display area 142 being leaked is smaller. Examples of the safety of the display area 142 include the safety of the installation position of the display area 142, and the safety of the installation position and installation direction of the display area 142. When the display area 142 is housed in the housing 130, the safety of the display area 142 may mean the safety of the user terminal 120.

In this embodiment, the safety management unit 1378 differs from the safety management unit 640 in that the safety management unit 1378 evaluates the safety of the display area 142 using the variation index derived by the variation index derivation unit 1374 or the variation image generated by the image generation unit 1376. The safety management unit 1378 may have a configuration similar to that of the safety management unit 640, except for the above differences. Details of the safety management unit 1378 will be described later.

Each frame of the dynamic image data 1340 may be an example of a dynamic image. The image generation unit 1376 may be an example of a dynamic image generation unit. The safety management unit 1378 may be an example of a viewability derivation unit.

Figure 14 shows an example of a process for generating the dynamic image data 1340. In this embodiment, the physical property information acquisition unit 614 includes a ring buffer 1472, and data for each frame included in the video data 1320 is stored in sequence in each of the multiple queues of the ring buffer 1472.

In this embodiment, for example, data of frames 1322 to 1332, which are normal images, are input to the variation index derivation unit 1374, and the variation index derivation unit 1374 derives the value of an index (which may be referred to as a variation index as described above) indicating the variation in color and/or brightness between the input frames. Furthermore, the image generation unit 1376 generates data of frame 1342, which is a variation image, based on the value of the variation index derived by the variation index derivation unit 1374. Similarly, data of frames 1324 to 1334 are input to the variation index derivation unit 1374, and the image generation unit 1376 generates frame 1344. The variation index derivation unit 1374 and the image generation unit 1376 repeat the same process to generate variation image data 1340.

In this embodiment, for the purpose of simplifying the explanation, an example of the generation process of the variation image data 1340 has been explained using as an example a case where the pixel values of the pixels in the entire area of each frame from frame 1322 to frame 1332 are input to the variation index derivation unit 1374, and the image generation unit 1376 generates frame 1342 corresponding to the entire area. However, the generation process of the variation image data 1340 is not limited to this embodiment.

In another embodiment, pixel values of pixels in a partial area of each frame are input to the variation index derivation unit 1374, and the image generation unit 1376 generates a frame corresponding to the partial area. Examples of normal images (sometimes referred to as target images) that are input to the variation index derivation unit 1374 and that are the subject of the variation image generation process include each of the multiple unit images 1240 described in relation to FIG. 12.

Fig. 15 shows an example of the internal configuration of the variation index derivation unit 1374. Using Fig. 15, an example of correction processing in the case where the camera 154 moves during the period in which the multiple frames input to the variation index derivation unit 1374 are captured will be described. In this embodiment, the variation index derivation unit 1374 includes a variation index calculation unit 1522, a movement degree determination unit 1524, and a correction unit 1526.

In this embodiment, the variation index calculation unit 1522 receives, for example, data of multiple frames that are the target image, and calculates the value of the variation index for each of multiple calculation units included in the target image. The variation index calculation unit 1522 may calculate the value of the variation index according to a procedure similar to the procedure described in relation to Figures 13 and 14.

In this embodiment, the movement degree determination unit 1524 determines the degree of movement (sometimes referred to as the movement degree) of the camera 154 during the period in which the above-mentioned multiple frames are captured. Examples of the movement degree include the direction and amount of movement. For example, the movement degree determination unit 1524 analyzes the video data 1320, and determines the movement degree of the camera 154 that captured the video data 1320 based on the movement degree of feature points in the images between the above-mentioned multiple frames.

In this embodiment, the correction unit 1526 corrects the value of the variation index for each of the multiple calculation units based on the degree of movement of the camera 154 determined by the degree of movement determination unit 1524. For example, the correction unit 1526 determines whether the camera 154 has moved during the above-mentioned period based on the degree of movement determined by the degree of movement determination unit 1524.

If it is determined that the camera 154 has not moved during the above period, the correction unit 1526 determines that correction is not necessary. In this case, the variation index derivation unit 1374 outputs the value calculated by the variation index calculation unit 1522 as the value of the variation index.

On the other hand, if it is determined that the camera 154 has moved during the above period, the correction unit 1526 determines that correction is necessary. For example, the correction unit 1526 first determines the correspondence between the calculation units among the multiple frames input to the variation index derivation unit 1374. Next, the correction unit 1526 executes the statistical processing described above between the corresponding calculation units, taking into account the correspondence between the calculation units among the frames, and calculates the value of the variation index for each calculation unit. In this case, the variation index derivation unit 1374 outputs the value calculated by the correction unit 1526 as the value of the variation index.

(An example of another embodiment)
In the present embodiment, an example of the correction process has been described using as an example a case in which the fluctuation index calculation unit 1522 and the correction unit 1526 calculate the value of the fluctuation index. However, the correction process is not limited to this embodiment.

In another embodiment, the variation index calculation unit 1522 may determine the calculation unit for each frame to be subjected to statistical processing depending on whether or not the camera 154 has moved. For example, before the variation index calculation unit 1522 calculates the variation index, the correction unit 1526 determines whether or not correction is necessary. If it is determined that correction is not necessary, the variation index calculation unit 1522 performs statistical processing between multiple calculation units arranged at the same position in each frame. On the other hand, if it is determined that correction is necessary, the variation index calculation unit 1522 determines the correspondence relationship of the calculation units between multiple frames, taking into account the degree of movement of the camera 154. The variation index calculation unit 1522 performs statistical processing between multiple corresponding calculation units.

The safety management unit 1378 will be described in detail using Figures 16, 17, 18, and 19. Figure 16 shows an example of the internal configuration of the safety management unit 1378. Figure 17 shows an example of information processing in the safety management unit 1378. Figure 18 shows an example of a two-dimensional image 1800 that is a normal image in which a marker is captured. Figure 19 shows an example of a panoramic image 1900 in which multiple dynamic images are synthesized.

Figure 16 shows an example of the internal configuration of the safety management unit 1378. In this embodiment, the safety management unit 1378 includes, for example, a derivation procedure determination unit 1130, an index value derivation unit 1140, an evaluation unit 1150, and an evaluation target determination unit 1620. In this embodiment, the evaluation target determination unit 1620 includes, for example, a variation image acquisition unit 1622, a combined image generation unit 1624, a unit image extraction unit 1626, a marker detection unit 1632, a pixel value change unit 1634, an imaging position determination unit 1636, and an imaging direction determination unit 1638.

The safety management unit 1378 differs from the safety management unit 640 in that it includes an evaluation target determination unit 1620. Except for the above differences, the safety management unit 1378 may have the same configuration as the safety management unit 640.

As described above, when the display area 142 and the camera 154 are housed in the same housing 130, the imaging position of the camera 154 approximately coincides with the installation position of the display area 142 or the user terminal 120. Similarly, the imaging direction of the camera 154 approximately coincides with the installation direction of the display area 142 or the user terminal 120.

In this embodiment, the evaluation target determination unit 1620 determines a target image to be evaluated for safety from the dynamic image data 1340. As described above, the target image may be the entire frame included in the dynamic image data 1340, or a part of the frame.

In this embodiment, the variation image acquisition unit 1622 acquires one or more variation images generated by the image generation unit 1376. When the camera 154 captures multiple videos with different imaging directions, the variation image acquisition unit 1622 may acquire multiple variation images for each of the multiple videos.

In this embodiment, the combined image generating unit 1624 generates a combined image by panoramic synthesis of the multiple variation images generated by the image generating unit 1376. The panoramic image 1900 described in relation to FIG. 19 may be an example of a combined image.

In this embodiment, the unit image extraction unit 1626 extracts one or more unit images to be evaluated from the varying image or combined image. Each unit image corresponds to the viewing angle of the display area 142 when the display area 142 is disposed at a specific position with a specific orientation. The unit image extraction unit 1626 may extract multiple unit images with different specific orientations.

In this embodiment, the marker detection unit 1632 detects a marker that appears in the video that is the source of the dynamic image. As described above, the marker may include information indicating the degree of viewability. The marker may include information indicating the degree of viewability and information indicating the size of the area to which the degree of viewability applies. As described above, examples of the marker include an AR marker, a QR code (registered trademark), and a barcode. Examples of the AR marker include ArUco and a chameleon code, but the AR marker is not limited to these. For example, a QR code (registered trademark) and a barcode may be used as the AR marker.

In this embodiment, the pixel value modification unit 1634 modifies the pixel value of the variable image when the marker detection unit 1632 detects a marker. The pixel value modification unit 1634 may modify the pixel value of the variable image based on information indicated by the detected marker. When the marker is a QR code (registered trademark) or a barcode, the pixel value modification unit 1634 may acquire the information indicated by the detected marker by referring to a database that stores the identification information of the marker in association with the content of information processing when the marker is detected.

For example, the pixel value modification unit 1634 determines the value of the variation index corresponding to an area in which a marker is captured among multiple areas of the normal image, based on information indicating the degree of viewability indicated by the marker. When the detected marker indicates the size of the area to which the degree of viewability is applied, the pixel value modification unit 1634 may identify the area to which the degree of viewability is applied among multiple areas of the normal image, based on the information indicated by the marker. The pixel value modification unit 1634 determines the pixel value of a pixel included in an area in the variation image or unit image corresponding to the above area, based on information indicating the degree of viewability indicated by the detected marker. For example, the pixel value modification unit 1634 changes the pixel value of the above pixel in the variation image or unit image to a value corresponding to the degree of viewability indicated by the detected marker.

In this embodiment, the imaging position determination unit 1636 determines the imaging position of the target image. The imaging position determination unit 1636 determines the imaging position of the target image based on, for example, the imaging condition data 1310.

In this embodiment, the imaging direction determination unit 1638 determines the imaging direction of the target image. When the center point of the target image coincides with the position of the optical axis of the camera 154 in the fluctuation image from which the target image is extracted, the imaging direction determination unit 1638 determines the imaging direction of the target image, for example, based on the imaging condition data 1310. On the other hand, when the center point of the target image does not coincide with the position of the optical axis of the camera 154 in the fluctuation image from which the target image is extracted, the imaging position determination unit 1636 determines the imaging direction of the target image, for example, based on the position of the optical axis of the camera 154 in the fluctuation image from which the target image is extracted, the position of the center point of the target image in the fluctuation image from which the target image is extracted, the focal length of the camera 154, and the imaging condition data 1310.

In this embodiment, the evaluation unit 1150 derives the degree of viewability of the output image by a third party other than the user of the camera 154 when the display area 142 is placed at a position where the camera 154 captures the video. The degree of viewability may be an example of an index indicating safety.

In this embodiment, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability based on the values of the variation indices of the multiple regions derived by the variation index derivation unit 1374. The pixel values of the target image determined as the evaluation target by the evaluation target determination unit 1620 indicate the values of the variation indices of the multiple regions derived by the variation index derivation unit 1374.

Therefore, in this embodiment, the evaluation unit 1150 derives the degree of viewability described above based on the target image. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability described above for each of the multiple target images.

In one embodiment, as described above, the target image may be a plurality of variable images or unit images with different imaging directions. This allows an evaluation value indicating the degree of viewability in each of the plurality of imaging directions to be derived.

In other embodiments, the target image may be a fluctuating image or a unit image at multiple time points during the period during which the user 22 continuously uses the camera 154. This allows the evaluation unit 1150 to monitor the safety of the user terminal 120 during the period during which the user 22 uses the user terminal 120.

The evaluation unit 1150 may calculate, for each of the multiple regions, a viewing index value, which is an index indicating the degree of possibility that the output image will be viewed from a position corresponding to each region in real space, based on the value of the variation index for each of the multiple regions. The evaluation unit 1150 may derive an evaluation value indicating the degree of viewability, based on the value of the viewing index for each of the multiple regions. In one embodiment, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability by summing up the values of the viewing index for each of the multiple regions. In another embodiment, the evaluation unit 1150 derives an evaluation value indicating the degree of viewability by calculating the average value of the viewing index for each of the multiple regions adjacent to a specific region (the sum of the viewing index values for each region divided by the number of regions).

In one embodiment, the evaluation unit 1150 calculates the value of the viewing index such that the value of the viewing index in an area with a large variation index value is greater than the value of the viewing index in an area with a small variation index value. In another embodiment, the evaluation unit 1150 calculates the value of the viewing index such that the value of the viewing index in an area close to the position of the optical axis of the camera 154 in the video is greater than the value of the viewing index in an area far from the position of the optical axis of the camera 154 in the video. In yet another embodiment, the evaluation unit 1150 calculates the value of the viewing index such that the value of the viewing index in an area close to the horizon is greater than the value of the viewing index in an area far from the horizon.

Figure 17 shows an example of information processing in the safety management unit 1378. An example of a procedure in which the safety management unit 1378 evaluates the safety of the display area 142 is described using Figure 17. As described above, the safety management unit 1378 evaluates the safety of the display area 142 using the fluctuation index derived by the fluctuation index derivation unit 1374 or the fluctuation image generated by the image generation unit 1376.

According to this embodiment, first, in S1722, the safety management unit 1378 acquires one or more variation images and the imaging positions and imaging directions corresponding to each variation image. For example, the safety management unit 1378 acquires variation image data 1340 and imaging condition data 1310 indicating the imaging conditions when the video data 1320 that is the source of the variation image data 1340 was captured.

Next, in S1724, the safety management unit 1378 determines whether or not there is an event for changing the pixel value of the variable image. If an event for changing the pixel value of the variable image is detected, the safety management unit 1378 changes the pixel value of the variable image in accordance with the detected event.

An example of an event for changing the pixel values of the dynamic image is when a predetermined type of marker is detected in a frame of the video data 1320 that is the source of the frame included in the dynamic image data 1340. For example, the safety management unit 1378 analyzes the frame of the video data 1320 to determine the presence or absence of a predetermined type of marker. Examples of the above marker include an AR marker, a QR code (registered trademark), and a barcode. Examples of the AR marker include ArUco and a chameleon code, but the AR marker is not limited to these. As described above, for example, a QR code (registered trademark) and a barcode may be used as the AR marker.

According to this embodiment, the marker includes information indicating the degree of safety of the position where the marker is attached or the object where the marker is attached, or includes information associated with the information indicating the degree of safety (e.g., identification information of the marker). The marker may include information regarding the range of effect of the marker, or may include information associated with the information regarding the range of effect of the marker (e.g., identification information of the marker).

This allows the safety management unit 1378 to acquire information indicating the position where the marker detected in the frame of the video data 1320 is attached or the degree of safety of the object to which the marker is attached. The safety management unit 1378 may acquire various types of information by referring to a database in which the identification information of the marker, information indicating the position where the marker is attached or the degree of safety of the object to which the marker is attached, and information regarding the range of the effect of the marker are associated.

The safety management unit 1378 changes the pixel value of the corresponding pixel in the variation image based on the position of the calculation unit or pixel where the marker is detected in the normal image and the degree of safety indicated by the marker. This allows the user 22 to install the user terminal 120 in an appropriate location that corresponds to reality.

For example, because a glass window transmits and reflects light, the value of the variation index of the pixel corresponding to the glass window in the normal image in the variation image is relatively large. However, if there is a low possibility that the inside of the room can be seen from outside, a marker indicating high safety is attached to the glass window in the real world. If a marker is not attached to the glass window, the user 22 cannot use the user terminal 120 in a position with his/her back to the glass window. On the other hand, if a marker is attached to the glass window, the user 22 can use the user terminal 120 in a position with his/her back to the glass window.

Next, in S1726, the safety management unit 1378 extracts a target image to be evaluated for safety from among the one or more variable images acquired in S1722. In one embodiment, the safety management unit 1378 evaluates the safety of the position and/or direction corresponding to the target image, using the entire region of the single variable image generated by the image generation unit 1376 as the target image. In another embodiment, the safety management unit 1378 evaluates the safety of the position and/or direction corresponding to the target image, using a partial region of the single variable image generated by the image generation unit 1376 as the target image. An example of the partial region of the variable image is a region corresponding to the viewing angle of the display area 142.

Next, in S1728, the safety management unit 1378 determines the imaging position and imaging direction corresponding to the extracted target image. For example, the safety management unit 1378 determines the imaging position and imaging direction based on the imaging condition data 1310 acquired in S1722.

Next, in S1730, the safety management unit 1378 determines an evaluation value regarding the safety of the target image. In addition, the safety management unit 1378 outputs the evaluation value regarding the safety of the target image in association with the imaging position and/or imaging direction of the target image.

Figure 18 shows an example of a two-dimensional image 1800. In Figure 18, marker 1822 shows an example of the marker described in relation to S1724 in Figure 17. Also, area 1824 shows an example of the range of effect of marker 1822.

Figure 19 shows a schematic diagram of an example of a panoramic image 1900. As described in relation to S1726 in Figure 17, the safety management unit 1378 extracts a target image to be evaluated for safety from one or more variation images acquired in S1722. In this case, the safety management unit 1378 may generate a single composite image by synthesizing multiple variation images generated from normal images captured in different directions.

For example, the safety management unit 1378 synthesizes the above multiple variable images to generate a panoramic image 1900. According to the example shown in FIG. 19, a single panoramic image 1900 is synthesized from variable images 1920, 1922, 1924, 1926, and 1928.

The safety management unit 1378 may extract an image (sometimes called a unit image) that shows an area that corresponds to the viewing angle of the display area 142 from the panoramic image 1900, and determine the unit image as the target image. The safety management unit 1378 may extract multiple unit images 1940 as the target image by shifting the position of the center 1942 of the extracted unit image 1940 in the left-right direction or the up-down direction along a predetermined dotted line 1950.

(An example of another embodiment)
In another embodiment, the panoramic image 1900 may be periodically refreshed. For example, a new panoramic image 1900 is generated at a predetermined time interval. In this case, a guide process for generating a new panoramic image 1900 may be executed. This allows the latest information of the real world to be reflected in the panoramic image 1900. As a result, the accuracy of the safety evaluation is improved.

In the embodiment described in relation to FIG. 19, an example of a technique for virtually expanding the angle of view of an imaging device (sometimes referred to as an angle of view expansion technique) has been described, taking as an example a case in which the control device 170 executes a process (sometimes referred to as panoramic processing) in which multiple variable images generated from normal images captured in different directions are synthesized to generate a single synthesized image (the panoramic image described above). Each of the multiple normal images or variable images described above may be referred to as a current image.

However, the angle of view extension technique is not limited to this embodiment. Other embodiments of the angle of view extension technique for the current image include detecting n-th reflection (n is an integer equal to or greater than 2) of electromagnetic waves and/or sound waves from the current image and detecting the fluctuation, dynamic change, etc. of an object existing outside the angle of view. For example, if the surface of the subject of the image is prone to reflect light, an object existing behind the imaging device that captured the image will be reflected on the surface of the subject. The control device 170 can detect the movement of an object located outside the angle of view of the imaging device by analyzing the fluctuation of light or shadow reflected on the surface of the subject. Examples of objects having surfaces that are prone to reflect electromagnetic waves and/or sound waves include mirrors, glass windows, whiteboards, walls with high directional reflection characteristics, TV screens that are turned off, and refrigerators. The control device 170 may execute a process of analyzing the image to detect the boundary of the subject and/or a process of identifying the type of the detected subject or a process of determining whether the subject has a reflective surface.

For example, the control device 170 focuses on the degree of fluctuation of n-th order reflection of light or shadow reflected on the surface of an object within the angle of view of the imaging device. If there is a large change in the degree of fluctuation of the n-th order reflected light, the control device 170 may determine that a moving object is present in the vicinity of the imaging device (e.g., outside the angle of view of the imaging device) and determine that the degree of safety is low.

More specifically, the control device 170 first acquires video data from an imaging device that captures the periphery of the display area. Next, the control device 170 analyzes the video data to generate a fluctuation image. Next, the control device 170 analyzes the fluctuation image to extract pixels or regions whose fluctuation index values are greater than a predetermined value. The control device 170 determines whether or not the above-mentioned pixels or regions correspond to an object having a reflective surface. If the above-mentioned pixels or regions correspond to an object having a reflective surface, the control device 170 determines that the pixel or region corresponds to outside the angle of view of the imaging device. This allows the control device 170 to detect a moving object that exists outside the angle of view of the imaging device.

The evaluation unit 1150 may be an example of a viewability derivation unit.

Figure 20 shows an example of a computer 3000 in which multiple aspects of the present invention may be embodied in whole or in part. At least a part of the information leakage prevention system 100 may be realized by the computer 3000. For example, at least a part of the user terminal 120 is realized by the computer 3000 or a part thereof. For example, at least a part of the user terminal 120 and the control device 170 are realized by the computer 3000 or a part thereof.

A program installed on the computer 3000 may cause the computer 3000 to function as or perform operations associated with an apparatus according to an embodiment of the present invention or one or more "parts" of the apparatus, and/or to perform a process or steps of the process according to an embodiment of the present invention. Such a program may be executed by the CPU 3012 to cause the computer 3000 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 3000 according to this embodiment includes a CPU 3012, a RAM 3014, a GPU 3016, and a display device 3018, which are interconnected by a host controller 3010. The computer 3000 also includes input/output units such as a communication interface 3022, a hard disk drive 3024, a DVD-ROM drive 3026, and an IC card drive, which are connected to the host controller 3010 via an input/output controller 3020. The computer also includes legacy input/output units such as a ROM 3030 and a keyboard 3042, which are connected to the input/output controller 3020 via an input/output chip 3040.

The CPU 3012 operates according to the programs stored in the ROM 3030 and the RAM 3014, thereby controlling each unit. The GPU 3016 acquires image data generated by the CPU 3012 into a frame buffer or the like provided in the RAM 3014 or into itself, and causes the image data to be displayed on the display device 3018.

The communication interface 3022 communicates with other electronic devices via a network. The hard disk drive 3024 stores programs and data used by the CPU 3012 in the computer 3000. The DVD-ROM drive 3026 reads programs or data from the DVD-ROM 3001 and provides the programs or data to the hard disk drive 3024 via the RAM 3014. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

The ROM 3030 stores therein a boot program, etc., which is executed by the computer 3000 upon activation, and/or a program that depends on the hardware of the computer 3000. The input/output chip 3040 may also connect various input/output units to the input/output controller 3020 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

The programs are provided by a computer-readable storage medium such as a DVD-ROM 3001 or an IC card. The programs are read from the computer-readable storage medium, installed in the hard disk drive 3024, RAM 3014, or ROM 3030, which are also examples of computer-readable storage media, and executed by the CPU 3012. The information processing described in these programs is read by the computer 3000, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing the operation or processing of information according to the use of the computer 3000.

For example, when communication is performed between computer 3000 and an external device, CPU 3012 may execute a communication program loaded into RAM 3014 and instruct communication interface 3022 to perform communication processing based on the processing described in the communication program. Under the control of CPU 3012, communication interface 3022 reads transmission data stored in a transmission buffer area provided in RAM 3014, hard disk drive 3024, DVD-ROM 3001, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 3012 may also cause all or a necessary portion of a file or database stored on an external recording medium such as the hard disk drive 3024, the DVD-ROM drive 3026 (DVD-ROM 3001), an IC card, etc. to be read into the RAM 3014, and perform various types of processing on the data on the RAM 3014. The CPU 3012 may then write back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 3012 may execute various types of processing on the data read from the RAM 3014, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 3014. The CPU 3012 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 3012 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The above-described program or software module may be stored in a computer-readable storage medium on the computer 3000 or in the vicinity of the computer 3000. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the above-described program to the computer 3000 via the network.

Although the present invention has been described above using an embodiment, the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear to those skilled in the art that various modifications or improvements can be made to the above embodiment. Furthermore, the details described for a specific embodiment can be applied to other embodiments to the extent that they are not technically inconsistent. It is clear from the claims that such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that the processes may be performed in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is necessary to perform the processes in this order.

For example, the present specification discloses the following:
(Item A1)
a peripheral state acquisition unit that acquires peripheral state information indicating a peripheral state of a display area on which an output image output by an image output device is displayed;
a viewability derivation unit that derives a degree of viewability of the output image by a third party different from a user of the image output device based on a state of the periphery of the display area indicated by the surrounding state information acquired by the surrounding state acquisition unit;
An information processing device comprising:
(Item A2)
a peripheral state acquisition step of acquiring peripheral state information indicating a peripheral state of a display area on which an output image outputted by an image output device is displayed;
a viewability derivation step of deriving a degree of viewability of the output image by a third party other than a user of the image output device based on a state of the periphery of the display area indicated by the surrounding state information acquired in the surrounding state acquisition step;
An information processing method comprising the steps of:
(Item B1)
A video acquisition unit that acquires video data including a plurality of frames;
(a) dividing each frame of the moving image into a plurality of predetermined regions; (b) deriving a value of a fluctuation index for each of the plurality of regions, the fluctuation index being an index indicating a fluctuation in color and/or brightness of each of the plurality of frames, based on pixel values of pixels included in each of the plurality of regions;
Equipped with
Each of the plurality of regions includes one or more pixels.
Image processing device.
(Item B2)
A video capture stage for capturing video data including a plurality of frames;
(a) dividing each frame of the video into a plurality of predetermined regions; and (b) deriving, for each of the plurality of regions, a value of a fluctuation index which is an index indicating a fluctuation in color and/or brightness of each region among the plurality of frames, based on pixel values of pixels included in each region.
having
Each of the plurality of regions includes one or more pixels.
Image processing methods.
(Item C1)
a peripheral data acquisition unit that acquires image data of an image having one or more objects as subjects arranged around a display area on which an output image outputted from an image output device is displayed, or peripheral data that is three-dimensional point cloud data or distance data of the one or more objects;
an object position determination unit that analyzes the surrounding data acquired by the surrounding data acquisition unit and determines a positional relationship between each of the one or more objects and the display area;
a viewability deriving unit that derives a degree of viewability of the output image by a third party different from a user of the image output device based on the positional relationship determined by the object position determining unit;
An information processing device comprising:
(Item C2)
a peripheral data acquisition step of acquiring image data of an image having one or more objects disposed around a display area on which an output image outputted from an image output device is displayed, or peripheral data which is three-dimensional point cloud data or distance data of the one or more objects;
an object position determination step of analyzing the surrounding data acquired in the surrounding data acquisition step to determine a positional relationship between each of the one or more objects and the display area;
a viewability deriving step of deriving a degree of viewability of the output image by a third party different from a user of the image output device based on the positional relationship determined in the object position determining step;
An information processing method comprising the steps of:

22 User 32 Third party 34 Third party 100 Information leakage prevention system 120 User terminal 130 Housing 140 Display device 142 Display area 150 Environmental sensor 152 Peripheral data generation device 154 Camera 156 Illuminance sensor 160 Input device 170 Control device 240 Representative point 242 Normal vector 244 Vector 246 Vector 250 Representative point 252 Optical axis 254 Vector 256 Vector 300 Work room 312 Wall 314 Wall 316 Wall 318 Wall 320 Floor 322 Glass window 330 Shield 342 Marker 344 Marker 346 Marker 610 Peripheral state acquisition unit 612 Structural information acquisition unit 614 Physical property information acquisition unit 616 Environmental information acquisition unit 618 Attribute information acquisition unit 622 Start detection unit 624 Timer unit 630 Virtual space construction unit 640 Safety management unit 652 Placement determination unit 654 Placement support unit 660 Output control unit 670 Storage unit 712 Three-dimensional point cloud acquisition unit 714 Distance image acquisition unit 716 Stereo image acquisition unit 718 Two-dimensional image acquisition unit 810 Data conversion unit 812 Marker position determination unit 814 Point cloud generation unit 820 Object position determination unit 830 First calculation unit 832 Point cloud data acquisition unit 834 Distance calculation unit 836 Angle calculation unit 840 Second calculation unit 842 Edge detection unit 844 Shielding angle calculation unit 910 Point cloud 924 Vector 926 Vector 1020 Shielding object 1022 Marker 1024 Marker 1040 Shielding object 1042 Marker 1044 Marker 1130 Derivation procedure determination unit 1132 Index setting unit 1134 Evaluation setting unit 1140 Index value derivation unit 1142 First index derivation unit 1144 Second index derivation unit 1146 Third index derivation unit 1148 Fourth index derivation unit 1150 Evaluation unit 1200 Image 1240 Unit image 1242 Center 1250 Dotted line 1310 Imaging condition data 1320 Video data 1322 Frame 1324 Frame 1332 Frame 1334 Frame 1340 Fluctuation image data 1342 Frame 1344 Frame 1374 Fluctuation index derivation unit 1376 Image generation unit 1378 Safety management unit 1472 Ring buffer 1522 Fluctuation index calculation section 1524 Movement state determination section 1526 Correction section 1620 Evaluation target determination section 1622 Fluctuation image acquisition section 1624 Combined image generation section 1626 Unit image extraction section 1632 Marker detection section 1634 Pixel value change section 1636 Imaging position determination section 1638 Imaging direction determination section 1800 Two-dimensional image 1822 Marker 1824 Area 1900 Panoramic image 1920 Fluctuation image 1922 Fluctuation image 1924 Fluctuation image 1926 Fluctuation image 1928 Fluctuation image 1940 Unit image 1942 Center 1950 Dotted line 3000 Computer 3001 DVD-ROM
3010 host controller 3012 CPU
3014 RAM
3016 GPU
3018 Display device 3020 Input/output controller 3022 Communication interface 3024 Hard disk drive 3026 DVD-ROM drive 3030 ROM
3040 Input/Output Chip 3042 Keyboard

Claims (11)

a peripheral data acquisition unit that acquires image data of an image having one or more objects disposed around a display area on which an output image outputted from an image output device is displayed, or peripheral data that is three-dimensional point cloud data or distance data of the one or more objects;
an object position determination unit that analyzes the surrounding data acquired by the surrounding data acquisition unit and determines a positional relationship between each of the one or more objects and the display area;
a viewability deriving unit that derives a degree of viewability of the output image by a third party different from a user of the image output device based on the positional relationship determined by the object position determining unit;
Equipped with
The viewability derivation unit derives the degree of viewability such that (i) the degree of viewability is greater for an object disposed at a position farther from the display area, and/or (ii) the degree of viewability is smaller for an object disposed at a position closer to the front of the display area.
Information processing device. a peripheral data acquisition unit that acquires image data of an image having one or more objects disposed around a display area on which an output image outputted from an image output device is displayed, or peripheral data that is three-dimensional point cloud data or distance data of the one or more objects;
an object position determination unit that analyzes the surrounding data acquired by the surrounding data acquisition unit and determines a positional relationship between each of the one or more objects and the display area;
a viewability deriving unit that derives a degree of viewability of the output image by a third party different from a user of the image output device based on the positional relationship determined by the object position determining unit;
Equipped with
The object position determination unit is
a point cloud data acquisition unit that acquires three-dimensional point cloud data of each of the one or more objects based on the peripheral data acquired by the peripheral data acquisition unit;
a distance calculation unit that calculates a distance between a representative point of the display area and each of a plurality of points virtually arranged on a surface of each of the one or more objects represented by the three-dimensional point cloud data acquired by the point cloud data acquisition unit;
having
The viewability derivation unit is
calculating, for each of the plurality of points, a value of a suppression index, which is an index indicating a degree to which each point suppresses the third party from viewing the output image, based on the distance for each of the plurality of points calculated by the distance calculation unit;
deriving the degree of viewability by summing the values of the inhibition index for each of the plurality of points;
Information processing device. The object position determination unit is
an angle calculation unit that calculates an absolute value of an angle between a normal direction at the representative point of the display area and a direction from the representative point of the display area toward each of the plurality of points arranged on the surface of each of the one or more objects;
and
The viewability derivation unit is
calculating a value of the suppression index for each of the plurality of points based on the distance for each of the plurality of points calculated by the distance calculation unit and the absolute value of the angle for each of the plurality of points calculated by the angle calculation unit;
deriving the degree of viewability by summing the values of the inhibition index for each of the plurality of points;
The information processing device according to claim 2 . a peripheral data acquisition unit that acquires image data of an image having one or more objects disposed around a display area on which an output image outputted from an image output device is displayed, or peripheral data that is three-dimensional point cloud data or distance data of the one or more objects;
an object position determination unit that analyzes the surrounding data acquired by the surrounding data acquisition unit and determines a positional relationship between each of the one or more objects and the display area;
a viewability deriving unit that derives a degree of viewability of the output image by a third party different from a user of the image output device based on the positional relationship determined by the object position determining unit;
Equipped with
The object position determination unit is
a point cloud data acquisition unit that acquires three-dimensional point cloud data of each of the one or more objects based on the peripheral data acquired by the peripheral data acquisition unit;
an angle calculation unit that calculates an absolute value of an angle between a normal direction at a representative point of the display area and a direction from the representative point of the display area toward each of a plurality of points arranged on a surface of each of the one or more objects;
having
The viewability derivation unit is
calculating, for each of the plurality of points, a value of a suppression index which is an index indicating a degree to which each point suppresses the third party from viewing the output image, based on an absolute value of the angle for each of the plurality of points calculated by the angle calculation unit;
deriving the degree of viewability by summing the values of the inhibition index for each of the plurality of points;
Information processing device. The image output device is
A housing and
The display area disposed on the housing;
a peripheral data generating device disposed in the housing and generating the peripheral data;
Equipped with
The peripheral data acquisition unit acquires the peripheral data generated by the peripheral data generation device,
The information processing device includes:
a two-dimensional image acquisition unit that acquires a two-dimensional image having the one or more objects to which the one or more markers are attached as subjects;
a marker position determination unit that analyzes the two-dimensional image acquired by the two-dimensional image acquisition unit and determines a positional relationship between at least a portion of the one or more markers and the display area;
a point cloud generating unit that generates a point cloud around the position of the marker whose positional relationship with the display area has been determined by the marker position determining unit;
Further equipped with
Each of the one or more markers includes identification information for identifying each of the one or more markers, and/or a pattern representing information regarding a point cloud generated around a position of each marker.
The information processing device according to claim 2 . An information processing device,
a peripheral data acquisition unit that acquires image data of an image having one or more objects disposed around a display area on which an output image outputted from an image output device is displayed, or peripheral data that is three-dimensional point cloud data or distance data of the one or more objects;
an object position determination unit that analyzes the surrounding data acquired by the surrounding data acquisition unit and determines a positional relationship between each of the one or more objects and the display area;
a viewability deriving unit that derives a degree of viewability of the output image by a third party different from a user of the image output device based on the positional relationship determined by the object position determining unit;
Equipped with
each of the one or more objects is an object having a set of markers indicating an edge of each object in a substantially horizontal direction;
The information processing device includes:
a two-dimensional image acquisition unit for acquiring a two-dimensional image having the one or more objects as subjects;
an end detection unit that analyzes the two-dimensional image acquired by the two-dimensional image acquisition unit, detects a plurality of markers included in the two-dimensional image, and detects one end and another end of each of the one or more objects in a substantially horizontal direction based on a combination of the plurality of markers;
an occlusion angle calculation unit that calculates, for each of the one or more objects, an occlusion angle that is an angle between a direction from the representative point of the display area toward the one end of each object and a direction from the representative point of the display area toward the other end of each object;
Further equipped with
The viewability deriving unit derives the degree of viewability based on the occlusion angle of each of the one or more objects calculated by the occlusion angle calculation unit.
Information processing device. an arrangement determination unit that determines an arrangement of the display area based on the degree of viewability derived by the viewability derivation unit;
Further equipped with
The arrangement of the display area includes a position and a direction in which the display area is installed,
the placement determination unit determines a placement of the display area such that the degree of viewability is smaller than a predetermined degree.
The information processing device according to claim 1 . the predetermined degree differs between a case where the display region is located within a range of an area having a predetermined characteristic and a case where the display region is located outside the range of the area;
The information processing device according to claim 7 . the peripheral data acquisition unit acquires the peripheral data at a plurality of points in time during a period in which the user continuously uses the image output device;
The viewability deriving unit derives the degree of viewability at each of the plurality of time points.
The information processing device according to claim 1 . an output control unit that controls output of an image by the image output device,
when the degree of viewability derived by the viewability derivation unit exceeds a predetermined degree, the output control unit stops output of the output image that was displayed before the degree of viewability exceeded the predetermined degree, or displays an image different from the output image in the display area.
The information processing device according to claim 9 . A program for causing a computer to function as the information processing device according to any one of claims 1 to 10 .