JP2020183997A - Screen display program, device and method - Google Patents

Abstract

To suppress information leakage due to screen peeping.SOLUTION: A display control unit is configured to: when the number of detection times that at least one of a face and a line of sight is detected from a camera image is less than a predetermined number, display a transparent display screen 88A in which a distribution ratio of pixels that are a display output position of an application image is made higher than that of pixels that are a display output position of a camera image; and when the number of detection times is more than the predetermined number, display a transparent display screen 88C in which the more the number of detection times is, the distribution ratio of the pixels that are the display output position of the camera image is higher than that of the pixels that are the display output position of the application image.SELECTED DRAWING: Figure 5

Description

The disclosed techniques relate to screen display programs, screen display devices, and screen display methods.

With the recent IoT (Internet of Things) technological innovation and acceleration of business use, image information such as cameras and videos is imported into information processing terminals such as users' PCs (Personal Computers) and smart device terminals, and linked with computer processing. The number of technologies that can be used is increasing. As a feature of these technologies, the processing and operations that the information processing terminal originally wants to perform and the image information are linked and displayed on the same desktop to improve operability and detect triggers for the processing that the information processing terminal originally wants to perform. The effect of is achieved.

Examples include videophone communication in which moving images captured by an in-camera are constantly transferred as image information in two-way communication such as SNS (Social Networking Service) and live Web, and crime prevention and disaster monitoring. These are based on the premise that the image or video of the camera projected on the screen is clearly and surely displayed by the operator of the information processing terminal.

On the other hand, when considering security measures, especially in information processing terminals used in BYOD (Bring Your Own Device), filters using polarized light and optical frequency characteristics are used as measures to prevent peeping in public environments. It's being used. However, these measures have problems that the screen of the process or operation that the operator originally wants to perform becomes difficult to see or the operability is deteriorated depending on the environment and physical characteristics.

Also, when considering the display for PCs used in a fixed environment such as the workplace, it is possible to handle shoulder hacking in cases where a display with a large screen size is used to perform processing that you do not want to be seen in the surroundings. think of. In this case, the utilization of the filter technology as described above has a large demerit that causes a decrease in the business and application operations that are originally desired to be performed on the PC, and is not suitable for actual use.

A camera that can be connected to a PC will be installed to support shoulder hacking on a large screen display. Then, if a mechanism for constantly displaying the image or video as a monitoring window on the information processing terminal to be operated is introduced, a blind spot such as behind the operator can be displayed on the screen.

In addition, a display device has been proposed that allows a plurality of images to be distributed and displayed on a transmissive display and another display that overlaps the display. When this display device acquires a drawing object representing an image to be displayed, it identifies the attributes of the image and determines that the foreground image and the background image are sorted based on the specified attributes. Then, the display device displays the foreground image on the transmissive display and the background image on another display according to the determination result.

Japanese Unexamined Patent Publication No. 2011-248209

In the above-mentioned method of displaying the monitoring window, the image or video of the camera is displayed in units of windows on the display screen. Therefore, when the operator is concentrating on the window of another application operating on the PC, it is possible that the image or video displayed in the monitoring window is overlooked.

Further, it is conceivable to use the display method using the transmissive display described above to transmit and superimpose either the image for the original operation or the image for monitoring the peep. However, in a transmissive display, the light transmittance and reflectivity of the display display element itself such as a liquid crystal or an organic EL (Electro Luminescence) are utilized. Therefore, since the transmittance is adjusted at the light intensity level, it is difficult to optimize the visibility between the operation image and the monitoring image, and it is not realistic to apply it to the application of peep prevention. ..

As one aspect, the disclosed technology aims to suppress information leakage due to screen peeping.

As one aspect, the disclosed technique causes a first image captured by an imaging device including the vicinity of the operator of the terminal in the imaging range and a second image to be operated to be displayed on the screen on the terminal. .. Then, the disclosed technique depends on at least one of the number of faces detected from the first image and the number of lines of sight to the screen detected from the first image, and at least the second image. Reduce visibility.

As one aspect, it has the effect of suppressing information leakage due to peeping on the screen.

It is a figure which shows the schematic structure of the screen display system which concerns on this embodiment. It is a functional block diagram of the screen display device which concerns on this embodiment. It is a figure for demonstrating the distribution of the pixel which becomes the display output position of a camera image and an application image. It is a figure for demonstrating the transparent display screen which displayed the camera image and the application image. It is a figure for demonstrating the change of the distribution ratio of the pixel which becomes the display output position of a camera image and an application image. It is a block diagram which shows the schematic structure of the computer which functions as an information processing apparatus equipped with the screen display device which concerns on this embodiment. It is a flowchart which shows an example of the screen display processing in this embodiment.

Hereinafter, an example of the embodiment according to the disclosed technology will be described with reference to the drawings.

As shown in FIG. 1, the screen display system 100 according to the present embodiment includes a screen display device 10, a display 20, and a camera 30 mounted on an information processing device 110 such as a personal computer. The information processing device 110 may be a tablet terminal, a smartphone, or the like in which the information processing device 110 itself is provided with the display 20 and the camera 30.

The display 20 displays various images (hereinafter, referred to as “application images”) such as window images of applications executed by the information processing device 110, icons arranged on the desktop, and jets. Further, the display 20 displays a moving image (hereinafter, referred to as “camera image”) captured by the camera 30. The camera image is an example of the first image of the disclosure technique, and the application image is an example of the second image of the disclosure technique.

The camera 30 captures a range including the operator 70 of the terminal and outputs the captured image. For example, the camera 30 is installed on the upper part of the display 20, and outputs an image captured with the operator 70 facing the display 20 and the back of the operator 70 as the imaging range.

Functionally, the screen display device 10 includes a detection unit 12 and a display control unit 14, as shown in FIG.

The detection unit 12 acquires the camera image output from the camera 30, processes the camera image, and detects and detects a region showing a person's face (hereinafter, simply referred to as “face”) from the camera image. The number of faces is passed to the display control unit 14. Further, the detection unit 12 may detect the line of sight to the screen displayed on the display 20 instead of or in combination with the face detection, and pass the number of the detected lines of sight to the display control unit 14. Good. Since conventionally known methods can be used for detecting the face and the line of sight from the camera image, detailed description thereof will be omitted here.

The display control unit 14 acquires the camera image and the application image, and controls so that both images are displayed on the display 20.

Here, a display position when displaying a plurality of windows on the display 20 will be described with reference to FIG. It should be noted that each of the screen 80 and the screen 82 of FIG. 3 is a schematic view in which each pixel of the display 20 is represented by an ellipse in order to explain the display position of each image. For the sake of explanation, the number of pixels is schematically represented as 13 × 10 pixels. Further, in FIG. 3, the desktop background, the icon / gadget, and the application window are examples of the application image.

As shown in the upper part of FIG. 3, in general, the OS (Operating System) is front and back depending on the size and position of the window of the application running on the information processing apparatus 110 and which window is active. Judge the window that becomes. Then, the OS calculates the display position of each window on the display 20, and the image information (pixels of each pixel) of the window corresponding to each pixel (hereinafter, also referred to as “display output position”) corresponding to the display position. Value) is output. In this case, the camera image window may be hidden behind the windows of other active app images.

In the present embodiment, by separating the display output positions of the camera image and the application image on the screen of the display 20, it is possible to virtually perform transparent display and always display the camera image.

Specifically, as shown in the screen 82 of the lower figure of FIG. 3, the pixels assigned as the display output position of the camera image (the pixels of the shaded ellipse in FIG. 3) and the pixels assigned as the display output position of the application image (the pixels). The white elliptical pixels in FIG. 3) are distributed at a predetermined ratio. FIG. 3 shows an example of 1: 1 distribution.

Therefore, as shown in FIG. 4, the display control unit 14 outputs the image information of the camera image 84 to the pixels assigned to the camera image 84, and the image information of the application image 86 is assigned to the application image 86. Output to pixels. As a result, the virtual transparent display screen 88 can be displayed with the resolutions of the camera image 84 and the application image 86 slightly lowered.

When there are a plurality of windows to be displayed as the application image, the size, position, and whether or not the windows are active are determined by the size, position, and activeness of each window among the pixels assigned to the application image, as described in the above figure of FIG. The display output position may be determined.

Further, the display control unit 14 determines the visibility of the application image displayed on the display 20 according to at least one of the number of faces and the number of lines of sight (hereinafter, referred to as “detection number”) delivered from the detection unit 12. Decrease. In the present embodiment, as shown in the screen 82 of the lower figure of FIG. 3, the pixels that are the display output positions are distributed between the camera image and the application image. Therefore, by changing this distribution ratio, the visibility of the application image is visible. Can be realized.

For example, the display control unit 14 increases the distribution ratio of the pixels that are the display output positions of the camera image as the number of detections increases. That is, the number of pixels for displaying the camera image is made larger than the number of pixels for displaying the window image.

FIG. 5 shows an example of the transparent display screen 88 in which the distribution ratio is changed. The transparent display screen 88A shows an example in which the distribution ratio of the pixels of the application image is higher than that of the pixels of the camera image, and the transparent display screen 88C shows an example in which the distribution ratio of the pixels of the camera image is higher than that of the pixels of the application image. The transparent display screen 88B is an example of 1: 1 distribution.

The information processing device 110 equipped with the screen display device 10 can be realized by, for example, the computer 40 shown in FIG. The computer 40 includes a CPU (Central Processing Unit) 41, a memory 42 as a temporary storage area, and a non-volatile storage unit 43. Further, the computer 40 includes an input / output device 44 such as an input device and a display 20, and an R / W (Read / Write) unit 45 that controls reading and writing of data to the storage medium 49. Further, the computer 40 includes a communication I / F (Interface) 46 connected to a network such as the Internet. The CPU 41, the memory 42, the storage unit 43, the input / output device 44, the R / W unit 45, and the communication I / F 46 are connected to each other via the bus 47.

The storage unit 43 can be realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The storage unit 43 as a storage medium stores a screen display program 50 for causing the computer 40 to function as the screen display device 10. The screen display program 50 has a detection process 52 and a display control process 54. The storage unit 43 also stores the programs of the OS and various applications operating in the information processing device 110, but the illustration is omitted in FIG.

The CPU 41 reads the screen display program 50 from the storage unit 43, expands it into the memory 42, and sequentially executes the processes included in the screen display program 50. The CPU 41 operates as the detection unit 12 shown in FIG. 2 by executing the detection process 52. Further, the CPU 41 operates as the display control unit 14 shown in FIG. 2 by executing the display control process 54. As a result, the computer 40 that has executed the screen display program 50 functions as the screen display device 10. The CPU 41 that executes the program is hardware.

The function realized by the screen display program 50 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.

Next, the operation of the screen display system 100 according to the present embodiment will be described. The display 20 and the camera 30 are activated in conjunction with the activation of the information processing device 110. The camera 30 captures a range including the operator 70 of the terminal and outputs the captured camera image. The operator 70 activates and operates the application to be operated by the information processing device 110. Then, the screen display device 10 executes the screen display process shown in FIG. 7. The screen display process is an example of a screen display method of the disclosed technology.

In step S12, the detection unit 12 and the display control unit 14 acquire the image information of the camera image output from the camera 30. Further, the display control unit 14 acquires the image information of the application image instructed to be displayed by the application.

Next, in step S14, the display control unit 14 assigns each image information of the acquired camera image and the application image as a display output position of each image at a predetermined initial distribution ratio (for example, 1: 1). Output to each of the generated pixels. As a result, the camera image and the application image are displayed on the display 20.

Next, in step S16, the detection unit 12 detects at least one of the face of the person and the line of sight to the display 20 from the acquired camera image, and passes the detected number of detections to the display control unit 14.

Next, in step S18, the display control unit 14 determines whether or not a peep is detected. The display control unit 14 determines that the peep is detected when at least one of the face and the line of sight other than the operator 70 is detected. For example, considering that the camera image also includes the face or line of sight of the operator 70, the display control unit 14 detects when two or more faces are detected and when two or more lines of sight are detected. If, at least in any case, it can be determined that a peep has been detected. If a peep is detected, the process proceeds to step S20, and if no peep is detected, the process proceeds to step S22.

In step S20, the display control unit 14 sets the pixel distribution ratio, which is the display output position of the camera image, to the pixel distribution ratio, which is the display output position of the application image, according to the number of detections passed from the detection unit 12. By making it higher, the visibility of the application image is lowered.

Next, in step S22, the display control unit 14 determines whether or not to end the display of the screen on the display 20. If the display is not finished, the process returns to step S12, and if the information processing device 110 is turned off, it is determined that the display is finished, and the screen display process is finished.

As described above, according to the screen display system according to the present embodiment, the screen display device displays a camera image including the vicinity of the operator of the information processing device 110 in the imaging range and an application image to be operated. Display on. Then, the screen display device reduces the visibility of the application image according to the number of at least one of the face and the line of sight detected from the camera image. As a result, the terminal operator can efficiently recognize the peep on the screen.

In addition, by reducing the visibility of the application image, it is possible to efficiently prevent peeping on the screen.

Further, it is not necessary to use a transmissive display by distributing the pixels assigned as the display output position of the camera image and the pixels assigned as the display output position of the application image to realize the virtual transparent display. Further, the transmittance can be adjusted more flexibly as compared with the case of the transmissive display in which the transmittance is adjusted by the light amount level, and the transmittance can be set to be easily visible to each operator.

In the above embodiment, an example of reducing the visibility of the application image according to the number of detected faces and lines of sight has been described, but the present invention is not limited to this. Regardless of the number of detections, when at least one of the face and the line of sight is detected in a predetermined number or more, the visibility of the application image may be reduced to a predetermined level.

Further, in the above embodiment, a case where the visibility of the application image is lowered by changing the distribution ratio of the pixels assigned as the display output position of the camera image and the pixels assigned as the display output position of the application image will be described. However, it is not limited to this.

For example, the camera image and the application image may be superimposed and displayed, and the transparency of the application image may be increased as the number of detections of at least one of the face and the line of sight increases. Further, as the number of detections of at least one of the face and the line of sight increases, at least one of the brightness, brightness, and contrast of the application image may be made smaller than the camera image. Further, as in the above embodiment, in addition to distributing the pixels to be assigned as the display output position, the visibility of the application image may be lowered by adjusting at least one of the brightness, the brightness, and the contrast. Good.

Further, in the above embodiment, the example of displaying the camera image on the entire screen has been described, but the display area of the camera image may be a part of the screen.

Further, in the above embodiment, the case where the visibility of the camera image is improved by reducing the visibility of the application image when the peep is detected has been described, but the present invention is not limited to this. An image that can be peeped without any problem, for example, a gadget of a clock, is designated in advance as an image that improves visibility when peeping is detected. Then, when the peep is detected, the visibility of the application image to be operated may be made lower than that of the designated image.

Further, in the above embodiment, the case where the information processing device is a fixed PC has been described as an example, but the disclosed technology can also be applied to a portable information processing device such as a tablet terminal or a smartphone. In this case, the peep may be detected by using the image captured by the in-camera as the camera image.

Further, in the above embodiment, the mode in which the screen display program is stored (installed) in the storage unit in advance has been described, but the present invention is not limited to this. The program according to the disclosed technology can also be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

The following additional notes will be further disclosed with respect to the above embodiments.

(Appendix 1)
On the computer
The first image captured by the imaging device including the vicinity of the operator of the terminal in the imaging range and the second image to be operated are displayed on the screen on the terminal.
At least the visibility of the second image is reduced according to at least one of the number of faces detected from the first image and the number of lines of sight to the screen detected from the first image.
A screen display program characterized by executing processing.

(Appendix 2)
In the process of lowering the visibility, when at least one of the number of faces and the number of lines of sight is a predetermined number or more, the number of pixels for displaying the first image is changed to the number of pixels for displaying the second image. The screen display program according to Appendix 1, wherein the processing is performed in excess of the number.

(Appendix 3)
The process of lowering the visibility is a process of increasing the number of pixels for displaying the first image as the number of at least one of the number of faces and the number of lines of sight increases. Alternatively, the screen display program described in Appendix 2.

(Appendix 4)
In the process of lowering the visibility, the first image and the second image are superimposed and displayed, and the larger the number of at least one of the number of faces and the number of lines of sight, the more the second image is displayed. The screen display program according to any one of Supplementary notes 1 to 3, wherein the process is for increasing the transparency of the image.

(Appendix 5)
The process of lowering the visibility is a process of reducing the brightness of the second image as the number of faces and at least one of the lines of sight increases. The screen display program according to any one of the above items.

(Appendix 6)
The process of lowering the visibility is a process of reducing the brightness of the second image as the number of faces and at least one of the lines of sight increases. The screen display program according to any one of the above items.

(Appendix 7)
The process of lowering the visibility is a process of reducing the contrast of the second image as the number of faces and at least one of the lines of sight increases. The screen display program according to any one of the above items.

(Appendix 8)
A display control unit that displays a first image captured by an imaging device that includes the vicinity of the operator of the terminal in the imaging range and a second image that is an operation target on a screen on the terminal.
A detection unit that detects at least one of the number of faces and the number of lines of sight with respect to the screen from the first image is included.
The display control unit lowers the visibility of at least the second image according to at least one of the number of faces and the number of lines of sight detected by the detection unit.
Screen display device.

(Appendix 9)
As a process for lowering the visibility, the display control unit sets the number of pixels for displaying the first image when at least one of the number of faces and the number of lines of sight is a predetermined number or more. The screen display device according to Appendix 8, wherein the number of pixels for displaying the image of the above is larger than the number of pixels.

(Appendix 10)
As a process for lowering the visibility, the display control unit is characterized in that the number of pixels for displaying the first image is increased as the number of faces and the number of lines of sight are increased. The screen display device according to Appendix 8 or Appendix 9.

(Appendix 11)
As a process for lowering the visibility, the display control unit superimposes and displays the first image and the second image, and the number of faces and the number of lines of sight are at least one of a large number. The screen display device according to any one of Supplementary note 8 to Supplementary note 10, wherein the transparency of the second image is increased.

(Appendix 12)
The display control unit is characterized in that, as a process for lowering the visibility, the brightness of the second image is reduced as the number of faces and at least one of the lines of sight are increased. -The screen display device according to any one of Appendix 11.

(Appendix 13)
The display control unit is characterized in that, as a process of lowering the visibility, the brightness of the second image is reduced as the number of faces and at least one of the lines of sight are increased. -The screen display device according to any one of Appendix 12.

(Appendix 14)
The display control unit is characterized in that, as a process for lowering the visibility, the contrast of the second image is reduced as the number of faces and at least one of the lines of sight are increased. -The screen display device according to any one of Appendix 13.

(Appendix 15)
The computer
A first image captured by an imaging device that includes the vicinity of the operator of the terminal in the imaging range and a second image to be operated are displayed on the screen on the terminal.
At least the visibility of the second image is reduced according to at least one of the number of faces detected from the first image and the number of lines of sight to the screen detected from the first image.
A screen display method characterized by executing a process.

(Appendix 16)
In the process of lowering the visibility, when at least one of the number of faces and the number of lines of sight is a predetermined number or more, the number of pixels for displaying the first image is changed to the number of pixels for displaying the second image. The screen display method according to Appendix 15, wherein the processing is performed in a number larger than the number.

(Appendix 17)
The process of lowering the visibility is a process of increasing the number of pixels for displaying the first image as the number of at least one of the number of faces and the number of lines of sight increases. Alternatively, the screen display method according to Appendix 16.

(Appendix 18)
In the process of lowering the visibility, the first image and the second image are superimposed and displayed, and the larger the number of at least one of the number of faces and the number of lines of sight, the more the second image is displayed. The screen display method according to any one of Supplementary note 15 to Supplementary note 17, wherein the process is for increasing the transparency of the image.

(Appendix 19)
The process of lowering the visibility is a process of reducing at least one of the brightness, brightness, and contrast of the second image as the number of faces and at least one of the lines of sight increases. The screen display method according to any one of Supplementary note 15 to Supplementary note 18, wherein the screen is displayed.

(Appendix 20)
On the computer
The first image captured by the imaging device including the vicinity of the operator of the terminal in the imaging range and the second image to be operated are displayed on the screen on the terminal.
At least the visibility of the second image is reduced according to at least one of the number of faces detected from the first image and the number of lines of sight to the screen detected from the first image.
A storage medium that stores a screen display program characterized by executing a process.

10 Screen display device 12 Detection unit 14 Display control unit 20 Display 30 Camera 40 Computer 41 CPU
42 Memory 43 Storage unit 49 Storage medium 50 Screen display program 100 Screen display system 110 Information processing device

Claims (9)

On the computer
The first image captured by the imaging device including the vicinity of the operator of the terminal in the imaging range and the second image to be operated are displayed on the screen on the terminal.
At least the visibility of the second image is reduced according to at least one of the number of faces detected from the first image and the number of lines of sight to the screen detected from the first image.
A screen display program characterized by executing processing. In the process of lowering the visibility, when at least one of the number of faces and the number of lines of sight is a predetermined number or more, the number of pixels for displaying the first image is changed to the number of pixels for displaying the second image. The screen display program according to claim 1, wherein the number of processes is larger than the number of processes. The process of lowering the visibility is a process of increasing the number of pixels for displaying the first image as the number of at least one of the number of faces and the number of lines of sight increases. The screen display program according to 1 or 2. In the process of lowering the visibility, the first image and the second image are superimposed and displayed, and the larger the number of at least one of the number of faces and the number of lines of sight, the more the second image is displayed. The screen display program according to any one of claims 1 to 3, wherein the process is for increasing the transparency of the image. Claims 1 to claim that the process of lowering the visibility is a process of reducing the brightness of the second image as the number of faces and at least one of the lines of sight increases. Item 4. The screen display program according to any one of items 4. Claims 1 to claim that the process of lowering the visibility is a process of reducing the brightness of the second image as the number of faces and at least one of the lines of sight increases. Item 5. The screen display program according to any one of items 5. Claims 1 to claim that the process of lowering the visibility is a process of reducing the contrast of the second image as the number of faces and at least one of the lines of sight increases. Item 6. The screen display program according to any one of items 6. A display control unit that displays a first image captured by an imaging device that includes the vicinity of the operator of the terminal in the imaging range and a second image that is an operation target on a screen on the terminal.
A detection unit that detects at least one of the number of faces and the number of lines of sight with respect to the screen from the first image is included.
The display control unit lowers the visibility of at least the second image according to at least one of the number of faces and the number of lines of sight detected by the detection unit.
Screen display device. The computer
A first image captured by an imaging device that includes the vicinity of the operator of the terminal in the imaging range and a second image to be operated are displayed on the screen on the terminal.
At least the visibility of the second image is reduced according to at least one of the number of faces detected from the first image and the number of lines of sight to the screen detected from the first image.
A screen display method characterized by executing a process.