JP7474304B1 - Display system, electronic device, and calibration method - Google Patents

Abstract

[Problem] The size of a non-display area in which pixels are not lit is set to the minimum necessary. [Solution] Light that has passed through a transparent area that is a part of a display panel in which a plurality of pixels are arranged is incident on a camera. A control unit causes a first pixel arranged in a blinking area that is a part of the transparent area to emit light and a second pixel arranged in a non-blinking area that is the other part of the transparent area to not emit light, causes the camera to capture a first image, and causes the camera to capture a second image without emitting light from the first pixel and the second pixel, detects a difference area in which a significant difference occurs between the first image and the second image from a captured image area that is the entire area of the captured image, and determines an area in the transparent area that corresponds to the captured image area as a non-display area in which pixels are not lit when the main purpose is to display the image captured by the camera based on the correspondence between the difference area and the blinking area. [Selected Figure] Figure 10

Description

This application relates to a display system, electronic device, and calibration method, for example, a display device equipped with a camera.

Some display devices are equipped with a camera. The camera may be installed on the back side of a transparent area that constitutes a part of the display surface. An image is displayed using these pixels. In the transparent area, some of the light projected from the subject passes through the gaps between the individual pixels. Such display devices are sometimes called CUD (Camera Under Display) systems. The transparent area includes at least a portion that transmits light that is incident within the field of view (FOV) of the camera.

For example, Patent Document 1 describes an OLED array substrate, a display panel, and a display device that include a first display region and a second display region. The first display region is adjacent to the second display region and includes first OLED pixels arranged in an array. The second display region includes second OLED pixels arranged in an array. The pixel density of the second OLED pixels is smaller than the pixel density of the first OLED pixels. A camera for capturing images is disposed on the back of the second display region.

When the camera is operated in a CUD system, the pixels arranged in the transparent area are turned off and no image is displayed. The area in which no image is displayed is also called a non-display area (black-out area). Making the area non-display prevents light emitted from the pixels from entering the camera. This is because if light emitted by the pixels were to enter the camera, a clear image would not be obtained.

JP 2022-21644 A

In information devices equipped with a CUD system, it is sometimes desirable to reduce the size of the non-display area. Reducing the size of the non-display area is desirable mainly in the following cases: when multiple windows are displayed at once, when a landscape window is used, when the design of the user interface (UI) is restricted by the constraints of the OS (Operating System), etc. Note that a landscape window is a horizontally long window whose vertical height is shorter than its horizontal height, unlike a portrait window which is vertically long.

The present disclosure has been made to solve the above problems, and a display system according to a first aspect includes a display panel in which a plurality of pixels are arranged, a camera, and a control unit, and light that has passed through a transparent region that is a part of the display panel is incident on the camera. The control unit causes the camera to capture a first image by emitting light from a first pixel arranged in a blinking region that is a part of the transparent region and not emitting light from a second pixel arranged in a non-blinking region that is the other part of the transparent region, and causes the camera to capture a second image without emitting light from the first pixel and the second pixel. A difference region is detected in a captured image region that is the entire region of the captured image, where a significant difference occurs between the first image and the second image, and based on the correspondence between the difference region and the blinking region, a region in the transparent region that corresponds to the captured image region is defined as a non-display region in which pixels are not lit when the main purpose is to display the image captured by the camera.

In the above display system, the control unit may cause pixels that are arranged in the blinking region and that are capable of emitting a predetermined color to emit light as the first pixels, and may define an area in which pixels that cause a significant difference in the signal value of the predetermined color are arranged as the difference area.

In the above display system, the control unit may cause the camera to capture the first image and the second image multiple times, accumulate the pixel-by-pixel differences between the first image and the second image to calculate a cumulative difference, and define as the difference region an area in which pixels whose absolute value of the cumulative difference exceeds a predetermined difference threshold are located.

In the above display system, the control unit may detect a partial image area, which is an area of the image defined by the boundary between a non-difference area, which is an area where the significant difference does not occur, and the difference area, calculate the displacement from the center of gravity of the partial image area to the center of gravity of the captured image area, and the ratio of the diameter of the non-blinking area to the diameter of the partial image area, and determine the non-display area based on the displacement from the non-blinking area and the ratio.

In the above display system, the shape of the flashing area may be annular and surround the outer edge of the non-flashing area.

In the above display system, the shape of the outer edge of the non-blinking area may be rectangular.

The electronic device according to the second aspect may include the display system described above.

The calibration method according to the third aspect is a calibration method for a display system including a display panel on which a plurality of pixels are arranged and a camera, in which light that has passed through a transparent region that is a part of the display panel is incident on the camera, and the display system executes the following steps: causing the camera to capture a first image by emitting light from a first pixel arranged in a blinking region that is a part of the transparent region and not emitting light from a second pixel arranged in a non-blinking region that is the other part of the transparent region; causing the camera to capture a second image without emitting light from the first pixel and the second pixel; detecting a difference region, which is a region where a significant difference occurs between the first image and the second image, from a captured image region that is the entire region of the captured image; and determining, based on the correspondence between the difference region and the blinking region, a region in the transparent region that corresponds to the captured image region as a non-display region in which pixels are not lit when the main purpose is to display the image captured by the camera.

According to this embodiment, the size of the non-display area in which no pixels are lit can be set to the minimum necessary.

1 is a front view showing an example of an external configuration of an electronic device according to an embodiment of the present invention; 2 is a cross-sectional view showing an example of the internal configuration of the electronic device according to the embodiment. FIG. 2 is a schematic block diagram illustrating an example of a hardware configuration of the electronic device according to the embodiment. FIG. 2 is a schematic block diagram illustrating an example of a functional configuration of the electronic device according to the present embodiment. FIG. 2 is a diagram showing an example of a captured image. 11A and 11B are diagrams illustrating examples of setting a blinking area and a non-blinking area according to the embodiment. 1A and 1B are diagrams illustrating a first image and a second image according to the present embodiment. 13A and 13B are diagrams illustrating other examples of setting the blinking area and the non-blinking area according to the embodiment. 11 is an explanatory diagram for explaining a method for setting a non-display area according to the embodiment. FIG. 10 is a flowchart illustrating a calibration process of a non-display area according to the embodiment.

Embodiments of the present disclosure will now be described with reference to the drawings. FIG. 1 is a front view showing an example of the external configuration of an electronic device 1 according to this embodiment. The electronic device 1 illustrated in FIG. 1 is configured as a notebook personal computer (notebook PC: Personal Computer). The electronic device 1 includes a display system 1d. The display system 1d includes a display panel 14, a camera module 35, and a control unit 100 (described below). The control unit 100 is built into the housing BD01 or BD03, and is not shown in FIG. 1.

A display panel 14 is provided on the main surface of the housing BD01. A camera module 35 is provided on the rear surface of the display panel 14. A keyboard 34k and a touch sensor 34t are provided on the main surface of the housing BD03. One side of each of the housings BD01 and BD03 is connected using hinge mechanisms 121a and 121b. One of the housings BD01 and BD03 can rotate relative to the other around that one side as a rotation axis.

The display panel 14 is a display device having a substantially flat plate shape. The display panel 14 has a plurality of pixels and a substrate, and the plurality of pixels are arranged two-dimensionally at regular intervals in a display area that constitutes most of the main surface of the substrate. A transmissive area CA is provided in a part of the display area. In the transmissive area CA, unlike the standard area that is the surrounding area, gaps are provided between the pixels. Light that arrives at the transmissive area CA passes through the gaps between the pixels, and a part of the light is incident on the camera module 35. The transmissive area CA is also called the camera under display area (CUD).

Light that has passed through the transparent area CA is incident on the camera module 35, and an image is captured by projecting the incident light onto the imaging surface. A non-display area (Black-out Area) BA is provided in a part of the transparent area CA. When the main purpose is to display the image captured by the camera module 35, the pixels arranged in the non-display area BA are not lit. In that case, the image captured by the camera module 35 is not displayed in the non-display area BA. By turning off the pixels arranged in the non-display area BA, the phenomenon of light emitted from the pixels being incident on the camera module 35 is prevented. The contrast is low and the contour of the subject is not obscured as in an image captured without the non-display area BA (see FIG. 5). Note that "when the main purpose is to display the captured image" means a case where the original purpose is to display the image, and does not include a case where the minimum area MBA is set as the non-display area BA as described below.

The transmission area CA is usually set to be larger than the range corresponding to the field of view of the camera module 35 (hereinafter, sometimes referred to as the "projected field of view area" (FOV)). This is because even a slight change in the positional relationship between the display panel 14 and the camera module 35 may cause light from nearby pixels to be incident on the camera module 35. As illustrated in FIG. 2, the camera module 35 is attached to the back surface of the display panel 14 using adhesive 35a, and is supported by the side of the jig 35j together with the display panel 14. The surface of the display panel 14 is covered with a protective film 14g. The protective film 14g is made of a transparent material such as glass. However, a mask print 14m is further applied to the surface of the jig 35j. This is to cover and conceal the jig 35j with the mask print 14m. In the example of FIG. 2, the mask print 14m is painted black.

In the comparative example, the entire transparent area CA was sometimes set as the non-display area BA. It is sufficient that the non-display area BA includes at least an area MBA (corresponding to a projection field area, sometimes referred to as a "minimum area (Minimum Black-out Area)" in the following description) that is defined by a field of view FOV (Field of View) centered on the optical axis OA from the optical center OC of the camera module 35 in the display area. The field of view FOV corresponds to the range in which transmitted light from the transparent area CA is incident on the optical system of the camera module 35. Therefore, the size of the non-display area BA tends to be larger than the minimum necessary size. The non-display area BA does not display any image and has a black spotted appearance. In contrast, in this embodiment, the control unit 100 executes the procedure described below to determine the minimum area MBA as the non-display area BA.

Next, an example of the hardware configuration of the electronic device 1 according to this embodiment will be described. FIG. 3 is a schematic block diagram showing an example of the hardware configuration of the electronic device 1 according to this embodiment. The electronic device 1 includes a CPU 11, a main memory 12, a video subsystem 13, a display panel 14, a chipset 21, a BIOS (Basic Input Output System) memory 22, an SSD (Solid State Drive) 23, a USB (Universal Serial Bus) connector 24, an audio system 25, a network card 26, an RTC (Real Time Clock) 27, an embedded controller (EC: Embedded Controller) 30, a power supply circuit 31, a battery 33, an input I/F (Interface) 34, and a camera module 35.

The CPU 11 is a central processing unit that controls the functions of each part of the electronic device 1. The CPU 11 executes processing instructed by commands written in a program deployed in the main memory 12, thereby realizing the functions. Note that in this application, executing processing instructed by commands written in a program may be referred to as "executing a program" or "running a program."

The main memory 12 has a read area for various programs executed by the CPU 11. The main memory 12 has a work area for storing various data acquired by the CPU 11 and various data used in the processes executed by the CPU 11. The programs executed by the CPU 11 include, for example, an OS (Operating System), various drivers for operating peripheral devices, various services/utilities, and application programs (sometimes referred to as "apps" in this application). The CPU 11 and the main memory 12 correspond to a system device that constitutes the computer system of the electronic device 1.

The video subsystem 13 performs drawing processing under the control of the CPU 11 to generate various display data and outputs it to the display panel 14. The video subsystem 13 includes, for example, a video controller and a video memory. The video controller generates drawing information according to drawing commands input from the CPU 11 and writes the generated drawing information to the video memory. The video controller reads the drawing information from the video memory and outputs display data indicating the read drawing information to the display panel.
The display panel 14 displays a display screen instructed by display data input from the video subsystem.

Various devices are connected to the chipset 21, which controls the operation or input/output of each device. The chipset 21 includes controllers for various data buses. For example, the chipset 21 includes a Universal Serial Bus (USB), a serial AT Attachment (Serial ATA), a Serial Peripheral Interface (SPI) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express bus, and a Low Pin Connect (LPC) bus. In the example of FIG. 3, the chipset 21 is connected to a BIOS memory 22, an SSD 23, a USB connector 24, an audio system 25, a network card 26, an RTC 27, an embedded controller 30, and a camera module 35.

The BIOS memory 22 stores system firmware used to control various devices such as the BIOS and the embedded controller 30. The BIOS memory 22 includes an electrically rewritable non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read Only Memory) or a flash ROM.
The SSD 23 is an example of an auxiliary storage device. The SSD 23 stores various data in addition to programs such as an OS, various drivers, various services/utilities, and apps. The electronic device 1 may be provided with another type of auxiliary storage device such as a hard disk drive (HDD) instead of the SSD 23 or in addition to the SSD 23.

The audio system 25 records, encodes, decodes, and inputs/outputs various types of audio data.
The network card 26 is connected to a network wirelessly or via a wire to perform data communication. The network card 26 is connected to other devices directly or via another network, for example, via a wireless LAN (Local Area Network).
The RTC 27 measures the date and time at that time. The RTC 27 consumes power supplied from a backup battery (not shown). The backup battery is provided separately from the battery 33. Therefore, the RTC 27 can constantly measure the date and time even if the supply of power from the power supply circuit 31 is stopped due to a shutdown of the electronic device 1.

The embedded controller 30 has a power management function that controls the power circuit 31 and an input control function for input data from the input I/F 34. As the power management function, the embedded controller 30 controls the power circuit 31 based on the operating state notified from the system device, and controls the power supplied to each device including the CPU 11. As the input control function, the embedded controller 30 controls whether or not input data is required from the input I/F 34. The embedded controller 30 is a microcomputer having a CPU and main memory that are separate from the CPU 11 and main memory 12. The embedded controller 30 is capable of steady operation regardless of the operating state of the CPU 11 and main memory 12.

The power supply circuit 31 controls the supply of power to each device according to the control of the embedded controller 30. The power supply circuit 31 includes, for example, a DC (Direct Current)/DC converter, a charge/discharge unit, and an AC (Alternating Current)/DC adapter. The DC/DC converter converts the voltage of the direct current power supplied from the charge/discharge unit to a voltage required by each device, and supplies the converted voltage to the device. The charge/discharge unit controls charging the battery 33 with the power supplied from the AC/DC adapter or discharging the battery 33 from the battery. The charge/discharge unit supplies, for example, a part or all of the power supplied from the AC/DC adapter to the DC/DC converter. The charge/discharge unit charges the battery 33 with the surplus power that is not consumed among the power supplied from the AC/DC adapter. When power is not supplied from the AC/DC adapter or when the power from the AC/DC adapter is insufficient, the charge/discharge unit supplies the power discharged from the battery 33 to the DC/DC converter. The AC/DC adapter converts AC power supplied from an external commercial power source into DC power, and supplies the converted DC power to the charge/discharge unit.

The battery 33 is a storage battery that can be charged with power supplied from the power supply circuit 31 and that discharges the charged power under the control of the power supply circuit 31 .
The input I/F 34 includes an input device that receives a user's operation and acquires operation data in response to the received operation. The keyboard 34k and the touch sensor 34t illustrated in Fig. 1 constitute a part of the input I/F 34. The input I/F 34 may include a separate touch sensor separate from the touch sensor 34t or in place of the touch sensor 34t. The separate touch sensor may be integrated with the display panel 14 so as to overlap with it, forming a touch panel.

The camera module 35 captures an image represented by light incident on the optical system. The camera module 35 outputs image data representing the captured image to the CPU 11 via the chip set 21. In this embodiment, light that has passed through a portion of the transmissive area CA is incident on the optical system of the camera module 35.

Next, an example of the functional configuration of the electronic device 1 according to the present embodiment will be described. Fig. 4 is a schematic block diagram showing an example of the functional configuration of the electronic device 1 according to the present embodiment. The electronic device 1 has a control unit 100, a storage unit 200, a display panel 14, and a camera module 35.
Control unit 100 sets a non-display area BA on display panel 14. The functions of control unit 100 are realized by any one of a graphic controller, video subsystem 13, and CPU 11, or a combination of any of these controllers. The graphic controller may be associated with display panel 14 and may be primarily intended to control the brightness, color tone, timing, etc. of an image displayed on display panel 14.
A part or all of the functions of the control unit 100 are realized by the graphic controller or the video subsystem 13 executing a predetermined program. A part or all of the functions of the control unit 100 may be realized by the CPU 11 executing a device driver or a predetermined application program dedicated to the graphic controller or the video subsystem 13.

The storage unit 200 stores programs executed by the control unit 100, data used by the control unit 100, data generated by the control unit 100, and the like. The storage unit 200 is realized using a storage medium (memory) provided in a controller that executes the functions of the control unit 100. When the CPU 11 executes the functions, the functions of the storage unit 200 may be realized by an auxiliary storage medium such as the main memory 12 or SSD 23.

The control unit 100 includes a display control unit 102, an imaging control unit 104, an image analysis unit 106, and a setting processing unit 108.
The display control unit 102 blinks pixels arranged in the blinking region BL in the display region of the display panel 14. The display control unit 102 repeats a light-on period and an off period in the blinking of the pixels. The light-on period and the off period are each set in advance in the display control unit 102 to a period sufficient for capturing one frame of image (for example, 0.1 to 1.0 seconds). A part of the transparent region CA is set in advance in the display control unit 102 as the blinking region BL. The non-blinking region TA corresponds to the region of the transparent region CA other than the blinking region BL. The display control unit 102 keeps the pixels in the non-blinking region TA off regardless of whether the current time is within the light-on period or the off period.

The imaging control unit 104 causes the camera module 35 to capture an image during the light-on period, and acquires first image data showing the captured image as a first image. The imaging control unit 104 causes the camera module 35 to capture an image during the light-off period, and acquires second image data showing the captured image as a second image. The imaging control unit 104 may repeat the process of acquiring the first image data and the second image data for each control cycle consisting of the light-on period and the light-off period. The imaging control unit 104 stores the acquired first image data and second image data in the storage unit 200.

The image analysis unit 106 reads out the first image data and the second image data stored in the memory unit 200. The image analysis unit 106 detects an area where a significant difference occurs between the luminance of the first image shown in the read out first image data and the luminance of the second image shown in the second image data as a difference area. The image analysis unit 106 calculates the difference between the luminance value representing the first image and the luminance value representing the second image for each pixel, and can detect an area having pixels where the absolute value of the calculated difference is equal to or greater than a predetermined difference threshold as a difference area.

Since the first image is an image captured during the light-on period and the second image is an image captured during the light-off period, the portion of the field of view FOV of the camera module 35 that corresponds to the blinking region BL is detected as the difference region. On the other hand, in the portion of the field of view FOV of the camera module 35 that corresponds to the non-blinking region TA, no significant difference in brightness occurs between the first image and the second image. The image analysis unit 106 stores difference region data indicating the detected difference region in the memory unit 200.

The setting processing unit 108 reads out the differential area data stored in the storage unit 200 and identifies the differential area indicated by the read out differential area data. Based on the geometric relationship between the identified differential area and the preset blinking area BL, the setting processing unit 108 determines an area in the transparent area CA corresponding to the captured image area forming the entire first image or the second image as a non-display area BA. The display panel 14 turns off the pixels arranged in the non-display area BA and does not display the display image indicated in the display data input from the video subsystem 13. The setting processing unit 108, for example, outputs non-display area information indicating the determined non-display area BA to the display control unit 102. The display control unit 102 sets a luminance value (for example, 0) corresponding to turning off for each pixel arranged in the non-display area BA indicated by the non-display area information of the display image, and maintains the luminance value of each pixel other than the non-display area BA unchanged. The display control unit 102 outputs display data indicating the luminance value of each pixel to the display panel 14. The setting processing unit 108 may stop supplying power to pixels arranged in the non-display area BA among the pixels arranged in the display area of the display panel 14, to turn them off. In this case, the setting processing unit 108 does not need to set luminance values for the pixels in the non-display area BA to the display control unit 102.

Next, an example of setting the blinking area BL and the non-blinking area TA according to this embodiment will be described. FIG. 6 is a diagram showing an example of setting the blinking area BL and the non-blinking area TA according to this embodiment. The blinking area BL is set in advance in the display control unit 102 so that a part or all of it and a part or all of the non-blinking area TA are included in an area corresponding to the known field of view FOV of the camera module 35. In the example of FIG. 6, the shape of the blinking area BL is a ring that surrounds the outer edge of the non-blinking area TA. The outer edges of the blinking area BL and the non-blinking area TA are each rectangular. The outer edge of the blinking area BL includes an area corresponding to the field of view FOV, and the inner edge of the blinking area BL or the outer edge of the non-blinking area TA is included in the area corresponding to the field of view FOV. With this arrangement, the image captured by the camera module 35 includes an area corresponding to the blinking area BL and an area corresponding to the non-blinking area TA.

Next, an example of the first image and the second image according to this embodiment will be described. Figures 7(a) and (b) respectively illustrate the first image and the second image. The illustrated first image and the second image show a common subject and background. The first image is captured during the light emission period of the pixels in the blinking area BL. Therefore, a significant difference in luminance occurs between the first image and the second image in the portion corresponding to the blinking area BL. In contrast, a significant difference in luminance does not occur between the first image and the second image in the portion corresponding to the non-blinking area TA. Therefore, based on the geometric relationship between the difference area where a significant difference in luminance occurs and the blinking area BL in the field of view FOV, the area in the transparent area CA corresponding to the captured image area, which is the entire first image or the second image, is determined as the minimum area MBA. By adopting this minimum area MBA as the non-display area BA, the size of the non-display area BA becomes the minimum required.

Next, other setting examples of the blinking area BL and the non-blinking area TA will be described. FIG. 8 is a diagram showing other setting examples of the blinking area BL and the non-blinking area TA according to this embodiment. The non-blinking area TA illustrated in FIG. 8 corresponds to the minimum area MBA corresponding to the field of view FOV in the display area of the display panel 14. The captured image area, which is the entire first image or the second image acquired from the camera module 35, corresponds to the non-blinking area TA. In this example, since the non-blinking area TA corresponds to the non-display area BA, the difference area corresponding to the blinking area BL set around the non-blinking area TA is not detected. However, even if the non-blinking area TA is larger than the non-display area BA, the difference area is not detected. Therefore, the setting processing unit 108 cannot immediately determine the non-blinking area TA as the non-display area BA just because it cannot detect the difference area. Therefore, when the setting processing unit 108 cannot detect the difference area, it sets a smaller non-blinking area TA. Then, the image analysis unit 106 determines whether or not a difference area is detected from the first image and the second image captured after the non-blinking area TA is set. When the setting processing unit 108 is successful in detecting the difference region, it can determine the non-display area BA based on the detected difference region.

Next, a specific example of a method for setting the non-display area BA according to this embodiment will be described. Fig. 9 is an explanatory diagram for explaining a method for setting the non-display area BA according to this embodiment. In the example of Fig. 9, it is assumed that the width and height of the partial image area corresponding to the non-blinking area TA are w1 and h1, the width and height of the captured image area are w2 and h2, the displacement from the center of gravity of the partial image area to the center of gravity of the captured image area is (Δx, Δy), and the coordinates of the width, height, and center of gravity of the non-blinking area TA at that time are W, H, and (X, Y), respectively.
In this case, the setting processing unit 108 sets the width, height, and center of the non-display area BA to W w2/w1, H h2/h1, and (X+Δx W/w1, Y+Δy H/h1), respectively.
The partial image region corresponds to a region of the captured image region that is defined by the boundary between a difference region and a non-difference region in which no significant difference occurs in luminance.

That is, the width of the non-display area BA is obtained by multiplying the width of the non-blinking area TA by the ratio of the width of the captured image area to the width of the partial image area. The height of the non-display area BA is obtained by multiplying the height of the non-blinking area TA by the ratio of the height of the captured image area to the height of the partial image area. The center coordinate of the non-display area BA is obtained by adding the displacement on the transparent area CA obtained by converting the displacement from the center coordinate of the partial image area to the center of the captured image area. The displacement on the transparent area CA is obtained by multiplying the horizontal and vertical displacements from the center of the partial image area to the center of the captured image area by the ratio of the width of the non-blinking area TA to the width of the partial image area and the ratio of the height of the non-blinking area TA to the height of the partial image area. Note that when determining the non-display area BA, the setting processing unit 108 may use a method that further takes into account requirements such as lens distortion correction and lens focal length in the camera module 35.

Next, an example of the calibration process of the non-display area BA according to the present embodiment will be described below. Fig. 10 is a flowchart illustrating the calibration process of the non-display area BA according to the present embodiment.
(Step S102) The display control unit 102 lights up the pixels arranged in the blinking region BL during the lighting period.
(Step S104) The shooting control unit 104 causes the camera module 35 to shoot a first image during the lighting period.

(Step S106) The display control unit 102 turns off the pixels arranged in the blinking region BL during the off period.
(Step S108) The shooting control unit 104 causes the camera module 35 to shoot a second image during the light-out period.

(Step S110) The display control unit 102 increments the number of times the blinking of the blinking area BL is repeated by 1, and counts the number of times the blinking is repeated. Before starting the process of FIG. 10, the display control unit 102 sets the number of times the blinking is repeated to 0 as an initial value.
The display control unit 102 determines whether the counted number of repetitions has reached a predetermined reference value for the number of repetitions (e.g., 5 to 10 times). If it is determined that the number of repetitions has reached the reference value (step S110: YES), the process proceeds to step S112. If it is determined that the number of repetitions has not reached the reference value (step S110: NO), the process returns to step S102.

(Step S112) The image analysis unit 106 calculates a cumulative difference by accumulating the difference obtained by subtracting the brightness value of the second image from the brightness value of the first image during repetition of blinking. The image analysis unit 106 identifies pixels whose absolute value of the cumulative difference exceeds a predetermined difference threshold, and detects an area including the identified pixel as a difference area.
(Step S114) The image analysis unit 106 determines the area within the transparent area CA relative to the captured image area as the non-display area BA based on the correspondence between the detected difference area and the blinking area. Then, the process of FIG. 10 ends.

In the above description, it is assumed that all pixels included in the blinking region BL are blinked simultaneously, but this is not limited to the above. If pixels that emit light of multiple colors are arranged on the display panel 14, the display control unit 102 may blink pixels capable of emitting light of a specific color and keep pixels capable of emitting light of other colors off. By setting a color that is unlikely to be used in the captured image (e.g., red) as the color to be blinked by pixels, it is possible to accurately detect difference regions where significant differences in luminance occur due to blinking.

The electronic device 1 is not limited to a notebook PC, and may be configured as other types of information devices, such as a tablet terminal device or a mobile phone. The display system 1d may be configured as a display device in which the display panel 14, the camera module 35, and a graphics controller are integrated. In the display system 1d, the controller, which is a part of the display system 1d, may be configured separately from the display panel 14 and the camera module 35.

In the above description, the blinking area BL has an annular shape surrounding the non-blinking area TA, but this is not limiting. As long as at least a part of the blinking area BL and at least a part of the non-blinking area TA are included within the field of view FOV of the camera module 35, and a geometric relationship can be derived between the blinking area BL and the difference image obtained from the first image and the second image, the shapes, sizes, positions, and numbers of the blinking area BL and the non-blinking area TA, as well as the presence or absence of an inclusion relationship between the blinking area BL and the non-blinking area TA, are arbitrary. For example, a blinking area BL consisting of one pixel may be set at each vertex of a triangle in the field of view of the camera module 35, and the other areas may be set as non-blinking areas TA.

As described above, the display system 1d according to the present embodiment includes a display panel 14 in which a plurality of pixels are arranged, a camera (for example, a camera module 35), and a control unit 100. Light transmitted through a transparent area CA, which is a part of the display panel 14, is incident on the camera. The control unit 100 causes the camera to capture a first image by emitting light from a first pixel arranged in a blinking area BL, which is a part of the transparent area CA, and not emitting light from a second pixel arranged in a non-blinking area TA, which is the other part of the transparent area CA, and causes the camera to capture a second image without emitting light from the first pixel and the second pixel. The control unit 100 detects a difference area, which is an area where a significant difference occurs between the first image and the second image, from the captured image area, which is the area of the entire image to be captured, and determines an area in the transparent area corresponding to the captured image area as a non-display area BA in which pixels are not lit when the main purpose is to display the image captured by the camera, based on the correspondence between the difference area and the blinking area.
According to this configuration, by using the relationship between the difference region in which a significant difference in luminance occurs due to the blinking of the first pixel arranged in the blinking region BL and the known blinking region BL, the region in the field of view of the camera corresponding to the captured image region in the transparent region CA can be defined as the non-display region BA. Therefore, the non-display region BA can be limited to the minimum size necessary to prevent light from entering the camera from the pixels.

In addition, the control unit 100 may cause a pixel among the pixels arranged in the flashing area BL that is capable of emitting a predetermined color (e.g., red) to emit light as a first pixel, and define an area in which pixels in which a significant difference occurs in the signal value of the predetermined color are arranged as a differential area.
According to this configuration, the difference region is determined based on the difference in luminance of a predetermined color between the first image and the second image, and therefore, by adopting a color that is unlikely to be used in the second image as the predetermined color, the difference region can be accurately detected.

In addition, the control unit 100 may cause the camera to capture the first image and the second image multiple times, accumulate the differences for each pixel between the first image and the second image to calculate a cumulative difference, and define as the difference region an area in which pixels whose absolute value of the cumulative difference exceeds a predetermined difference threshold are located.
According to this configuration, an area where the accumulated difference obtained by accumulating the differences between the first image and the second image captured multiple times is significant is obtained as the difference area. Since the noise components that may occur in each capture are relatively reduced compared to the signal components in the accumulated difference, the difference area can be accurately detected.

In addition, the control unit 100 may detect a partial image area, which is an area of an image defined by the boundary between a non-difference area, which is an area where no significant difference occurs between the first image and the second image, and a difference area, calculate the displacement from the center of gravity of the partial image area to the center of gravity of the captured image area and the ratio of the diameter of the non-blinking area to the diameter of the partial image area, and determine a non-display area BA from the non-blinking area based on the calculated displacement and ratio.
With this configuration, the non-display area BA can be determined by a simple calculation using the displacement from the center of gravity of the partial image area to the center of gravity of the captured image area and the ratio of the diameter of the non-blinking area TA to the diameter of the partial image area.

The blinking area BL may also be shaped like a ring that surrounds the outer edge of the non-blinking area TA.
According to this configuration, the field of view of the camera can be set to include at least a part of the blinking area BL and the entire non-blinking area TA, so that the non-display area BA can be easily set based on the partial image area and the non-blinking area TA.

The shape of the outer edge of the non-blinking area may be oblong (eg, rectangular).
According to this configuration, the width and height can be used as the diameter of the partial image area and the diameter of the non-display area, respectively, so that the non-display area BA can be determined by a simple calculation.

This embodiment may be configured as a display device including a graphic controller having the functions of the control unit 100 and a display panel 14 on which the first pixel and the second pixel are arranged. The display device may include a camera module 35 on the back surface of the transmissive area CA.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configurations are not limited to the above-mentioned embodiments, and include designs within the scope of the gist of the present invention. The configurations described in the above-mentioned embodiments can be combined in any manner.

1...electronic device, 1d...display system, 12...main memory, 13...video subsystem, 14...display panel, 21...chipset, 22...BIOS memory, 23...SSD, 24...USB connector, 25...audio system, 26...network card, 27...RTC, 30...embedded controller, 31...power supply circuit, 33...battery, 34...input I/F, 35...camera module, 100...control unit, 102...display control unit, 104...photography control unit, 106...image analysis unit, 108...setting processing unit, 200...storage unit, CA...transparent area, BA...non-display area, BL...flashing area, TA...non-flashing area

Claims (8)

The display device includes a display panel having a plurality of pixels arranged thereon, a camera, and a control unit.
The camera receives light that has passed through a transmissive region that is a part of the display panel,
The control unit is
causing a first pixel arranged in a blinking region that is a part of the transparent region to emit light and causing a second pixel arranged in a non-blinking region that is the other part of the transparent region not to emit light to cause the camera to capture a first image;
causing the camera to capture a second image without emitting light from the first pixel and the second pixel;
A difference region is detected, which is a region where a significant difference occurs between the first image and the second image, from a captured image region which is a region of the entire captured image;
Based on the correspondence between the difference region and the blinking region, a region in the transparent region that corresponds to the captured image region is determined as a non-display region in which pixels are not lit , and the image captured by the camera is displayed on the display panel.
Display system. The control unit is
a pixel that is capable of emitting a predetermined color among the pixels arranged in the blinking region is caused to emit light as the first pixel;
The display system according to claim 1 , wherein a region in which pixels having a significant difference in signal value of the predetermined color are arranged is defined as the difference region. The control unit is
causing the camera to capture the first image and the second image a plurality of times;
accumulating the differences between the first image and the second image to calculate an accumulated difference;
The display system according to claim 1 , wherein a region in which pixels having an absolute value of the accumulated difference exceeding a predetermined difference threshold value are disposed is defined as the difference region. The control unit is
A partial image region is detected, which is a region of an image defined by a boundary between a non-difference region, which is a region in which the significant difference does not occur, and the difference region;
A displacement of the center of gravity of the partial image area to the center of gravity of the captured image area, and
Calculating a ratio of a diameter of the non-blinking region to a diameter of the partial image region;
The display system of claim 1 , further comprising: determining the non-display area based on the displacement from the non-blinking area and the ratio. The display system according to claim 4 , wherein the blinking area has an annular shape surrounding an outer edge of the non-blinking area. The display system according to claim 4 , wherein the outer edge of the non-blinking area has a rectangular shape. An electronic device comprising the display system according to claim 1. A display panel having a plurality of pixels arranged thereon and a camera,
A calibration method for a display system in which light transmitted through a transmission area that is a part of the display panel is incident on the camera, comprising the steps of:
The display system comprises:
A step of causing a first pixel arranged in a blinking region that is a part of the transparent region to emit light and causing a second pixel arranged in a non-blinking region that is the other part of the transparent region not to emit light, and causing the camera to capture a first image;
causing the camera to capture a second image without illuminating the first pixel and the second pixel;
detecting a difference region, which is a region in which a significant difference occurs between the first image and the second image, from a captured image region, which is a region of the entire captured image;
and determining an area within the transparent area that corresponds to the captured image area as a non-display area in which pixels are not lit based on the corresponding relationship between the differential area and the blinking area , and displaying an image captured by the camera on the display panel .

Images