JP4573001B2 - Image compression method, image processing apparatus, image display apparatus, and image display system - Google Patents

Description

  The present invention relates to a compression method for compressing an input external image, and an image processing device, an image display device, and an image display system for compressing an image by the compression method. In particular, by displaying an external image on a transparent display unit, an image display device that allows an observer to visually recognize the external image and the real image through the display unit at the same time, for example, a see-through type head mounted display (hereinafter referred to as HMD). ).

  Examples of HMDs that allow an observer to visually recognize a real image and an external image at the same time include HMDs disclosed in Patent Documents 1 and 2. The HMD disclosed in Patent Document 1 includes a transparent display unit, and displays an external image that is an image input from the outside on the display unit, thereby simultaneously observing an external image and a real image through the display unit. (See-through type HMD). On the other hand, the HMD described in Patent Literature 2 captures an optical image through a display unit to obtain a real image, and generates a display image by synthesizing an external image with the real image and displays the display image on the display unit ( Video see-through HMD). In this video see-through type HMD, an external image corresponding to a real image is created based on a part of information of the real image. The HMD described in Patent Literature 3 includes an imaging unit that generates a real image by imaging, and communicates the real image and the external image with an external image processing apparatus.

JP 2000-184370 A Japanese Patent Laid-Open No. 2007-219082 JP-A-5-21650

  By the way, in communication between the HMD and an external image processing apparatus, when data having a large amount of information such as a moving image is communicated, a delay problem may occur. For this reason, when communicating data that requires real-time properties, such as the above moving image, the amount of data to be communicated is reduced to reduce the amount of transmission, thereby suppressing delay. However, if the information amount of the image data to be communicated is reduced, there is a problem that the image quality of the image is deteriorated. For example, if the transmission amount is reduced by reducing the number of pixels of the image sent from the image processing device to the HMD (reducing the resolution), interpolation processing or the like is performed before displaying on the display unit to compensate for the number of pixels. However, the image quality of the displayed image deteriorates.

  Accordingly, the present invention provides a compression method capable of suppressing deterioration in image quality and reducing the transmission amount, an image processing device, an image display device, and an image display system that perform image compression using the compression method. With the goal.

  The video display device of the present invention compares an external image input from the outside of the device and a local background image showing a real image visually recognized by an observer for each corresponding position, and the local background image of the external image An external image update processing unit that sets an area similar to the target area and performs update processing that is processing for reducing the sharpness on the target area of the external image, and the external image after the update processing And an external image encoding unit for encoding.

  In the image processing apparatus having the above configuration, the update process performed by the external image update processing unit may be a smoothing process.

  If comprised in this way, the signal value which shows the high frequency component reduced by smoothing will be omitted efficiently. Therefore, it is possible to reduce the information amount of the external image. For example, when an external image is converted into a signal sequence composed of frequency domain signals and a smoothing process is applied when the signal sequence is omitted, the amount of information can be suitably reduced. Note that this signal sequence is obtained by arranging signal values according to the magnitude of the frequency component, and adjacent equal signal values may be abbreviated.

  In the image processing apparatus having the above-described configuration, the update process performed by the external image update processing unit may be a resolution reduction process.

  With this configuration, the number of pixels in the target area is reduced, and thus the amount of information can be reduced. Note that it is preferable to reduce the resolution of only a predetermined component of a pixel value, particularly a component indicating a color, because deterioration in image quality of an external image can be suppressed.

In the image processing apparatus having the above configuration, the pixel value of the pixel of the external image includes a luminance component and a color difference component , and the external image update processing unit reduces the sharpness with respect to the luminance component . The update process with a small effect is performed or the update process is not performed, and the update process with a large effect of reducing the sharpness with respect to the color difference component may be performed.

  If comprised in this way, it will become possible to suppress efficiently the degradation of the image quality which an observer feels when displaying an external image.

  Further, in the image processing apparatus having the above configuration, the external image update processing unit compares the local background image and the external image for each block that is an area having a predetermined size, and among the blocks of the external image. A thing similar to the block of the local background image may be set as the target area. In particular, the external image update processing unit compares the representative values of the blocks of the external image and the local background image, and the representative value is the median of the pixel values of the pixels included in the block. The average value of the pixel values of the pixels included in the block and the value of the direct current component obtained by converting the pixel value into a frequency domain signal may be used.

  If comprised in this way, it will become possible to set an object area | region simply in a block unit.

  In the image processing apparatus having the above configuration, when the variance of pixel values of pixels belonging to the block of the external image is larger than a predetermined value, the block may not be set as the target region.

  With this configuration, it is possible to exclude the block from the target area when it is inappropriate as the target area, for example, when an edge portion is included in the image in the block. As a result, it is possible to suppress blurring of the image due to blurring of an edge portion of the image. In addition, it is preferable to exclude blocks having a large variance in advance because the number of blocks to be compared can be reduced.

  In the image processing apparatus having the above configuration, the external image update processing unit divides each of the external image and the local background image into regions where pixel values are substantially equal, and pixel values of pixels belonging to the region. When the external image and the local background image are overlapped with each other, a region that overlaps a region with a substantially equal label may be set as a target region.

  If comprised in this way, it will become possible to set an object area | region accurately. In particular, it is possible to set a target region having a pixel unit boundary.

  In the image processing apparatus having the above-described configuration, after the pixel values of the pixels of the external image and the local background image are reduced in gradation, the target region may be set by being divided for each region. Absent.

  If comprised in this way, it will become possible to reduce the kind of label attached and to enlarge each area | region and to enlarge the object area | region set. Therefore, it is possible to sufficiently obtain the effect of the update process.

  In the image processing apparatus having the above configuration, the external image update processing unit includes each of the target area set in the external image and a corresponding area corresponding to the target area of the local background image. Linear separation may be performed on the pixel values of the pixels, and the target area may be excluded when separation is possible.

  If comprised in this way, when the set object area | region is unsimilar and unsuitable, it becomes possible to exclude the object area | region. Therefore, it is possible to set the target area with higher accuracy.

  The image display system of the present invention includes the image processing device described above and an image display device that communicates with the image processing device, and the image display device is subjected to transmission processing from the image processing device. An external image reception processing unit that receives and processes the external image after processing, a display unit that displays the external image after the update processing and allows a real image to pass therethrough, and shows the real image by imaging An imaging unit that generates a real image; and a real image transmission processing unit that transmits the real image to the image processing device, and the image processing device transmits the real image to be transmitted from the image display device. A local image that is an image showing a superimposed portion of the real image that is visually recognized through the external image after the update process displayed on the display unit and a real image reception processing unit that performs reception processing; And local background image generating unit that generates from the real image, and further comprising an external image transmission processing unit, a for transmitting processing the external image after the updating process on the image display device.

  If comprised in this way, the real image imaged with the image display apparatus will be input into an image processing apparatus. Therefore, the image processing apparatus can record a real image, transmit it to another apparatus, and display it.

  An image display system of the present invention includes the image processing device according to any one of claims 1 to 11 and an image display device that communicates with the image processing device, and the image display device includes: An external image reception processing unit that receives and processes the external image after the update process that is transmitted from the image processing apparatus, and displays the external image after the update process and allows a real image to pass through to be visible. A display unit; an imaging unit that generates a real image representing the real image by imaging; and the local unit that represents a superimposed portion of the real image that is visually recognized through the external image after the update process displayed on the display unit. A local background image generation unit that generates a background image from the real image, and a local background image transmission processing unit that transmits the local background image to the image processing device, the image processing device comprising: A local background image reception processing unit for receiving the local background image transmitted from the image display device; and an external image transmission processing unit for transmitting the external image after the update processing to the image display device. It is characterized by that.

  With this configuration, an image that is transmitted from the image display device to the image processing device is a local background image. Therefore, it is possible to reduce the transmission amount as compared with the case where a real image is transmitted.

  The image display device of the present invention is an image display device that displays an image, wherein the image display device communicates an image with another image processing device having the same configuration as the image display device, An external image generating unit for other devices that generates an external image to be displayed on another image display device, and a local background for other devices that receives and processes a local background image for other devices that is transmitted from the other image display device The image reception processing unit is compared with the external image for other device and the local background image for other device for each corresponding position, and the region similar to the local background image for other device of the external image for other device is the target region And an external image update processing unit for the other device that performs an update process that is a process for reducing the sharpness on the target area of the external image for the other device, and the external for the other device after the update process. Display the image as the other image An external image transmission unit for other device that performs transmission processing on a device, an external image reception processing unit for own device that receives and processes an external image for own device that is transmitted from the other image display device, and the external image for the own device A display unit that transmits a real image and is visible, an imaging unit that generates a real image showing the real image by imaging, and a visual recognition through the external image for the own device displayed on the display unit A local background image for the own device that generates a local background image for the own device indicating a superimposed portion of the real image to be generated from the real image, and the local background image for the own device is transmitted to the other image display device A local background image transmission processing unit for the own device to be processed.

  If comprised in this way, it will become possible for a some image display apparatus to communicate an image directly. In particular, an image processing apparatus that is an intermediary apparatus can be eliminated.

  In the image display device having the above-described configuration, the external device external image generation unit may generate the external device external image from the real image generated by the imaging unit.

  The image compression method according to the present invention targets a first step of comparing an external image and a local background image for each corresponding position, and an area determined to be similar to the local background image of the external image by the first step. A second step of setting as a region, a third step of performing an update process that is a process of reducing the sharpness on the target region set by the second step of the external image, and a step after the update process And a fourth step of encoding the external image.

  According to the present invention, it is possible to reduce the amount of information of an external image by reducing the sharpness of a region similar to a local background image in the external image. Such a similar region is a region that is difficult for an observer to recognize when an external image is displayed. Therefore, even if the amount of information is reduced, it is possible to make it difficult for the observer to perceive deterioration in image quality.

These are the perspective views which showed an example of the structure of the image display apparatus (see-through type HMD) which concerns on embodiment of this invention. These are sectional drawings which show an example of the display operation of a display part. These are block diagrams shown about the structure of the 1st Example of the image display system which concerns on embodiment of this invention. These are the flowcharts which show about operation | movement of the image display system of 1st Example. FIG. 4 is a schematic diagram showing a state where DCT coefficients after quantization in one macroblock are arranged in a matrix. These are sectional drawings shown about each field of view (field angle) of an image pick-up part and an observer. These are schematic diagrams of a local background image and an external image showing an outline of the update process. These are the flowcharts which show the outline | summary of an update process. These are block diagrams shown about the structure of 2nd Example of the image display system which concerns on embodiment of this invention. These are flowcharts which show operation | movement of the image display system of 2nd Example. These are block diagrams shown about the structure of 3rd Example of the image display system which concerns on embodiment of this invention. These are the flowcharts which show about operation | movement of the image display system of 3rd Example. These are the flowcharts which show about operation | movement of the image display system of 3rd Example. These are schematic diagrams of a local background image and an external image shown for the first example of image comparison and target area setting. These are schematic diagrams of a local background image, an external image, and a superimposed image shown in a second example of image comparison and target area setting. These are graphs of pixel values shown for the third example of image comparison and target area setting, and show a case where separation is possible. These are graphs of pixel values shown for the third example of image comparison and target area setting, and show a case where separation is impossible.

  Hereinafter, embodiments of a compression method, an image processing apparatus, and an image display apparatus according to the present invention will be described with reference to the drawings. First, a configuration example of a see-through HMD which is an example of an image display device will be described with reference to FIGS. 1 and 2.

<< Image display device >>
FIG. 1 is a perspective view showing an example of the configuration of an image display device (see-through type HMD) according to an embodiment of the present invention. As shown in FIG. 1, the image display apparatus 1 of this example is based on glasses. The image display device 1 generates a real image by imaging the frame 2, a display unit 3 that displays an external image, a right-eye lens 31 that is a part of the display unit 3, and a left-eye lens 4. An imaging unit 5, a main body unit 6 that communicates with an external image processing apparatus (not shown), and a cable 7 that connects the display unit 3 and the imaging unit 5 to the main body unit 6 are provided. The main unit 6 supplies an external image and power to the display unit 3 through the cable 7 and acquires a real image generated by the imaging unit 5. Further, the main body 6 transmits a real image to the image processing apparatus and receives an external image from the image processing apparatus.

  The display unit 3 includes the right-eye lens 31 and a projection unit 32 that projects an optical image based on an external image input via the cable 7 onto the right-eye lens 31. The projection unit 32 is provided on the upper part of the right-eye lens 31 and includes a light source (not shown) and a display element (not shown) for projecting an optical image of an external image. In addition, you may comprise a light source by LED (Light Emitting Diode) and LD (Laser Diode). Furthermore, the display element may be configured by a liquid crystal panel that selectively transmits light emitted from the light source and projects an optical image.

  An example of the display operation of the display unit 3 is shown in FIG. FIG. 2 is a cross-sectional view illustrating an example of the display operation of the display unit. As shown in FIG. 2, the optical image of the external image projected from the projection unit 32 is incident on the right eye lens 31 from the upper end thereof. The incident optical image proceeds to the lower part while being totally reflected in the right-eye lens 31, and is reflected by the holographic optical element HOE provided at the lower part of the right-eye lens 31. Then, the reflected optical image enters the eyes of the observer, so that the observer visually recognizes the external image. At the same time, the real image that has passed through the right-eye lens 13 is also incident on the eyes of the observer and is visually recognized by the observer. Therefore, the observer can visually recognize the real image and the external image at the same time. In other words, the observer can visually recognize a real image on which an external image is superimposed.

  As shown in FIG. 1, the imaging unit 5 is provided integrally with the projection unit 32 of the display unit 3, and is in a direction visually recognized by the observer (the direction opposite to the direction in which the observer of the right-eye lens 31 exists). Direction) is taken. As described above, the real image captured and generated by the imaging unit 5 is input to the main body unit 6 via the cable 7. Note that the imaging unit 5 may include a solid-state imaging device (for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS)).

  The lens (lens 4 for the left eye) that does not constitute the display unit 3 may be configured not to be provided in the image display device 1 or may be provided as a dummy lens as in this example. In addition, although the configuration in which the projection unit 32 is provided only in the one-eye lens (the right-eye lens 31) is shown, the configuration may be provided in the binocular lens. Similarly, the imaging unit 5 may be configured to be provided in a binocular lens.

  In addition, the configuration including the main body 6 and the cable 7 has been described. However, the main body 6 and the cable 7 may be omitted by providing the function of the main body 6 inside the projection unit 32.

  Further, as an external image acquired by the main body unit 6 from the image processing apparatus, for example, a display image of a computer or a mobile phone, a moving image captured by a web camera, or the like can be applied. In addition, a real image captured by another image display device (HMD) may be used as an external image.

<< Image display system >>
<First embodiment>
Next, a first embodiment of an image display system including the above-described image display device and image processing device will be described with reference to the drawings. FIG. 3 is a block diagram showing the configuration of the first example of the image display system according to the embodiment of the present invention. Parts that are the same as those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted.

  As shown in FIG. 3, the image display system S of the present example includes the image display device 1 and the image processing device 20 described above. As described above, the image display device 1 transmits a real image to the image processing device. Further, the image processing device 20 transmits an external image to the image display device 1. The image communication between the image display device 1 and the image processing device 20 may be wireless communication using a wireless local area network (LAN) or infrared rays, or wired network communication using a wired LAN.

  The image display device 1 includes an imaging unit 5 that captures a real image to generate and output a real image, and a real image encoding unit that encodes the real image output from the imaging unit 5 to generate and output a real image signal. 11, a real image transmission unit 12 that transmits a real image signal output from the real image encoding unit 11 to the image processing device 20, and an external image reception unit 13 that receives an external image signal transmitted from the image processing device 20. An external image decoding unit 14 that decodes an external image signal received by the external image receiving unit 13 to generate and output an external image, and a display unit that displays the external image output from the external image decoding unit 14 3.

  Further, the image processing device 20 generates a real image by decoding the real image signal received by the real image receiving unit 21 and the real image receiving unit 21 that receives the real image signal transmitted from the image display device 1. A real image decoding unit 22 that outputs, a local background image generation unit 23 that generates and outputs a local background image based on the real image output from the real image decoding unit 22, and a local background image generation unit 23 that outputs the local background image. An external image update processing unit 24 that performs an update process on an external image input from the outside based on the local background image and outputs an external image after the update process, and an external image output from the external image update process unit 24 An external image encoding unit 25 that generates and outputs an external image signal by encoding, an external image transmission unit 26 that transmits the external image signal output from the external image encoding unit 25 to the image display device 1, and Provided.

  Next, the operation of the image display system S of the present embodiment will be described with reference to the drawings. FIG. 4 is a flowchart showing the operation of the image display system of the first embodiment. FIG. 4 shows a case where an external image corresponding to one real image is displayed. When a real image is continuously captured and 00 external images are displayed, a series of operations shown in FIG. Shall be repeated.

  As shown in FIG. 4, first, the imaging unit 5 captures a real image to generate a real image (STEP 1). Next, the real image encoding unit 11 compresses and encodes the real image to generate a real image signal (STEP 2). At this time, for example, a compression coding method using discrete cosine transform (hereinafter referred to as DCT) (for example, JPEG (Joint Photographic Experts Group), Motion-JPEG, MPEG (Moving Picture Experts Group), etc.) is applied. It doesn't matter.

  In the case of using DCT, first, a spatial domain signal expressed by a pixel value for each position (pixel) is converted into a frequency domain signal expressed by a signal value (DCT coefficient) for each frequency component. At this time, the image may be divided into a plurality of blocks (hereinafter referred to as macroblocks), and DCT may be performed for each macroblock. The obtained DCT coefficient is compressed by being quantized.

  As the quantization process, for example, it is possible to apply a process of dividing each of the DCT coefficients by a quantization constant that is a predetermined value and rounding off fractions after the decimal point. Note that the quantization constant for dividing the DCT coefficient of the low frequency component may be set small, and the quantization constant for dividing the DCT coefficient of the high frequency component may be set large.

  The DCT coefficient obtained by the above processing becomes a signal sequence by performing, for example, zigzag scanning. The zigzag scan is shown in FIG. FIG. 5 is a schematic diagram showing a state where DCT coefficients after quantization in one macroblock are arranged in a matrix. In particular, 8 × 8 DCT coefficients obtained by performing DCT on an 8 × 8 pixel macroblock are shown. In FIG. 5, the DCT coefficient on the right indicates a component with a higher vertical frequency, and the lower DCT coefficient indicates a component with a higher horizontal frequency. That is, the frequency component indicated by the DCT coefficient at the upper left is the smallest (the component having a frequency of 0, that is, a direct current component), and the frequency component indicated by the DCT coefficient at the lower right is the largest.

  In the zigzag scan, a signal sequence is generated by scanning while moving back and forth in an oblique direction from the upper left DCT coefficient Ddc to the lower right DCT coefficient D63. That is, a signal sequence in which DCT coefficients are arranged in the order of Ddc, D1, D2, D3, D4, D5,..., D61, D62, and D63 is generated. The obtained signal sequence is encoded by, for example, run length encoding. This is a coding in which when the value X of the DCT coefficient appears m times continuously in the signal sequence, the portion is collectively abbreviated as mX.

The above encoding is performed for each component of the pixel value of the real image. For example, when a real image is composed of component signals, encoding is performed for each of Y (luminance component), C _b (blue color difference component), and C _r (red color difference component). . In the case where the real image is composed of components such as R (red), G (green), B (blue), H (hue), S (saturation), and V (brightness), each component is similarly processed. Is encoded.

  The real image signal generated in STEP 2 is transmitted from the real image transmission unit 12 to the image processing device 20 (STEP 3). The transmitted real image signal is received by the real image receiving unit 21 of the image processing apparatus 20 (STEP 4).

The real image signal received in STEP 4 is decoded by the real image decoding unit 22 to generate a real image (STEP 5). At this time, processing reverse to the above-described encoding is performed. For example, decoding of an encoded signal sequence, writing of a signal sequence into a matrix, inverse quantization (for example, processing for multiplying each DTC coefficient by the above quantization constant), and inverse discrete cosine transform, A real image is generated in order. Similarly to the above-mentioned encoding, decoding is performed, for example, every component of the YC _b C _r.

  The real image obtained in STEP 5 is converted into a local background image in the local background image generation unit 23 (STEP 6). Conversion to a local background image will be described with reference to FIG. FIG. 6 is a cross-sectional view showing the fields of view (view angles) of the imaging unit and the observer, and shows the same cross section as FIG.

  As shown in FIG. 6, the imaging unit 5 and the position of the observer's eyes are slightly different. Therefore, the actual image obtained by the imaging unit 5 is different from the actual image that the observer is actually viewing. Therefore, in STEP 6, the real image is converted to generate a local background image that is an image showing the real image visually recognized by the observer. In particular, it is assumed that an image showing a real image visually recognized through an external image displayed on the display unit 3 by the observer becomes a local background image.

  As the conversion process, for example, a process of deleting an unnecessary part from the real image or a process of converting the coordinates of the pixels in the real image is performed. Note that the technique disclosed in Japanese Patent Application Laid-Open No. 2003-91720 may be used for the conversion to the local background image. This technique generates an image viewed from the observer's viewpoint by virtually converting the viewpoint of the imaging unit 5 (specifically, converting the coordinates of the real image).

  Next, the external image update processing unit 24 updates the acquired external image based on the local background image generated in STEP 6 (STEP 7). An overview of the update process will be described with reference to FIGS. FIG. 7 is a schematic diagram of a local background image and an external image showing an outline of the update process, and FIG. 8 is a flowchart showing an outline of the update process.

  As shown in FIG. 8, when the update process is started, comparison of both images is performed for the first corresponding region (region that overlaps the real image (local background image) when the external image is displayed on the display unit 3). Is performed (STEP 71). As a result of the comparison in STEP 71, the region in the external image determined to be similar to both images is set as a region to be updated (hereinafter referred to as a target region) (STEP 72). For example, in the example shown in FIG. 7, the area A2 in the external image is the target area, and the area A1 in the local background image corresponding to this area is the corresponding area.

  Update processing is executed for the target area set in the external image in STEP 72 (STEP 73). The update process is a process for reducing the sharpness of an image, for example, a process for smoothing pixel values of pixels in a target area or a process for reducing resolution. In this embodiment, for the sake of concrete explanation, the case where the update process is a smoothing process will be mainly described. Details of the operations of STEP 71 to STEP 73 will be described later.

  The updated external image generated in STEP 7 is encoded by the external image encoding unit 25 to generate an external image signal (STEP 8). When the smoothing process is applied as the update process, it is preferable to perform encoding as described in STEP 2 (that is, encoding using DCT).

  Specifically, the target region that has been updated in STEP 7 has a high-frequency component reduced by the smoothing process. Therefore, when DCT is performed on the macroblock including the target region and the obtained DCT coefficients are quantized, the DCT coefficients of the high-frequency components (that is, the DCT coefficients near the lower right in FIG. 5) are likely to be zero. . Then, in the second half of the signal sequence generated by the zigzag scan, 0 is continuous. At this time, if run-length encoding is performed, it is possible to abbreviate consecutive zeros in the latter half. That is, the amount of information of the external image signal can be reduced.

  The external image signal generated in STEP 8 is transmitted from the external image transmission unit 26 to the image display device 1 (STEP 9). The transmitted external image signal is received by the external image receiving unit 13 of the image display device 1 (STEP 10).

  The external image signal received in STEP 10 is decoded by the external image decoding unit 14 to generate an external image (STEP 11). At this time, similar to the above-described real image decoding unit 22, processing reverse to the encoding is performed. Then, the obtained external image is displayed to the observer by the display unit 3.

  The external image is displayed superimposed on a real image visually recognized by the observer. Of the displayed external image, the target region similar to the real image has a reduced high-frequency component (definition). However, since such a target area is an area similar to the superimposed real image, it is difficult for the observer to accurately recognize details (for example, fine shades of color). Therefore, even if the high-frequency component in the target region is reduced, the image quality is hardly deteriorated by the observer. Therefore, it is possible to reduce the transmission amount without causing the observer to feel the deterioration of the image quality of the external image.

  Although the case where the same encoding method is applied to the real image encoding unit 11 and the external image encoding unit 25 has been described, different encoding methods may be applied to each. However, it is preferable to apply the above-described encoding method to the external image encoding unit 25. The above-described encoding method is an encoding method in which abbreviated notation is performed on a signal sequence composed of frequency domain signals. Specifically, it is an encoding method in which consecutive signals having the same value are abbreviated for a signal sequence in which signal values are arranged in order of the magnitude of frequency components.

  Moreover, although run length encoding was illustrated as an encoding method, the encoding method of this invention is not this limitation. For example, a so-called zero-run length coding that omits only the signal value 0 may be used, and when 0 continues to the end of the signal sequence (that is, when the signal values of components above a certain frequency are all 0) ) May be used in which an EOB (End Of Block) code is output and abbreviated.

  Further, as shown in FIG. 3, the real image output from the real image decoding unit 22 of the image processing device 20 is used by another person (for example, a person who works together with an observer who uses the image display device 1). The image processing apparatus 20 may be configured to output a real image in order to be confirmed or recorded.

<Second embodiment>
Next, a second embodiment of the image display system will be described with reference to the drawings. FIG. 9 is a block diagram showing the configuration of the second example of the image display system according to the embodiment of the present invention, and corresponds to FIG. 3 showing the first example. In addition, the same number is attached | subjected about the part similar to 1st Example, and the detailed description is abbreviate | omitted.

  Similar to the first embodiment, the image display system Sa of the present embodiment includes an image display device 1a and an image processing device 20a. However, the point that the image display device 1a is provided with the local background image generation unit 23a is different from the first embodiment.

  The image display device 1a includes an imaging unit 5, a local background image generation unit 23a that generates and outputs a local background image based on a real image output from the imaging unit 5, and a local output that is output from the local background image generation unit 23a. A local background image encoding unit 11a that encodes a background image to generate and output a local background image signal, and a local background image that transmits a local background image signal output from the local background image encoding unit 11a to the image processing device 20a A transmission unit 12a, an external image reception unit 13, an external image decoding unit 14, and a display unit 3 are provided.

  In addition, the image processing device 20a decodes the local background image signal received by the local background image receiving unit 21a that receives the local background image signal transmitted from the image display device 1a and the local background image receiving unit 21a, and performs local processing. A local background image decoding unit 22a that generates and outputs a background image, an external image update processing unit 24, an external image encoding unit 25, and an external image transmission unit 26 are provided.

  Next, the operation of the image display system Sa of the present embodiment will be described with reference to the drawings. FIG. 10 is a flowchart showing the operation of the image display system of the second embodiment, and corresponds to FIG. 4 showing the operation of the image display system of the first embodiment. It should be noted that the same STEP numbers are assigned to parts that operate in the same manner as in FIG. 4, and detailed descriptions thereof are omitted.

  As shown in FIG. 10, first, the imaging unit 5 captures a real image and generates a real image (STEP 1). In this embodiment, the local background image generation unit 23a then generates a local background image from the real image output from the imaging unit 5 (STEP 6a). Note that the local background image generation method is the same as the generation method by the local background image generation unit 23 of the first embodiment.

  The generated local background image is encoded by the local background image encoding unit 11a to generate a local background image signal (STEP 2a). The generated local background image signal is transmitted from the local background image transmission unit 12a to the image processing device 20a (STEP 3a). Note that the encoding method by the local background image encoding unit 11a is the same as the encoding method by the real image encoding unit 11 of the first embodiment.

  The local background image signal transmitted in STEP 3a is received by the local background image receiving unit 21a of the image processing device 20a (STEP 4a). The received local background image signal is decoded by the local background image decoding unit 22a to generate a local background image (STEP 5a). Note that the decoding method by the local background image decoding unit 22a is the same as the decoding method by the real image decoding unit 22 of the first embodiment.

  Further, the following operations are the external image update processing by the external image update processing unit 24 (STEP 7), the external image encoding after the update processing by the external image encoding unit 25 (STEP 8), and the external image transmission unit 26. External image signal transmission by the external image signal (STEP 9), external image signal reception by the external image reception unit (STEP 10), external image decoding by the external image decoding unit (STEP 11), and display of the external image by the display unit 3 (STEP 12) And are the same as those in the first embodiment.

  Even if it is configured as in the present embodiment, the same effect as in the first embodiment can be obtained. That is, it is possible to reduce the transmission amount without causing the observer to strongly feel the deterioration of the image quality of the external image.

  Further, in the present embodiment, a local background image is generated in the image display device 1a and is transmitted to the image processing device 20a instead of a real image. That is, a local background image having a smaller amount of information than a real image is communicated. Therefore, the transmission amount between the image display device 1a and the image processing device 20a can be further reduced as compared with the first embodiment.

  In FIG. 9, the local background image output from the local background image decoding unit 22a of the image processing device 20a is illustrated as a configuration that is not output from the image processing device 20a, but may be configured to be output. In particular, when a local background image is sufficient for an image to be confirmed or recorded by another person, the second embodiment can be applied. However, when it is necessary to output a real image from the image processing apparatus, it is preferable to apply the first embodiment.

<Third embodiment>
Next, a third embodiment of the image display system will be described with reference to the drawings. FIG. 11 is a block diagram showing the configuration of the third example of the image display system according to the embodiment of the present invention, and corresponds to FIG. 3 showing the first example. In addition, the same number is attached | subjected about the part similar to 1st Example, and the detailed description is abbreviate | omitted.

  The image display system Sb of the present embodiment includes a plurality of image display devices that have both the functions of the image display device and the functions of the image processing device shown in the first and second embodiments. In the image display system Sb of the present embodiment, a plurality of image display devices having the same configuration communicate directly.

  Hereinafter, an image display apparatus that will be mainly described is referred to as a first image display apparatus 100, and an image display apparatus that communicates with the first image display apparatus 100 is referred to as a second image display apparatus 101. In the following description, “first” attached to the name indicates a display operation of the first image display apparatus 100, and “second” attached to the second image display apparatus 101. The display operation will be shown.

  Hereinafter, only the configuration and operation of the first image display device 100 will be described, and the configuration and operation of the second image display device 101 will be omitted as being the same. Note that the configuration and operation of the second image display apparatus 101 are obtained by replacing “first” and “second” in the following description with “second” and “first”, respectively.

  The first image display apparatus 100 generates and outputs a first local background image based on the imaging unit 5 that generates and outputs a first reality image by imaging, and the first reality image output from the imaging unit 5. A local background image generation unit 23b; a first local background image encoding unit 11b that encodes the first local background image output from the first local background image generation unit 23b to generate and output a first local background image signal; The first local background image transmission unit 12b that transmits the first local background image signal output from the first local background image encoding unit 11b to the second image display device 101, and the second local image that generates and outputs the second external image. A second external image generation unit 31, a second local background image reception unit 21b that receives a second local background image signal transmitted from the second image display device 101, and a second local background image reception unit 21b that receives the second local background image signal. 2 Local background image signal A second local background image decoding unit 22b that generates and outputs a second local background image by decoding, and a second external image generation based on the second local background image output from the second local background image decoding unit 22b The second external image update processing unit 24b that performs the update process on the second external image output from the unit 31 and outputs the second external image after the update process, and the second external image update processing unit 24b that outputs the second external image. A second external image encoding unit 25b that encodes two external images to generate and output a second external image signal; and a second external image signal output from the second external image encoding unit 25b The second external image transmission unit 26b that transmits to the 101, the first external image reception unit 13b that receives the first external image signal transmitted from the second image display device 101, and the first external image reception unit 13b. First external image signal The includes a first external image decoding unit 14b for generating a first external image by the decoded output, a display unit 3 for displaying a first external image output from the first external image decoding unit 14b, a.

  Next, the operation of the image display system Sb of the present embodiment will be described with reference to the drawings. FIGS. 12 and 13 are flowcharts showing the operation of the image display system of the third embodiment, and correspond to FIG. 4 showing the operation of the image display system of the first embodiment. It should be noted that the same STEP numbers are assigned to parts that operate in the same manner as in FIG. 4, and detailed descriptions thereof are omitted. FIG. 12 is a flowchart for the display operation of the first image display device 100 performed by the first image display device 100. FIG. 13 is a flowchart for the display operation of the second image display device 101 performed by the first image display device 100.

  As shown in FIG. 12, first, the imaging unit 5 captures a real image and generates a first real image (STEP 1b). Next, the local background image generation unit 23b generates a first local background image from the first real image output from the imaging unit 5 (STEP 6b). The first local background image generation method is the same as the generation method by the local background image generation unit 23 of the first embodiment.

  The generated first local background image is encoded by the first local background image encoding unit 11b to generate a first local background image signal (STEP 2b). The generated first local background image signal is transmitted from the first local background image transmission unit 12b to the second image display device 101 (STEP 3b). Note that the encoding method by the local background image encoding unit 11b is the same as the encoding method by the real image encoding unit 11 of the first embodiment.

  In this embodiment, the second image display device 101 generates a first external image to be displayed on the first image display device (STEP α). Specifically, the second image display device 101 generates a first external image, updates the first external image using the first local background image transmitted from the first image display device 100, and executes this. The operation of transmitting to the first image display device 100 is performed. Note that the operation of STEP α of the second image display apparatus 101 is the same as the operation of the first image display apparatus 100 described later with reference to FIG.

  The signal transmitted from the second image display device 101 is received by the first external image receiving unit 13b of the first image display device 100 (STEP 10b). The generated first external image signal is decoded by the first external image decoding unit 14b to generate a first external image (STEP 11b). Then, the first external image is displayed on the display unit 3 (STEP 12). The decoding method by the first external image decoding unit 14b is the same as the decoding method by the external image decoding unit 14 of the first embodiment.

  The first image display device 101 also performs the operation shown in FIG. 13 (that is, the operation for the display operation of the second image display device 101) simultaneously with the operation of FIG. As shown in FIG. 13, first, the second local background image signal transmitted from the second image display device 101 is received by the second local background image receiving unit 21b (STEP 4b). The received second local background image signal is decoded by the second local background image decoding unit 22b to generate a second local background image (STEP 5b). Note that the decoding method by the second local background image decoding unit 22b is the same as the decoding method by the real image decoding unit 22 of the first embodiment.

  In the first image display device 100, the second external image generation unit 31 generates a second external image to be displayed on the second image display device 101 (STEP β). For example, the second external image may be generated using part or all of the first real image captured by the imaging unit 5. In addition, the second external image may be generated using an image captured by an image capturing unit other than the image capturing unit 5, or an image recorded in a memory (not shown) or the first image display device 100 may be externally provided. The second external image may be generated by using the image input from. The timing for generating and outputting the second external image is not limited to the example in FIG. 13, and it may be generated and output at any timing as long as it is before STEP 7 b described later.

  And the external image update process part 24b performs the update process of the 2nd external image produced | generated by STEP (beta) based on the 2nd local background image produced | generated by STEP5b (STEP7b). The update process by the second external image update processing unit 24b is the same as the update process of the external image update processing unit 24 of the first embodiment.

  The second external image after update processing generated by the second external image update processing unit 24b is encoded by the second external image encoding unit 25b, and a second external image signal is generated (STEP 8b). The second external image signal generated in STEP 8b is transmitted from the second external image transmission unit 26b to the second image display device 101 (STEP 9b).

  The operation of STEPα by the second image display device 101 shown in FIG. 12 is the same as the operation of the first image display device 100 shown in FIG. 13 (STEPs 4b, 5b, β, 7b-9b). Note that the operation of the second image display device 101 (that is, STEP α) is indicated by replacing “first” and “second” as described above.

  Even if it is configured as in the present embodiment, the same effect as in the first embodiment can be obtained. That is, it is possible to reduce the transmission amount without causing the observer to strongly feel the deterioration of the image quality of the external image. In particular, when the amount of information of the first external image and the second external image is large and the amount of transmission increases when these images are directly transmitted and received, it is preferable to apply the configuration of this embodiment.

  In this embodiment, it is possible to eliminate the need for an image processing apparatus that is an intermediary apparatus. For example, when mutual display of a plurality of image display devices is performed (that is, when an external image displayed on one side is an image based on a real image captured by the other image display device), the first embodiment or the second example If the configuration of the embodiment is applied, an image processing apparatus that performs update processing and transmission / reception of external images is indispensable. However, with the configuration of this embodiment, the image display apparatuses 100 and 101 can perform update processing and transmission / reception of external images, respectively, so that mutual display can be performed without requiring an image processing apparatus. It becomes possible.

  In the example illustrated in FIG. 11, the first image display device 100 generates the first local background image and transmits the first local background image to the second image display device 101 (that is, the same configuration as in the second embodiment). The first actual image may be transmitted to the second image display device 101, and the second local image may be generated by the second image display device 101 (that is, the same configuration as in the first embodiment). However, there is no special circumstance that requires the first actual image on the second image display device 101 side, such as when the second image display device 101 records the first actual image or transmits it to others. As far as the transmission amount is reduced, the configuration shown in FIG. 11 is preferable.

<Modification>
In the first to third embodiments described above, the external image before the update process and the local background image are described as having the same size (number of pixels), but may be different. If the sizes are different, the external image update processing unit converts the size of one of the images (for example, increasing the number of pixels by interpolation processing or reducing the number of images by thinning processing). It doesn't matter. If comprised in this way, since the pixel number of both the images compared at the time of an update process will become the same, it will become possible to compare easily. In this case, conversion may be performed so that the external image does not become larger than the display resolution of the display unit.

  Further, the image display device and the image processing device in the first and second embodiments, and the first image display device and the second image display device in the third embodiment may communicate with each other by wire or communicate by radio. It doesn't matter.

<< Update process >>
Specific examples of update processing performed in the above-described image display system will be described below with reference to the drawings. In particular, the processing of STEP 71 and STEP 72 shown in FIG. 8 (image comparison and target area setting) and the processing of STEP 73 (execution of update processing) will be described.

<Image comparison and target area setting>
[First example]
A first example of image comparison and target area setting will be described with reference to the drawings. FIG. 14 is a schematic diagram of a local background image and an external image shown for a first example of image comparison and target area setting.

  In this example, the local background image and the external image are divided into the same number of blocks having a predetermined size. The example shown in FIG. 14 shows a case where the local background image and the external image are each divided into (N + 1) × (M + 1) blocks (N and M are natural numbers). Further, the position of each block is expressed by the coordinates of [x, y], and the value of x increases toward the right, and the value of y increases toward the bottom. Also, the upper left block of the local background image is A1 [0,0], and the lower right block is A1 [N, M]. The same applies to the external image, where the upper left block is A2 [0,0] and the lower right block is A2 [N, M].

  For each block of the local background image and the external image, a block at a corresponding position (that is, a block where x and y are both equal) is compared. For example, as shown in FIG. 14, A1 [s, t] and A2 [s, t] are compared (where 0 ≦ s ≦ N and 0 ≦ t ≦ M).

  As a comparison method, for example, it is possible to use a method of determining a representative value of a block using an average value or a median value of pixels included in a block to be compared, and comparing the representative value. It is.

The representative value is calculated for each pixel value component such as YC _r C _b described above and compared. Then, when the representative values of all the components are similar (for example, when the difference value of each component is equal to or smaller than a predetermined size, or when the representative values after gradation reduction are substantially equal), the compared blocks Are determined to be similar. Then, the block of the external image is set as the target area. The present invention is not limited to the pixel values having the components of YC _r C _b, it is possible to similarly compare process even pixel values with a component such as RGB or HSV.

  If comprised in this way, it will become possible to perform a similar determination only by comparing the representative value for every block, Therefore A comparison process can be performed rapidly. In particular, as each block is set larger, the determination accuracy is degraded, but the processing can be speeded up. Conversely, to improve the accuracy of determination, the block is set to be small. About the size of a block, it can adjust suitably from such a viewpoint.

  Note that when the encoding method using the frequency domain signal as described above is used for communication, a component having a frequency of 0 (DC component) may be used as a representative value. Since this component indicates the average pixel value in the block, it can be used as a representative value. Further, for example, the external image update processing unit 24 and the second external image update from the real image decoding unit 22 in FIG. 3, the local background image decoding unit 22a in FIG. 9, and the second local background image decoding unit 22b in FIG. The processing unit 24b may acquire each DC component. However, when a DC component is acquired from the real image decoding unit 22 in FIG. 3, it is necessary to acquire a DC component relating to a block after conversion into a local background image.

  If comprised in this way, it will become possible to calculate the representative value of each block using the direct-current component obtained for every macroblock, without calculating a representative value from a pixel value, and can simplify a process. Furthermore, if the macroblock size and the block size of this comparison processing are made the same, the direct current component of the macroblock can be directly used for the representative value of the block, and the processing can be further simplified. This is preferable because it is possible.

  Alternatively, the variance of the pixel values of each block of the external image may be calculated, and blocks having a variance greater than a predetermined value may be excluded from the target region. There is a high possibility that a block having a large variance includes an edge portion (for example, a contour portion of an object and a pixel value that can change sharply). If the sharpness is reduced by performing an update process to be described later on such a portion, the edge portion may be blurred and the external image may become unclear. Therefore, it is possible to improve the image quality of the external image by excluding blocks with such a large variance from the target area. Furthermore, by performing this exclusion process prior to the comparison process, it is possible to suppress the unnecessary comparison process and simplify the process.

[Second example]
A second example of image comparison and target area setting will be described with reference to the drawings. FIG. 15 is a schematic diagram of a local background image, an external image, and a superimposed image shown in a second example of image comparison and target area setting.

In this example, area division is performed based on the pixel value. In this case, first, components of the respective pixel values of the local background image and an external image (e.g., YC _r C _b, etc.) to the low gradation of each. Then, on the condition that the respective pixel values after the gradation reduction are substantially equal, a labeling process (a process for forming a label by connecting adjacent pixels having substantially the same pixel value and attaching a label) is performed. At this time, a label corresponding to the pixel value of the pixel belonging to the region is attached. When a region of a certain label surrounds a minimum region of another label (for example, a region corresponding to several pixels), processing for regarding the minimum region as a region of a certain label may be performed.

  Further, for example, a threshold value division method may be used in which a threshold value of a pixel value is detected based on a histogram of pixel values, and a region is divided by determining whether the pixel values are substantially equal based on the threshold value. . Alternatively, a region expansion method may be used in which a neighboring pixel having a similar pixel value starts from a certain pixel and attempts to form a region. In addition to these methods, various well-known region segmentation methods can be used. Even when these methods are used, it is preferable to reduce the gradation of the image in advance as described above.

  In FIG. 15, the areas divided by the above method and the labels attached to the respective areas are displayed together. Note that the label G1 in the local region image and the label g1 in the external image indicate labels attached to regions including pixels with substantially equal pixel values. Similarly, G5 and g5 also indicate labels attached to regions including pixels having substantially the same pixel value.

  As shown in the superimposed image of FIG. 15, when an external image is superimposed on a local area image, an area where labels are substantially equal (for example, an area where G1 and g1 overlap or an area where G5 and g5 overlap) Set as.

  When the target area is set in this way, an external image area similar to the local area image can be accurately set as the target area. In particular, it is possible to set a target region having a pixel unit boundary. Therefore, the quality of the external image displayed on the display unit 3 can be improved.

  Also, by reducing the gradation of the pixel values before dividing the area, it is possible to reduce the types of labels to be attached and enlarge each area to increase the set target area. Therefore, it is possible to sufficiently obtain the effect of reducing the information amount by the update process.

[Third example]
A third example of image comparison and target area setting will be described with reference to the drawings. FIG. 16A and FIG. 16B are pixel value graphs showing a third example of image comparison and target area setting. It should be noted that the method of this example is a method that is preferably applied as a determination process for determining whether or not each target region set by the methods of the first example and the second example described above is appropriate. is there. FIG. 16 shows a case where the pixel value of each pixel of the image includes three types of components YC _r C _b .

The determination process of this example performs determination based on the distribution of pixel values of pixels provided in the target region of the external image and the corresponding region of the local background image. Specifically, as shown in FIG. 16A and FIG. 16B, after plotting each pixel in the region of both images based on the pixel value, a predetermined separation method is tried and a set of pixel values of the local background image And whether or not a set of pixel values of the external image is separable. The graph shown in FIGS. 16A and 16B are three-dimensional graph axes are perpendicular each YC _r C _b, respectively show pixels in the local background image triangle, the pixel of the external image in cross .

  FIG. 16A shows a case where separation is possible as a result of performing linear separation on a set of pixel values of the local background image and a set of pixel values of the external image by a predetermined method (the broken lines in the figure are , Straight line drawn by linear separation). In such a case, the target area of the external image and the corresponding area of the local background image are not similar. That is, when the sharpness of the target area of the external image is reduced and displayed on the display unit 3, it is easy for the observer to confirm the deterioration of the image quality. Therefore, such an external image area is excluded from the target area.

  On the other hand, FIG. 16B shows a case where separation is impossible as a result of performing linear separation on a set of pixel values of the local background image and a set of pixel values of the external image by a predetermined method. In such a case, the target area of the external image and the corresponding area of the local background image are similar. That is, when the sharpness of the target area of the external image is reduced and displayed on the display unit 3, it is difficult for the observer to confirm the deterioration of the image quality. Therefore, such an external image area is not excluded from the symmetric area.

  By performing the determination process in this way, it is possible to improve the selection accuracy of the target area of the external image. Therefore, it is possible to further improve the image quality of the external image displayed on the display unit 3.

  As a predetermined method for performing the above linear separation, for example, various known methods such as a support vector machine, a neural network, a k-neighbor method, and a k-means method can be applied.

  In the first to third examples, the determination of similarity between the local background image and the external image is performed using all types of components included in the pixel value of the pixel, and the target region is set. The determination of similarity may be performed using a predetermined type of component. For example, the similarity determination may be performed using only the color component.

<Execution of update process>
Next, update processing executed on the target area of the external image selected by the above method will be described. As described above, in the present embodiment, for example, a smoothing process can be applied as the update process. When the smoothing process is applied as the update process, for example, a filter process using an n × n average value filter (for example, a filter whose filter coefficients are all 1) as shown in the following formula (1) may be performed. Absent. In the following formula (1), f (i, j) is a pixel value in the target area before the update process, and g (i, j) is a pixel value in the target area after the update process. To do. [N / 2] represents a maximum integer not exceeding n / 2. N is a natural number of 2 or more.

  By calculating in this way, it is possible to reduce high frequency components in the target region. Therefore, as described above, the information amount of the encoded signal can be reduced, and the transmission amount can be reduced.

  In addition to the smoothing filter, a median filter (for example, a filter that outputs a median value of pixel values of a calculation target pixel and its surrounding pixels as a pixel value of a calculation target pixel after filtering) or a Gaussian filter (for example, The smoothing process may be performed using a filter whose filter coefficient is substantially equal to the Gaussian distribution. Other known smoothing filters may be used.

Further, such update processing may be performed only on a specific component of the pixel value, or the processing content may be varied depending on the type of the pixel value. For example, when each pixel is provided with each component of YC _r C _b, the smoothing process is not performed on the luminance component Y that the human (observer) feels sensitively, or the degree of smoothing may be reduced. I do not care. On the other hand, the color difference component C _r C _b (particularly the blue color difference component C _b ) that humans feel insensitive is subjected to a smoothing process. In particular, the degree of smoothing may be increased for these components. If comprised in this way, it will become possible to suppress degradation of the image quality which an observer feels efficiently.

  Further, in order to change the degree of smoothing processing for various pixel values, the filter size may be changed as appropriate, or the applied filter and filter coefficient may be changed as appropriate.

  Further, in the above description, the case where the smoothing process is performed in order to reduce the amount of information after encoding has been mainly described, but the low resolution of the pixel value component (particularly, the color component) of the target region. Similarly, it is possible to reduce the information amount of the encoded signal. Note that it is preferable to perform interpolation processing or the like appropriately at the time of decoding or reproduction to return the number of pixel value components of the target area whose resolution has been reduced to the number before the resolution reduction.

  Further, although the HMD has been described as an example of the image display device, the present invention can be applied to, for example, a head-up display (HUD).

  As mentioned above, although each embodiment of the present invention was described, the scope of the present invention is not limited to this, and can be executed with various modifications without departing from the gist of the invention.

  The present invention is applicable to HMD and HUD.

1, 1a Image display device 2 Frame 3 Display unit 3
31 Lens for right eye 32 Projection unit 4 Lens for left eye 5 Imaging unit 6 Body unit 7 Cable 11 Real image encoding unit 11a Local background image encoding unit 11b First local background image encoding unit 12 Real image transmitting unit 12a Local Background image transmitting unit 12b First local background image transmitting unit 13 External image receiving unit 13b First external image receiving unit 14 External image decoding unit 14b First external image decoding unit 20, 20a Image processing device 21 Real image receiving unit 21a Local background image receiving unit 21b Second local background image receiving unit 22 Real image decoding unit 22a Local background image decoding unit 22b Second local background image decoding unit 23, 23a Local background image generation unit 23b First local background image generation Unit 24 external image update processing unit 24b second external image update processing unit 25 external image encoding unit 25b second external image encoding unit 26 External image transmission unit 26b Second external image transmission unit 31 Second external image generation unit 100 First image display device 101 Second image display device S, Sa, Sb Image display system HOE holographic optical element

Claims (15)

An external image input from the outside of the device and a local background image showing a real image visually recognized by an observer are compared for each corresponding position, and an area similar to the local background image of the external image is set as a target area. An external image update processing unit that performs an update process on the target area of the external image, which is a process of setting and reducing sharpness;
An image processing apparatus comprising: an external image encoding unit that encodes the external image after the update process.   The image processing apparatus according to claim 1, wherein the update process performed by the external image update processing unit is a smoothing process.   The image processing apparatus according to claim 1, wherein the update process performed by the external image update processing unit is a resolution reduction process. The pixel value of the pixel of the external image includes a luminance component and a color difference component , and the external image update processing unit performs the update process with a small effect of reducing the sharpness on the luminance component , or The image processing apparatus according to any one of claims 1 to 3, wherein the update process is not performed, and the update process having a great effect of reducing a sharpness is performed on the color difference component .   The external image update processing unit compares the local background image and the external image for each block that is a region of a predetermined size, and is similar to the block of the local background image among the blocks of the external image The image processing apparatus according to claim 1, wherein an object is the target area. The external image update processing unit compares representative values of the blocks of the external image and the local background image,
The representative value is a median value of pixel values of pixels included in the block, an average value of pixel values of pixels included in the block, and a DC component value obtained by converting the pixel value into a frequency domain signal. The image processing apparatus according to claim 5, wherein the calculation is performed using any one of them.   The image processing apparatus according to claim 5 or 6, wherein when the variance of pixel values of pixels belonging to the block of the external image is larger than a predetermined value, the block is not set as the target region. .   The external image update processing unit divides each of the external image and the local background image into regions where pixel values are substantially equal, and attaches labels according to pixel values of pixels belonging to the region, The image processing apparatus according to any one of claims 1 to 4, wherein when the image and the local background image are overlapped, a region that overlaps a region having a substantially equal label is set as a target region.   The external image update processing unit sets the target region by dividing the pixel value of each pixel of the external image and the local background image and then dividing the pixel value for each region. Item 9. The image processing apparatus according to Item 8.   The external image update processing unit performs linear separation on pixel values of respective pixels included in the target region set in the external image and a corresponding region corresponding to the target region of the local background image. The image processing apparatus according to claim 1, wherein the target region is excluded when the separation is possible. The image processing apparatus according to any one of claims 1 to 10,
An image display device communicating with the image processing device,
The image display device
An external image reception processing unit that receives and processes the external image after the update process that is transmitted from the image processing apparatus;
A display unit that displays the external image after the update process and allows a real image to be transmitted and visible;
An imaging unit that generates a real image showing the real image by imaging;
A real image transmission processing unit that transmits the real image to the image processing device,
The image processing apparatus is
A real image reception processing unit for receiving and processing the real image to be transmitted from the image display device;
A local background image generation unit that generates the local background image from the real image, the local background image being an image showing a superimposed portion of the real image that is visually recognized through the external image after the update process displayed on the display unit;
An image display system, further comprising: an external image transmission processing unit configured to transmit the external image after the update process to the image display device. The image processing apparatus according to any one of claims 1 to 10,
An image display device communicating with the image processing device,
The image display device
An external image reception processing unit that receives and processes the external image after the update process that is transmitted from the image processing apparatus;
A display unit that displays the external image after the update process and allows a real image to be transmitted and visible;
An imaging unit that generates a real image showing the real image by imaging;
A local background image generation unit that generates the local background image indicating the superimposed portion of the real image that is visually recognized through the external image after the update process displayed on the display unit;
A local background image transmission processing unit for transmitting the local background image to the image processing device,
The image processing apparatus is
A local background image reception processing unit for receiving and processing the local background image transmitted from the image display device;
An image display system, further comprising: an external image transmission processing unit configured to transmit the external image after the update process to the image display device. In an image display device that displays an image,
The image display device communicates an image with another image processing device having the same configuration as the image display device,
An external image generation unit for another device that generates an external image to be displayed on the other image display device;
A local background image reception processing unit for other machine that receives and processes a local background image for other machine that is transmitted from the other image display device;
The external image for the other device and the local background image for the other device are compared for each corresponding position, and an area similar to the local background image for the other device of the external image for the other device is set as a target region, and is clear An external image update processing unit for other device that performs an update process, which is a process of reducing the degree, on the target area of the external image for other device;
An external image transmission unit for other device that transmits the external image for other device after the update process to the other image display device;
A self-machine external image reception processing unit for receiving and processing a self-machine external image to be transmitted from the other image display device;
A display unit that displays the external image for the own device and allows the user to visually recognize a real image;
An imaging unit that generates a real image showing the real image by imaging;
A local background image generation unit for the own device that generates a local background image for the own device that shows a superimposed portion of the real image visually recognized through the external image for the own device displayed on the display unit;
An image display device comprising: a local background image transmission processing unit for own device that transmits the local background image for own device to the other image display device.   The image display device according to claim 13, wherein the external image generating unit for other devices generates the external image for other devices from the real image generated by the imaging unit. A first step of comparing, for each corresponding position, an external image input from the outside of the device and a local background image indicating a real image visually recognized by an observer;
A second step of setting, as a target region, a region determined by the first step to be similar to the local background image of the external image;
A third step of performing an update process, which is a process of reducing the sharpness, on the target area set by the second step of the external image;
And a fourth step of encoding the external image after the update process.

Images