JP2021175063A - Method for controlling image display device, and image display device - Google Patents

Abstract

To provide a method for controlling an image display device that can prevent an increase in processing load in association with detection of a communication error, and an image display device.SOLUTION: A projector 2 performs Miracast connection with a computer 1, receives image information encoded in H.264 or H. 265 and including a key frame and a difference frame represented by a difference from the key frame from the computer 1 by NAL unit, detects a start code added to the NAL unit, decodes the received image information, displays an image based on the decoded image information, and when the start code is not detected, requests the computer 1 to re-transmit the key frame.SELECTED DRAWING: Figure 5

Description

The present invention relates to a control method for an image display device and an image display device.

As shown in Patent Document 1, a technique of adding a CRC (Cyclic Redudancy Check) code to each basic unit of image information is known in order to detect a communication error when image information is wirelessly transmitted to another device. ing. The device that receives this image information checks each basic unit using the added CRC code to determine whether or not a communication error has occurred, and if it is determined to be a communication error, the transmitting side. Requests the device of the above to retransmit the image information.

Japanese Unexamined Patent Publication No. 2008-9108

However, the detection of a communication error using the CRC code involves an operation, so there is a problem that the processing load increases. Therefore, in a situation where it is necessary to process image information of a moving image having a large amount of data at high speed, smooth image display may be hindered.

The control method of the image display device is as follows: H. 264 or H. An image information including a key frame encoded by 265 and a difference frame represented by a difference from the key frame is received from the image supply device in units of NAL units, and a start code added to the NAL unit is received. Is detected, the image information is decoded, an image is displayed based on the decoded image information, and when the start code is not detected, the image supply device is requested to retransmit the keyframe. ..

The image display device is an image display device that is wirelessly connected to the image supply device, and is an image display device. 264 or H. A communication unit that receives image information including a key frame encoded by 265 and a difference frame represented by a difference from the key frame from the image supply device in units of NAL units, and a decoding that decodes the image information. A processing unit, a display unit that displays an image based on the image information decoded by the decoding processing unit, a detection unit that detects a start code added to the NAL unit, and the start code by the detection unit. The image supply device is provided with a request processing unit that requests the image supply device to retransmit the key frame when the image is not detected.

Explanatory drawing which shows the image display system. A block diagram showing a schematic configuration of a computer. The block diagram which shows the schematic structure of a projector. The block diagram which shows the schematic structure of the image projection part provided in a projector. A flowchart for explaining the operation of the projector when performing mirroring.

Hereinafter, the image display system 100 of this embodiment will be described with reference to the drawings.
FIG. 1 is an explanatory diagram showing an image display system 100 of the present embodiment.
As shown in FIG. 1, the image display system 100 includes a computer 1 as an image supply device and a projector 2 as an image display device. The computer 1 is, for example, a notebook-type personal computer, and is configured to integrally include a display unit 12 for displaying an image, a keyboard 16 as an input device, and the like. The projector 2 receives the supply of image information from the computer 1 and projects an image corresponding to the image information on a projection surface SC such as a screen or a wall surface. The computer 1 and the projector 2 are connected by a wireless LAN (Local Area Network), and image information and the like are wirelessly supplied from the computer 1 to the projector 2.

In this embodiment, the computer 1 and the projector 2 correspond to Miracast (registered trademark). Miracast is a communication standard for transmitting a data stream such as an image using Wi-Fi Direct (registered trademark) that enables P2P (Peer to Peer) type wireless connection as a communication path. RTP (Real Time Transport Protocol) that operates on UDP (User Datagram Protocol) is used as the data stream transmission protocol in Miracast, and RTSP that operates on TCP (Transmission Control Protocol) as the protocol that controls transmission. (Real Time Streaming Protocol) is used. By transmitting an image from the computer 1 to the projector 2 by this Miracast, so-called mirroring is possible in which the image displayed on the display unit 12 of the computer 1 is also displayed on the projector 2. That is, when the projector 2 mirrors the computer 1 by Miracast, the display image IM2 projected by the projector 2 becomes the same as the display image IM1 displayed on the display unit 12 of the computer 1.

FIG. 2 is a block diagram showing a schematic configuration of the computer 1.
As shown in FIG. 2, the computer 1 includes a control unit 10, a storage unit 11, a display unit 12, a display control unit 13, an image processing unit 14, a wireless communication unit 15, and a keyboard 16. It is configured.

The control unit 10 includes one or a plurality of processors, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The control unit 10 controls the operation of the computer 1 by operating according to the program stored in the ROM or the program read from the storage unit 11 into the RAM.

The storage unit 11 includes a storage device such as a hard disk drive or a solid state drive. The storage unit 11 stores the installed OS (Operating System), application programs, various data, and the like.

The display unit 12 is configured to include a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display, and displays an image based on image information.

The display control unit 13 is composed of one or a plurality of processors and the like, and controls the display of an image by the display unit 12 based on the control of the control unit 10. The display control unit 13 is connected to a VRAM (Video RAM) (not shown), and based on the control of the control unit 10, an image is stored in the VRAM according to the OS, application program, various data, and the like. The information-based image is displayed on the display unit 12. The display control unit 13 may be configured by a dedicated processing device such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The image processing unit 14 performs predetermined processing on the image transmitted by Miracast based on the control of the control unit 10. Specifically, the image processing unit 14 acquires image information corresponding to the display image IM1 of the display unit 12 from the display control unit 13, and is a moving image coding standard used in Miracast. 264 or H. Image information is encoded according to 265.

H. 264 or H. The image information encoded in 265 includes a key frame encoded without using the inter-frame prediction and a difference frame encoded using the inter-frame prediction and represented by a difference from the key frame. In this way, the image information including the key frame and the difference frame is divided into basic units called NAL (Network Abstraction Layer) units suitable for transmission on the network and encapsulated. A predetermined bit string called a start code is added to the head of each NAL unit, and it is possible to identify the start position of the NAL unit. The image processing unit 14 transmits the processed image information to the projector 2 via the wireless communication unit 15. The image processing unit 14 may be configured by the processor constituting the control unit 10 executing a predetermined program, or may be configured by another processor or a dedicated processing device including an ASIC, FPGA, or the like. May be good.

The wireless communication unit 15 is configured to include a wireless LAN device that communicates by wireless LAN, and wirelessly communicates with an external device such as a projector 2 based on the control of the control unit 10. When mirroring by Miracast, the wireless communication unit 15 makes a Miracast connection with the projector 2 based on the control of the control unit 10, and transmits the encoded image information to the projector 2 by RTP. Further, when the wireless communication unit 15 receives the control information from the projector 2 by the RTSP, the wireless communication unit 15 outputs the received control information to the control unit 10.

The keyboard 16 is an input device that accepts input of characters, control codes, and the like by key operation. The keyboard 16 transmits an operation signal based on the operation performed by the user to the control unit 10.

Next, the configuration of the projector 2 will be described.
FIG. 3 is a block diagram showing a schematic configuration of the projector 2, and FIG. 4 is a block diagram showing a schematic configuration of an image projection unit 27 provided in the projector 2.
As shown in FIG. 3, the projector 2 includes a control unit 20, a storage unit 21, an input operation unit 22, a wireless communication unit 23 as a communication unit, an image buffer 24, a decoding processing unit 25, and image processing. It is configured to include a unit 26 and an image projection unit 27.

The control unit 20 is configured to include one or a plurality of processors, and controls the operation of the projector 2 by operating according to a control program stored in the storage unit 21. Further, the control unit 20 includes a detection unit 20a and a request processing unit 20b as functional blocks realized by the program. The detection unit 20a has a function of detecting the start code attached to the NAL unit, and the request processing unit 20b has a function of requesting the computer 1 to retransmit the keyframe.

The storage unit 21 includes a RAM which is a volatile memory, a ROM which is a non-volatile memory, and the like. The RAM is used for temporary storage of control data and the like, and the ROM stores a control program, control data and the like for controlling the operation of the projector 2.

The input operation unit 22 includes a plurality of operation keys for the user to give various instructions to the projector 2. The operation keys included in the input operation unit 22 are a "power key" for switching the power on and off, a "menu key" for displaying a setting menu for making various settings, and an item of the setting menu. There are "direction keys" and so on. When the user operates various operation keys of the input operation unit 22, the input operation unit 22 outputs an operation signal according to the operation content of the user to the control unit 20. A remote controller (not shown) capable of remote control may be used as the input operation unit 22. In this case, the remote controller transmits an infrared operation signal according to the operation content of the user, and the remote controller signal receiving unit (not shown) receives this and transmits it to the control unit 20.

The wireless communication unit 23 is configured to include a wireless LAN device that communicates by wireless LAN, and wirelessly communicates with an external device such as a computer 1 based on the control of the control unit 20. As described above, the projector 2 is compatible with Miracast, and can perform mirroring to display the display image IM1 displayed by the computer 1 on the projector 2. In this case, the wireless communication unit 23 makes a Miracast connection with the computer 1 based on the control of the control unit 20, and receives the image information from the computer 1 on the Miracast. The control unit 20 stores the image information received by the wireless communication unit 23 in the image buffer 24. Further, the wireless communication unit 23 can transmit control information to the computer 1 or the like by RTSP based on the control of the control unit 20.

The image buffer 24 is a buffer that is composed of a RAM or the like and stores image information received by the wireless communication unit 23.

The decoding processing unit 25 decodes the image information transmitted by Miracast, that is, the image information stored in the image buffer 24, based on the control of the control unit 20, and outputs the decoded image information to the image processing unit 26.

The image processing unit 26 performs various processes on the decoded image information based on the control of the control unit 20. The processing performed by the image processing unit 26 on the image information includes a processing for adjusting the image quality, a processing for superimposing a menu image, and the like. The image processing unit 26 outputs the processed image information to the image projection unit 27. The decoding processing unit 25 and the image processing unit 26 may be configured by one or a plurality of processors or the like, or may be configured by a dedicated processing device such as an ASIC or FPGA.

As shown in FIG. 4, the image projection unit 27 includes a light source 31, three liquid crystal light bulbs 32R, 32G, 32B as a light modulator, a projection lens 33 as a projection optical system, a light bulb drive unit 34, and the like. It is configured. The image projection unit 27 corresponds to a display unit, and the light emitted from the light source 31 is modulated by the liquid crystal light valves 32R, 32G, 32B to form image light, and this image light is transmitted from the projection lens 33. The image is projected and displayed on the projection surface SC.

The light source 31 includes a discharge type light source lamp such as an ultra-high pressure mercury lamp or a metal halide lamp, or a solid light source such as a light emitting diode or a semiconductor laser. The light emitted from the light source 31 is converted into light having a substantially uniform brightness distribution by an integrator optical system (not shown), and separated into red, green, and blue light components, which are the three primary colors of light, by a color separation optical system (not shown). After that, it is incident on the liquid crystal light valves 32R, 32G, and 32B, respectively.

The liquid crystal light bulbs 32R, 32G, and 32B are each composed of a transmissive liquid crystal panel (not shown) in which a liquid crystal is sealed between a pair of transparent substrates. A rectangular image forming region 32i composed of a plurality of pixels arranged in a matrix is formed on each liquid crystal panel, and a driving voltage can be applied to the liquid crystal for each pixel.

The light bulb drive unit 34 forms an image in the image forming region 32i of the liquid crystal light bulbs 32R, 32G, 32B. Specifically, the light valve drive unit 34 applies a drive voltage corresponding to the image information input from the image processing unit 26 to each pixel of the image forming region 32i, and allows each pixel to transmit light according to the image information. Set to rate. The light emitted from the light source 31 is modulated for each pixel by passing through the image forming regions 32i of the liquid crystal light valves 32R, 32G, 32B, and image light corresponding to the image information is formed for each colored light. The formed image light of each color is combined for each pixel by a color synthesis optical system (not shown) to become image light representing a color image, and is magnified and projected onto the projection surface SC by the projection lens 33. As a result, an image based on the image information transmitted from the computer 1, that is, the same image as the display image IM1 displayed on the display unit 12 of the computer 1 is displayed as the display image IM2 on the projection surface SC (FIG. 1).

Next, the operation of the image display system 100 will be described.
FIG. 5 is a flowchart for explaining the operation of the projector 2 when performing mirroring, that is, the control method of the projector 2.
When the user instructs the computer 1 to start mirroring while the projector 2 is waiting for mirroring, the computer 1 requests the projector 2 to connect to Miracast via the wireless communication unit 15. conduct. When the control unit 20 of the projector 2 receives this request via the wireless communication unit 23, the control unit 20 starts the operation according to the flow shown in FIG.

In step S101, the control unit 20 makes a Miracast connection with the computer 1 that made the request and starts mirroring. After that, the computer 1 periodically encodes the image information of the display image IM1 and sequentially transmits the image information to the projector 2.

In step S102, the control unit 20 receives the image information transmitted from the computer 1 in units of NAL units via the wireless communication unit 23, and stores the received image information in the image buffer 24. Although omitted in the flowchart, when new image information is stored in the image buffer 24, the decoding processing unit 25 decodes the stored image information and outputs it to the image processing unit 26. Then, when the image processing unit 26 performs various processes based on the control of the control unit 20 and outputs the processed image information to the image projection unit 27, the image corresponding to the image information is output from the image projection unit 27 to the projection surface SC. Is projected on. If the received image information is a key frame, the control unit 20 keeps the received image information stored in the image buffer 24 until the next key frame is received. The decoding processing unit 25 uses this key frame when decoding the difference frame received thereafter.

In step S103, the detection unit 20a of the control unit 20 detects the start code added to each NAL unit with respect to the image information stored in the image buffer 24. Then, in step S104, the control unit 20 determines whether or not the start code is detected by the detection unit 20a, and if it is detected, shifts the process to step S105. On the other hand, if the start code is not detected, that is, if the start code is not detected, the process proceeds to step S107. The non-detection of the start code may occur when a communication error occurs due to radio wave interference, a malfunction of the computer 1, or the like, that is, when image information is not normally transmitted or received.

When the start code is detected and the process shifts to step S105, the control unit 20 determines whether or not the computer 1 has requested the disconnection of the Miracast connection via the wireless communication unit 23. For example, when the user performs an operation for disconnecting the Miracast connection to the computer 1, the computer 1 requests the projector 2 to disconnect the Miracast connection. When the computer 1 requests disconnection, the control unit 20 of the projector 2 shifts the process to step S106, disconnects the Miracast connection with the computer 1, and ends the flow. On the other hand, when there is no request for disconnection from the computer 1, the control unit 20 returns the process to step S102 and repeats the above operation.

If the start code is not detected in step S104 and the process shifts to step S107, the control unit 20 requests the computer 1 to retransmit the keyframe via the wireless communication unit 23. Specifically, the request processing unit 20b of the control unit 20 transmits a wfd-idr-request message, which is control information for requesting the retransmission of the key frame, to the computer 1 by RTSP. When the computer 1 receives this control information, it encodes the image information of the display image IM1 to be transmitted next as a key frame and transmits it to the projector 2.

In step S108, the control unit 20 clears the image buffer 24. Specifically, the control unit 20 erases the image information of the key frame stored in the image buffer 24. The timing of clearing the image buffer 24 may be before requesting the retransmission of the key frame.

In step S109, the control unit 20 determines whether or not the non-detection of the start code in step S104 has occurred continuously a predetermined number of times. Then, if the predetermined number of times has not been reached, the process is returned to step S102, and the above operation is repeated. On the other hand, when the non-detection of the start code occurs continuously a predetermined number of times, the control unit 20 determines that recovery is difficult, shifts the process to step S106, and disconnects the Miracast connection with the computer 1. Perform processing and end the flow.

Since the projector 2 operates as described above, the computer 1 shall continue to transmit only the difference frame after transmitting the key frame once, unless the projector 2 requests the re-transmission of the key frame. Is also possible. In this case, it is possible to reduce the amount of image information transmitted from the computer 1.

As described above, according to the projector 2 of the present embodiment and the control method thereof, when the start code attached to the NAL unit is not detected, the re-transmission of the key frame is requested. That is, since the projector 2 detects the occurrence of a communication error based on the presence or absence of the start code, it is possible to suppress an increase in the processing load as compared with the detection involving calculation such as CRC. As a result, smooth image display is possible even in a situation where it is necessary to process image information of a moving image having a large amount of data at high speed.

Further, according to the present embodiment, when the non-detection of the start code is continuously performed a predetermined number of times, the Miracast connection with the computer 1 is disconnected, so that the image display is continuously disturbed due to a communication error. Can be suppressed.

The above embodiment may be changed as follows.

In the above embodiment, the computer 1 requests the projector 2 for the Miracast connection, but the projector 2 may request the computer 1 for the Miracast connection.

In the above embodiment, the computer 1 is adopted as the image supply device, but the image supply device is not limited to the computer 1, and may be another device such as a smartphone or a tablet terminal. Further, Miracast is not limited to the mode used for mirroring, and may be a mode in which a moving image captured by an image pickup device such as a camera is transferred to an image display device such as a projector 2 and displayed.

In the above embodiment, the mode in which the computer 1 and the projector 2 communicate with each other by Miracast is shown, but the communication standard is not limited to Miracast, and other communication standards such as WiDi and AirPlay (registered trademark) may be adopted. good.

In the above embodiment, the transmissive liquid crystal light bulbs 32R, 32G, and 32B are used as the light modulation device, but a reflection type light modulation device such as a reflection type liquid crystal light bulb can also be used. Further, a digital mirror device or the like that modulates the light emitted from the light source 31 can also be used by controlling the emission direction of the incident light for each micromirror as a pixel. Further, the configuration is not limited to the configuration in which a plurality of optical modulation devices are provided for each color light, and a configuration in which a plurality of color lights are modulated by time division by one optical modulation device may be used.

In the above embodiment, the projector 2 is adopted as the image display device, but the image display device is not limited to the projector 2, and may be another image display device such as a liquid crystal display or an organic EL display. In these cases, a display screen composed of a liquid crystal panel, an organic EL panel, or the like corresponds to a display unit.

1 ... Computer, 2 ... Projector, 10 ... Control unit, 11 ... Storage unit, 12 ... Display unit, 13 ... Display control unit, 14 ... Image processing unit, 15 ... Wireless communication unit, 16 ... Keyboard, 20 ... Control unit, 20a ... Detection unit, 20b ... Request processing unit, 21 ... Storage unit, 22 ... Input operation unit, 23 ... Wireless communication unit, 24 ... Image buffer, 25 ... Decoding processing unit, 26 ... Image processing unit, 27 ... Image projection unit , 31 ... light source, 32R, 32G, 32B ... liquid crystal light valve, 32i ... image forming area, 33 ... projection lens, 34 ... light valve drive unit, 100 ... image display system, IM1, IM2 ... display image, SC ... projection surface ..

Claims (5)

Wirelessly connect to the image supply device,
H. 264 or H. Image information including keyframes encoded by 265 and difference frames represented by differences from the keyframes is received from the image supply device in units of NAL units.
The start code attached to the NAL unit is detected and
Decode the image information and
An image is displayed based on the decoded image information, and the image is displayed.
A control method for an image display device that requests the image supply device to retransmit the keyframe when the start code is not detected. The first aspect of the present invention is a control method for an image display device.
A method for controlling an image display device that receives the image information from the image supply device by Miracast (registered trademark). The method for controlling an image display device according to claim 1 or 2.
A control method of an image display device that clears a buffer for storing the image information when requesting the image supply device to retransmit the keyframe. The method for controlling an image display device according to any one of claims 1 to 3.
A method for controlling an image display device that disconnects the connection with the image supply device when the start code is not continuously detected a predetermined number of times. An image display device that is wirelessly connected to an image supply device.
H. 264 or H. A communication unit that receives image information including keyframes encoded by 265 and difference frames represented by differences from the keyframes from the image supply device in units of NAL units.
A decoding processing unit that decodes the image information,
A display unit that displays an image based on the image information decoded by the decoding processing unit, and
A detector that detects the start code attached to the NAL unit,
A request processing unit that requests the image supply device to retransmit the keyframe via the communication unit when the start code is not detected by the detection unit.
An image display device equipped with.

Images