WO2015132957A1

WO2015132957A1 - Video device and video processing method

Info

Publication number: WO2015132957A1
Application number: PCT/JP2014/055980
Authority: WO
Inventors: 甲　展明; 浩朗伊藤; 稲田　圭介; 健木佐貫; 吉澤　和彦; 益岡　信夫
Original assignee: 日立マクセル株式会社
Priority date: 2014-03-07
Filing date: 2014-03-07
Publication date: 2015-09-11

Abstract

An image pickup apparatus (1) transmits, to a display apparatus (2), a video signal of a first frame rate, interpolation data to be used for a frame rate conversion, and scan information (number of simultaneous scans, scan directions, line/frame shutters). The display apparatus (2) uses the interpolation data to convert the received video signal of the first frame rate to a video signal of a second frame rate higher than the first frame rate, and displays the video signal as converted. At this moment, the display apparatus (2) determines, on the basis of the received scan information, a display scan scheme (number of simultaneous scans, scan directions, timings to light the backlight) of a display element (26). Alternatively, the image pickup apparatus (1) acquires, from the display apparatus (2), display scan information of the display element (26) and determines an image pickup scan scheme of an image pickup element (11) on the basis of the acquired display scan information. In this way, the magnitude and variation of the delay time from an image pickup timing of the image pickup element (11) until a display timing of the display element (26) can be reduced.

Description

Video equipment and video processing method

The present invention relates to a video device and a video processing method for transmitting a video signal from a camera or the like and performing video processing for displaying the video signal with good timing on a display device.

In relation to a frame rate conversion (FRC) technique for generating a video signal having a frame rate higher than that of an input video signal, Patent Document 1 discloses that “a video signal having a frame rate higher than the frame rate of a video signal to be encoded and transmitted”. It is possible to perform higher-accuracy FRC processing by performing motion determination and transmitting the motion determination result separately. Further, Patent Document 2 discloses that reference control information including a motion vector is superimposed and transmitted in a blanking period of a video signal, that is, “the playback device 1 and the display device 5 are in accordance with the HDMI (registered trademark) standard. In the case of inputting / outputting a signal, “the encoding unit 3 superimposes the reference control information as a kind of AUX data in the data island period of FIG. 5 and the decoding unit 6 separates the reference control information from the data island period”. It is described.

JP 2013-174798 A (see paragraph 0049) JP 2007-274679 A (see paragraph 0039)

When a video imaged by a camera or the like is transmitted as a video signal and its frame interpolation data and displayed by interpolating the frame on a display device, there is a considerable delay time from the imaging timing to the display timing. Also, depending on the combination of the scanning method of the camera and the scanning method of the display device, the delay time may vary within the screen and image distortion may occur. The occurrence of such delay time and variations in the screen become obstacles when performing precision work while watching fast moving camera images, and must be reduced as much as possible. The size of the delay time includes a portion depending on the scanning method, the video signal transmission method, and the frame interpolation method of the camera and the display device, respectively, but was not particularly taken into consideration in the prior art including the patent document.

An object of the present invention is to reduce the size of delay time and the variation in the screen from the imaging timing of the camera to the display timing of the display device.

In order to solve the above problems, for example, the configuration described in the claims is adopted. For example, in a video device that displays an output video of a video output device, from the video output device, a first video signal, interpolation data for interpolating a frame of the first video signal, An input unit that receives scanning information when generating the video, and a second that interpolates a frame of the first video signal using the first video signal and the interpolation data received by the input unit. A display signal processing unit for generating the video signal, a display element for displaying the first video signal, the second video signal generated by the display signal processing unit, the display signal processing unit, and the display element A control unit for controlling the display element, wherein the control unit determines a display scanning method of the display element based on the scanning information received by the input unit, and the display signal processing unit is controlled by the control unit. The determined display element Generating said second video signal based on the display scanning method.

Alternatively, in a video device that outputs video to a display device, the second video signal having a frame rate lower than the frame rate of the first video signal from the first video signal output by the video source, and the second A video signal processing unit that generates interpolation data for interpolating a frame of the video signal, an output unit that transmits the second video signal and the interpolation data to the display element, and the video signal A control unit that controls a processing unit, wherein the control unit acquires display scan information of the display device, information on the second video signal that can be processed, and the interpolation data, and the display scan information and the A format of the second video signal output from the video signal processing unit and the interpolation data is determined based on information on the second video signal that can be processed and the interpolation data, and the video signal is determined. Processing unit, the generating the interpolated data and the second image signal based on the format of the control unit has determined.

According to the present invention, it is possible to realize a system in which the size and variation of the delay time from the imaging timing of the camera to the display timing of the display device are reduced.

1 is a block diagram illustrating a configuration of an imaging / display system according to Embodiment 1. FIG. 4A and 4B illustrate various scanning methods for an image sensor and a display element. The figure explaining the imaging of an image signal, transmission, and a display timing (when an imaging scanning system is a single scan). The figure explaining the imaging of an image signal, transmission, and a display timing (when an imaging scanning system is a dual scan). The figure which put together the delay time in each scanning system. FIG. 4 is a diagram illustrating an overall configuration of an imaging / display system according to a second embodiment. The figure which shows the internal structure of the encoder 4. FIG. The figure which shows the internal structure of STB7. FIG. 10 is a diagram for explaining imaging, transmission, and display timing of a 3D video signal according to Embodiment 3 (when the imaging scanning method is single scan). The figure explaining the imaging of 3D video signals, transmission, and a display timing (when an imaging scanning system is a dual scan). The figure which put together the delay time in each scanning system of 3D display. FIG. 10 is a diagram illustrating a configuration example of a display device according to a fourth embodiment. The figure which shows the display timing at the time of shortening the scanning time of a display element. FIG. 10 is a diagram illustrating a configuration example of a display device according to a fifth embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram illustrating the configuration of the imaging / display system according to the first embodiment. In the first embodiment, for example, a consumer imaging device (camera) 1 and a display device (display) 2 are connected by HDMI (abbreviation of High Definition Multimedia Interface, a registered trademark of HDMI, LLC), etc. Assume a system in which a video signal is transmitted to the display device 2 to display a video.

The internal configuration of the imaging apparatus 1 includes an imaging device 11,

memories

12 and 16, a superimposition unit 13, an 8B / 10B encoder 14, an output unit 15, a compression unit 17, an error correction coding unit (ECC) 18, and a control unit 19. .

On the other hand, the internal configuration of the display device 2 includes an input unit 21, a 10B / 8B decoder 22, a separation unit 23, a display signal processing unit 31, a display element 26, a control unit 27, and an error correction decoding unit 28. The display signal processing unit 31 includes

memories

24 and 30, a scanning processing unit 25, and an expansion unit 29.

In this system, for example, the imaging device 1 captures an image at a frame rate of 120 Hz, and interpolated data generated by compressing the even frame is superimposed on the odd frame and transmitted to the display device 2, and the display device 2 interpolates with the odd frame. The even frame is restored from the data, and the frame rate is returned to 120 Hz together with the odd frame to display the video. The operation of each part will be described later.

FIG. 2 is a diagram for explaining various scanning methods of the image sensor and the display element.
(A) is a single scan method, in which pixels are horizontally scanned from left to right as indicated by an arrow 111, and pixels are vertically scanned from top to bottom as indicated by an arrow 112. This is a general scanning method, and is adopted by many image sensors and display elements.

(B) (c) (d) is a dual scan method, which divides the screen into two screens, an upper screen and a lower screen, and simultaneously scans the upper screen and the lower screen. When the number of pixels increases as the resolution increases, the selection time for one row and each pixel is shortened, and high-speed driving of the image sensor and the display element becomes difficult. Therefore, by dividing the screen into upper and lower parts and scanning them simultaneously, the scanning time for each row can be reduced by half compared to the single scan method, and high-speed driving can be supported. That is, as indicated by arrows 121, 131, and 141, horizontal scanning is performed from left to right, while vertical scanning is simultaneously performed on the upper screen and the lower screen.

(B) is vertical scanning from top to bottom as indicated by

arrows

122 and 123, and is called up-down screen same direction scanning. (C) performs vertical scanning from the center to the top and bottom as indicated by

arrows

132 and 133, and is referred to as vertical screen reverse scanning (center start). In (d), vertical scanning is performed from the upper part and the lower part to the central part as indicated by arrows 142 and 143, which is referred to as vertical screen reverse scanning (up / down start).

In the dual scan method of (b) to (d), moving image display is performed by matching the timing of imaging or displaying the boundary between the upper screen and the lower screen, that is, the pixel immediately above and the pixel immediately below the center of the screen. To smooth. For this reason, in the same direction scanning of the upper and lower screens in (b), the scanning of the lower screen may be delayed by one frame compared to the upper screen. (C) In the upper and lower screen reverse scanning in (d), it is preferable that the scanning timing of the upper screen and the lower screen is substantially the same.

FIG. 3 is a diagram for explaining an example of imaging, transmission, and display timing of a video signal in the imaging / display system of FIG. Here, it is assumed that the image pickup device 11 performs a single scan at a frame rate of 120 Hz (FIG. 2A). The horizontal axis indicates time, and the vertical axis indicates the scanning position in the vertical direction. A thick arrow (for example, reference numeral 201) indicates scanning timing for odd frames, and a thin arrow (for example, reference numeral 202) indicates scanning timing for even frames. Further, a solid line arrow (for example, reference numeral 211) indicates an uncompressed video signal, and a broken line arrow (for example, reference numeral 212) indicates interpolation data. As a unit of time axis, a period T of one frame corresponding to a frame rate of 120 Hz is used.

(A) is the imaging timing, the imaging device 11 performs a single scan at a frame rate of 120 Hz (period T), and the scanning timings of the first to sixth frames are indicated by arrows 201 to 206.

(B) is the transmission timing, and the odd-numbered

frames

201, 203, 205 output from the image sensor 11 are set to 60 Hz single scan signals 211, 213, 215 (time width 2T) to extend the time axis. Further, in order to restore the even frames 202, 204, and 206, 120 Hz interpolated

data

212, 214, and 216 are generated and superimposed on the 60 Hz single scan signal and output. For example, when transmitting by HDMI, it may be superposed as an Info packet format in the horizontal blanking period of a 60 Hz single scan signal.

(C) to (f) show display timings in various scanning methods by the display element 26, and display an image of each frame at a period T. (C) is a case of 120 Hz single scan, and the display timings of the first to fifth frames are indicated by 221 to 225, respectively. (D), (e), and (f) are 120 Hz dual scans, (d) is the same scan in the top and bottom screen, (e) is the top and bottom screen reverse scan (center start), and (f) is the top and bottom screen reverse scan. This is the case of (starting up and down). Display timings at the top of the screen are indicated by arrows 231 to 236, 251 to 255, 271 to 274, respectively, and display timings at the bottom of the screen are indicated by arrows 241 to 245, 261 to 265, and 281 to 284, respectively.

The operation of the imaging / display system will be described below with reference to FIGS.
First, the operation of the imaging apparatus 1 will be described. As shown in FIG. 3A, the image sensor 11 captures an image with a single scan at a frame rate of 120 Hz and outputs an uncompressed video signal. Of these, odd frames are stored in the memory 12 and even frames are stored in the memory 16. The odd-frame video signals (201, 203, 205 in FIG. 3A) accumulated in the memory 12 are read out over a time twice as long as the writing (about 2T) (60 Hz for transmission ( Video signals (period 2T) (FIG. 3 (b) 211, 213, 215) are generated. For this reason, at the end of transmission of the 60 Hz video signal (frame 211), a delay 219 corresponding to a vertical effective display period of 120 Hz (about 1 T) occurs compared to the end of scanning of the corresponding imaging signal (frame 201).

The compression unit 17 uses the odd frame and even frame video signals stored in the

memories

12 and 16 to generate interpolation data (120 Hz conversion data) for restoring the even frame video from the odd frame. The interpolation data may be any of difference information (equivalent to P frame) with the immediately preceding odd frame, mixing ratio or difference information between the preceding and succeeding odd frames (equivalent to B frame), motion vector related data, and the like.

The error correction encoding unit 18 performs error correction encoding on the interpolation data generated by the compression unit 17 in order to prevent errors in the process of transmitting the video signal to the display device 2. This is because the error that occurs in the interpolated data (even frame) is larger than the error that occurs in the uncompressed video signal (odd frame). In the error correction coding, the coding method may be selected by an instruction from the control unit 19 that detects the error rate of the transmission system. When the transmission system error rate is large, the quality of the display video can be ensured by increasing the compression rate of the compression unit 17 and reducing the interpolation data, and then selecting an error correction coding method with high error correction strength. it can.

The superimposing unit 13 superimposes the error correction encoded even frame interpolation data on the blanking period of the odd frame uncompressed video signal. For example, in HDMI, it is preferable to superimpose interpolation data on an Info frame packet subjected to error correction coding. The superimposed video signal is serialized (converted from 8-bit data to 10-bit data) by the 8B / 10B encoder 14, and becomes a non-compressed video signal interface compliant signal such as HDMI. Then, it is transmitted from the output unit 15 to the display device 2.

Among the transmission signals in FIG. 3B, the start timing of the even-

frame interpolation data

212, 214, and 216 is longer than the start timing of the corresponding even-numbered

frames

202, 204, and 206 in the imaging signal in FIG. Delayed by 218. This takes into account the processing time by the compression unit 17 and the error correction coding unit 18. In addition, the transmission time of the

interpolation data

212, 214, 216 can be set to the same time width as the imaging scanning time (1T) of the even frames 202, 204, 206, but the interpolation data is concentrated in time. The transmission band may be insufficient. Therefore, in this example, like the transmission signals 211, 213, and 215 of the 60 Hz uncompressed video signal, the transmission time of the

interpolation data

212, 214, and 216 is doubled and transmitted at a cycle of 2T.

In FIG. 3B, the transmission of the interpolation data is continued in the vertical blanking period of the 60 Hz uncompressed video signal to be transmitted, but the transmission may be paused. In addition, in the case where the compression unit 17 generates even-frame interpolation data (for example, 212) with reference to the preceding and following odd-numbered frames (for example, 201 and 203), the size of the delay time 218 is 1 frame (1T) or more. It becomes.

The control unit 19 is responsible for overall control of the imaging apparatus 1 and grasps the operation status of each component unit and gives necessary instructions. In particular, the control unit 19 reads an EDID (Extended Display Identification Data) in which information such as the display capability of the display device 2 that is the transmission destination of the output unit 15 is described, and determines whether it has a function capable of processing the interpolation data. To do. Then, the memory 16, the compression unit 17, and the error correction coding unit 18 are operated only when they can be processed, so that power saving can be achieved.

Further, the control unit 19 uses the transmission system of this embodiment, “60 Hz uncompressed video”, based on information such as transmission capability, error rate, power consumption, remaining battery level, image quality level requested by the user, and display delay time. It may be determined whether to select “signal + interpolated data transmission” and “120 Hz uncompressed video signal transmission” or “60 Hz uncompressed video signal (without interpolation data) transmission”, which is another transmission method.

Subsequently, the operation of the display device 2 will be described. The input unit 21 of the display device 2 receives the transmission signal (60 Hz uncompressed video signal and 120 Hz conversion data) of FIG. 3B transmitted from the imaging device 1. The 10B / 8B decoder 22 restores the received signal to a video signal. The separation unit 23 separates the signal, the 60 Hz uncompressed video signal (odd frame) is sent to the memory 24, and the 120 Hz conversion data (even number frame) and other metadata are sent to the error correction decoding unit (ECC) 28. Output.

The error correction decoding unit 28 outputs reliable interpolation data for 120 Hz after the error correction processing to the decompression unit 29, and is necessary for the 120 Hz video signal restoration that the 120 Hz conversion data is included. Parameter information and the like are transmitted to the control unit 27. As the separation unit 23 and the error correction decoding unit 28, an info frame extraction processing circuit defined by HDMI can be used.

The decompression unit 29 restores the even frame of the 120 Hz video signal from the 60 Hz uncompressed video signal obtained from the memory 24 (that is, the odd frame of the 120 Hz video signal) and the interpolation data for 120 Hz conversion obtained from the error correction decoding unit 28. To the memory 30. When the display element 26 is a single scan system (FIG. 3C), the scanning processing unit 25 converts the odd frame video signal (cycle 2T) in the memory 24 and the even frame video signal (cycle 2T) in the memory 30 respectively. The time axis is converted so as to scan in half the time (cycle T), and the result is combined and output to the display element 26.

The scanning processing unit 25 generates a video signal in accordance with the scanning method of the display element 26, and the display element 26 displays an image of each frame at the scanning timing shown in FIGS.

FIG. 3C shows a case where the display element 26 is a 120 Hz single scan system (FIG. 2A). The scanning start timing of each frame 221 to 225 of the 120 Hz video signal to be displayed is delayed by one frame period (1T) 228 from the scanning start timing of each frame 201 to 205 of the corresponding imaging signal (FIG. 5A). . This is because the frames 211 to 215 transmitted in the time width 2T are displayed with the time axis converted to the time width 1T and displayed without missing one frame of video. This is because it is necessary to receive and restore all the images for one frame to be displayed. For example, the display of the even-numbered frame 222 can be restored immediately after the interpolation data 212 is input. However, when the display ends, the display must be synchronized with the end of transmission 217 of the interpolation data 212.

Note that when the even frame 222 to be displayed is restored with reference to the

odd frames

211 and 213 on both the front and rear sides, the restoration is made after waiting for the input of the subsequent frame 213. Therefore, the display timing of each frame is further delayed by one frame and delayed by the time width 2T.

Although the above description is based on the case where the display element 26 is of the 120 Hz single scan scanning method, the same applies to the case of the 240 Hz single scan method. However, the scanning processing unit 25 uses the motion vector included in the 120 Hz conversion data obtained from the error correction decoding unit 28 together with the restoration to the 120 Hz uncompressed video signal, and converts the frame rate to 240 Hz. I do.

FIG. 3D shows a case where the display element 26 is a 120 Hz dual scan system (same direction scanning, FIG. 2B). In the dual scan method (same direction scanning), the screen is divided into upper and lower parts and simultaneously vertically scanned in the same direction from top to bottom. At this time, the lower screen is displayed with a delay of one frame in order to prevent a time lag of the video at the center of the screen, so that the entire screen is vertically scanned from the top to the bottom over a period of 2 frames (2T). For example, the transmitted first frame 211 is divided into a portion 231 that is displayed over the time 1T on the upper screen and a portion 241 that is displayed over the next time 1T on the lower screen.

In the dual scan method, there is no conversion of the time axis of the transmission signal and the display signal, and the time axis conversion process by the scanning processing unit 25 is not necessary. That is, since the display video is not lost in the middle of the frame due to the time axis conversion, the display start timing on the upper screen can be almost the same as the reception start timing of each frame. If the decompression unit 29 does not take much time to restore, the start point of the second frame 232 to be displayed may substantially coincide with the reception start point of the interpolation data 212 for second frame restoration.

The display timing at each position in the screen depends on the reception or restoration timing of the video signal at that position, that is, almost the transmission timing. For example, the connection portion (the center portion of the screen) between the upper screen and the lower screen is delayed by about ½ frame (0.5 T) compared to the imaging timing. The display end timing of the lower screen is delayed by about 1 frame (1T) from the imaging end timing. For example, the display end point of the first frame 241 is delayed by a time 248 from the imaging end point of the first frame 201. The delay time 248 is obtained by adding processing time and transmission delay time in the compression unit 17 and the expansion unit 29 to one frame (1T).

In order to adjust the scanning timing of the lower screen with a delay of one frame from the upper screen, for example, the lower frame second frame 242 is delayed by a horizontal blanking period of 120 Hz from the interpolation data 212 to be transmitted. As a result, it is delayed by a time 238 from the start timing of the imaged third frame 203. This delay time 238 is substantially equal to the delay time 218 of the 120 Hz conversion data.

FIGS. 3E and 3F show a case where the display element 26 is a dual scan system (reverse scanning), and FIG. 3E shows a case where scanning is simultaneously performed from the center to the upper and lower ends of the screen (FIG. 2C). , (F) shows a case where scanning is simultaneously performed from the upper end and the lower end of the screen to the center (FIG. 2 (d)). Even in these scans, the time axis conversion is unnecessary, and can be realized by changing the reading order of the

memories

24 and 30 in the display device 2 or changing the scanning order in the scanning processing unit 25.

When scanning is started from the center of the screen in FIG. 3E, the delay time from imaging to display causes a delay 258 of approximately 2 frames (2T) at the upper end of the screen and a delay 269 of approximately 1 frame at the lower end of the screen. Produce. In the center of the screen, a delay of 1.5 frames between them occurs.

When scanning from the top and bottom of the screen to the center of the screen in FIG. 3F, the delay time from imaging to display causes a delay of approximately 2 frames at the top of the screen and a delay of approximately 1 frame at the bottom of the screen. 289. At the center of the screen, a delay of 2.5 frames is generated by adding 0.5 frames to them.

From the above, when the image sensor 11 is single scan, the delay times are compared for each scanning method of the display element 26. In the case of the single scan display method (FIG. 3C), the entire screen is delayed by approximately one frame (1T). In the dual scan display method, the delay time is minimized in the case of the same direction scan method (FIG. 3D). In other words, there is a delay of approximately 1 frame at the bottom of the screen, but there is no delay at the top of the screen (delayed by the time required to compress / expand even frames), and there is a delay of approximately 0.5 frames at the center of the screen. This is advantageous.

The above description is based on the case where the image sensor 11 is a single scan method (FIG. 2A), but next, a case of a dual scan method (FIG. 2B) is shown.

FIG. 4 is a diagram illustrating another example of image signal imaging, transmission, and display timing. Here, it is assumed that the image sensor 11 performs dual scanning at a frame rate of 120 Hz (FIG. 2B).

(A) is the imaging timing, the imaging device 11 performs dual scanning at a frame rate of 120 Hz (period T), and the scanning timings of the first to sixth frames are indicated by arrows 301 to 306. However, the scanning time for one screen is about 2T.

(B) is the transmission timing, and the

odd frames

301, 303, and 305 output from the image sensor 11 are output as

frames

311, 413, and 315 at the same timing. The even frames 302, 304, and 306 are converted into 120 Hz interpolated

data

312, 314, and 316 and superimposed on the odd frames and output.

(C) to (f) show display timings in various scanning methods by the display element 26, and display an image of each frame at a period T. (C) is a case of 120 Hz single scan, and display timings 321 to 325 are shown for the first to fifth frames, respectively. (D), (e), and (f) are 120 Hz dual scans, (d) is the same scan in the top and bottom screen, (e) is the top and bottom screen reverse scan (center start), and (f) is the top and bottom screen reverse scan. This is the case of (starting up and down). Display timings at the top of the screen are indicated by arrows 331 to 336, 351 to 355, 371 to 374, respectively, and display timings at the bottom of the screen are indicated by arrows 341 to 345, 361 to 365, 381 to 384, respectively.

The operation of the imaging / display system will be described below with reference to FIGS.
First, as shown in FIG. 4A, the imaging device 11 in the imaging apparatus 1 captures an image with a dual scan of 120 Hz and outputs an uncompressed video signal. The imaging time for one screen is about 2T. Odd frames are stored in the memory 12 and even frames are stored in the memory 16. The odd-frame video signals (301, 303, and 305 in FIG. 4A) stored in the memory 12 are read at the same timing, and the 60-Hz video signals for transmission (311 in FIG. 4B) are read. 313, 315). Therefore, the memory 12 may be passed through and output to the superimposing unit 13.

The compression unit 17 uses the odd-numbered frame and even-numbered frame video signals stored in the

memories

12 and 16 to generate interpolation data for restoring the even-numbered frame video from the odd-numbered frame. Since a delay time 318 occurs for generating the interpolation data, the transmission start timing of the

interpolation data

312, 314, 316 is delayed by the time 318 from the imaging start timing of the corresponding even frames 302, 304, 306. Since the interpolation data generation method and the operations of the error correction encoding unit 18, the superimposing unit 13, the 8B / 10B encoder 14, and the control unit 19 are the same as the operations in FIG.

The operation of the display device 2 displays the image of each frame at the scanning timing shown in FIGS. 4C to 4F in accordance with the scanning method of the display element 26. Since these display timings are the same as those in FIG. 3, the delay time from the imaging timing (FIG. 4A) will be described.

In the single scan display method of FIG. 4C, a delay 328 corresponding to one frame at the upper end of the screen, a delay of about 0.5 frame at the center of the screen, and a delay close to 0 frame at the lower end of the screen (for compression / decompression of even frames). Such a time 329) occurs.

In the dual scan display method (same direction scanning) in FIG. 4D, there is a delay of almost 0 frames in the entire screen. On the upper screen, there is only a delay of time 349, which is the time required for compression / decompression of even frames, and on the lower screen, the delay time 338 of the upper screen plus the time equivalent to the blanking period of 120 Hz frame. Therefore, it is more advantageous than the single scan display method (FIG. 4C).

In the dual scan display method (reverse direction, start of screen center) of FIG. 4 (e), the upper end of the upper screen has a delay time 358 of about 2 frames. The lower screen has a delay 369 of almost 0 frames, which is slightly earlier than that shown in FIG.

In the dual scan display method of FIG. 4 (f) (reverse direction, start of screen up / down), there is a delay of about 2 frames 378 at the top of the screen, no blur at the bottom of the screen, and a delay of about 2 frames at the center.

FIG. 5 is a table summarizing the delay times in each scanning method described in FIGS. 3 and 4. The delay time from the imaging timing to the display timing is shown as an approximate number in frame units (period T) for the combination of the scanning method of the imaging element and the display element. The numerical values in the upper, middle, and lower stages of each case indicate the delay times at the top, center, and bottom of the screen, respectively. In the middle stage, two numerical values are shown, the first numerical value is the delay time at the lower end of the upper screen, and the second numerical value is the delay time at the upper end of the lower screen.

In this table, scanning methods that have not been explained so far are added for comparison. First, the scanning method (g) of the display element is similar to (d) in the dual scanning of the same direction scanning, but is a method of scanning the upper screen and the lower screen with the same frame video signal. The delay time in the method (g) is the same as (d) on the lower screen, but one frame is added to (d) on the upper screen.

Furthermore, the scanning mode of the image sensor is divided by shutter mode. What has been described so far is the “line shutter” mode, which shifts the shutter timing in the screen in accordance with the scanning. On the other hand, the added “frame shutter” mode captures the entire screen at the same timing and sequentially scans readout. The delay time in the frame shutter mode is longer than that in the line shutter mode. Assuming that the shutter is released immediately before the start of scanning, for example, the delay time is increased by 0 frame at the upper end of the screen, 0.5 frame at the center, and 1 frame at the lower end, compared to the case of single scanning with a line shutter. . In the case of the frame shutter mode, there is no difference in delay time between the image pickup device of the single scan method and the dual scan method.

From the results of FIG. 5, it can be said that the following combinations are desirable for improving the delay time.
First, a desirable scanning method of the display element with respect to the scanning method of the image sensor is as follows.
(A1) When the image pickup device is in the line shutter mode, the display delay time is minimized when the dual scan method ((d): upper and lower screens are scanned in the same direction) is selected as the display device. From the viewpoint of minimizing the delay time variation in the screen, if the image sensor is a single scan system, the display element may be switched to the single scan system.
(A2) When the image sensor is in the frame shutter mode, the delay time variation in the screen can be reduced by selecting the dual scan method ((e): start of the screen center) or the single scan method for the display element.

For the above control, the display device may extract the metadata attached to the video signal from the imaging device from the info frame or the like, detect the scanning method of the imaging device, and select the scanning method of the display device.

Next, a desirable scanning method of the image sensor with respect to the scanning method of the display element is as follows.
(B1) When the display element is a single scan system, if the single scan system (line shutter mode) is selected as the image sensor, the delay time variation in the display screen is minimized.
(B2) When the display element is a dual scan system, the display delay time is minimized when the dual scan system (line shutter mode) is selected for the image sensor.
(B3) When the display element is a dual scan system, if the dual scan system (line shutter mode) is selected as the image sensor, the delay time variation in the display screen is minimized.

For the above control, the imaging device may read the display capability information of the display device from EDID or the like, detect the scanning method of the display element, and select the scanning method of the imaging device.

In order to suppress variations in delay time, the scanning processing unit 25 may create a video at an intermediate timing from the preceding and following video signals and motion vectors, and reversely correct the delay time in the screen obtained from FIG. In addition, when the display element uses a backlight and the image sensor is in the frame shutter mode, the display timing in the screen can be adjusted by lighting the backlight in accordance with the opening time of the frame shutter at the same timing after scanning is completed. It can also be combined.

As described above, the display device transmits display scanning information (single / dual scan, scanning direction, backlight lighting timing) to the imaging device, or conversely, the imaging device captures imaging scanning information (single / dual scanning, scanning direction, (Line / frame shutter, opening time) is added to the video data and transmitted to the display device, so that each scanning method is selected to be optimum. Accordingly, it is possible to realize an imaging / display system with a short delay time from the imaging timing to the display timing or with little variation in the delay time.

In the second embodiment, an imaging / display system assuming a broadcasting system will be described.
FIG. 6 is a diagram illustrating the overall configuration of the imaging / display system according to the second embodiment. The system includes an imaging device (camera) 3, an encoder 4, a server 5, a broadcasting station 6, a set top box (STB) 7, and a display device (display) 2. In this system, the imaging device 3 corresponds to the imaging device 11 in the first embodiment (FIG. 1), and the encoder 4 and the STB 7 perform compression / decompression processing of a video signal according to the broadcast standard. Hereinafter, the configuration and basic processing of each device will be described.

The video signal picked up by the imaging device 3 at a frame rate of 120 Hz, for example, the encoder 4 H.264 and H.264. The data is compressed and encoded based on the H.265 standard and stored in the server 5. An editing station (not shown) edits the stored video for broadcasting using a video editing device or the like, and stores it again in the server 5. The broadcast station 6 transmits the broadcast video signal stored in the server 5 to the STB 7 in the home through a broadcast radio wave, a cable TV line, an Internet line, or the like. Of course, the video signals of the imaging device 3 and the encoder 4 may be transmitted to the STB 7 as they are as a live program. The STB 7 extracts a video signal that the user views from a plurality of received video signals and transmits the video signal to the display device 2, and the display device 2 displays the received video.

It is desirable that a fast-moving video such as a sports program is transmitted with a high frame rate video signal of 120 Hz, for example, and the display device also displays at a high frame rate of 120 Hz. However, since there are popular STBs and display devices that do not support high frame rates, it is desirable that the broadcast signal be compatible with both 60 Hz and 120 Hz display devices. In order to realize this, a method of transmitting a 60 Hz video signal and interpolation data for increasing the frame rate is effective.

As a specific transmission method, H.264 H.264 and H.264. Using the MVC format (Multiview VideodingCoding) defined in H.265, the 60Hz video signal for the base view (Base View) and the interpolation data for 120Hz conversion to the non-base view (non-Base View) (equivalent to 60Hz video signal) ) Should be assigned. The STB 7 and display device 2 compatible with 60 Hz display only the base view. The STB 7 and the display device 2 compatible with 120 Hz also decode and display the non-base view. Thereby, it is possible to support both 60 Hz and 120 Hz display methods with one broadcast signal. By using the above MVC format for increasing the frame rate, it is possible to restore the 120 Hz video signal more faithfully than when using the motion determination result (motion vector).

When I frames (Intra-coded Frames), P frames (Predicted Frames), and B frames (Bi-directional Predicted Frames) are allocated in GOP (Group Of Pictures) units, the base view (60 Hz video signal) is non-base The view (interpolated data for 120 Hz conversion) is preferably BBBB. The base view has the effect of reducing the amount of data by reducing the B frame to shorten the delay time associated with compression / decompression and making all non-base views B frames.

Next, the format of the video signal transmitted and received between the devices shown in FIG. 6 will be described.
The imaging device 3 outputs a 120 Hz baseband digital video signal to the encoder 4 by SDI (Serial Digital Interface). The encoder 4 generates a base view of a 60 Hz video signal and a non-base view of 120 Hz conversion data in the MVC format, and outputs them to the server 5 through a packet format network, for example, Ethernet (registered trademark).

The server 5 outputs a video signal in MVC format to the broadcast station 6 via Ethernet. By performing input / output of the server 5 via Ethernet, the broadcast station 6 can easily receive the live program output from the encoder 4 through the server 5. The broadcasting station 6 transmits a video signal in the MVC format to the STB 7 via a broadcast radio wave, a cable television network, the Internet, or the like.

When the STB 7 is connected to the display device 2 via HDMI, the display capability of the display device 2 can be detected by a mechanism such as EDID (Extended Display Identification Data) or CEC (Consumer Electronics Control). If the display device 2 supports only 60 Hz non-compression, a 60 Hz uncompressed video signal obtained by decoding only the base view is transmitted by, for example, HDMI. If the display device 2 supports only 120 Hz non-compression, the display device 2 transmits a 120-Hz uncompressed video signal obtained by decoding both the base view and the non-base view.

Also, if the display device 2 is compatible with 60 Hz uncompressed video signal with 120 Hz interpolation data, the 60 Hz uncompressed video signal obtained by decoding the base view is converted into an InfoFrame (120 Hz interpolation data of the non-base view as metadata. It is added to Info Frame) and transmitted. Using this interpolation data, the display device 2 can display a frame rate of 120 Hz picked up by the image pickup device 3 at a higher frame rate, such as 240 Hz, which is higher than this, thereby enhancing motion performance.

Even if HDCP (High-bandwidth Digital Content Protection) for content protection is applied to HDMI, since the info frame is not encrypted, the interpolation data may be encrypted and transmitted by the info frame. The decryption key may be a key obtained from HDCP authentication, or may be simplified by using a part of uncompressed video data restored from HDCP as a key.

Next, the encoder 4 and the STB 7 will be described in detail.
FIG. 7 is a diagram showing an internal configuration of the encoder 4. The encoder 4 includes an input unit 41,

memories

42 and 46,

compression units

43 and 47, a packet processing unit 44, an output unit 45, and a control unit 48.

The input unit 41 receives an uncompressed video signal with a frame rate of 120 Hz output from, for example, SDI from the imaging device 3 in FIG. Odd frames are stored in the memory 42 and even frames are stored in the memory 46. The compression unit 43 compresses the odd-numbered frame video signal stored in the memory 42.

On the other hand, the compression unit 47 compresses the even frame video signal stored in the memory 46 with reference to the odd frame video and the compressed video signal. By referring to the odd frame video, compression can be performed more efficiently than the odd frame video. Thus, as described in the first embodiment, difference information, the mixing ratio of the previous and subsequent frames, the motion vector, and the like are generated as the interpolation data for 120 Hz conversion.

The packet processing unit 44 packetizes the odd frames compressed by the compression unit 43 and the even frames compressed more efficiently by the compression unit 47, and the output unit 45 converts the packetized video data into, for example, a LAN (Local (Area Network) or the like to the server 5 in FIG. The control unit 48 controls each unit in the encoder 4 in accordance with the compression method designated by the server 5 or the operator, and at the same time, the imaging device information (single / dual scan, scanning direction, line / frame shutter, opening time) and compression. System information is assigned as metadata of video data. Further, the control unit 48 transmits the display scanning information (single / dual scan, scanning direction, backlight lighting timing) of the display device 2 designated by the server 5 or the operator to the imaging device 3.

As described above, the encoder 4 generates optimal compressed video data based on the imaging scanning information of the imaging device 3 and the display scanning information of the display device 2, and gives the imaging scanning information to the display device 2 side. Transmit to. As a result, as described in the first embodiment, it is possible to realize a broadcasting system in which the delay time from the imaging timing to the display timing is short or the delay time variation is small.

FIG. 8 is a diagram showing the internal configuration of the STB 7. The STB 7 includes an input unit 71, a packet processing unit 72, a separation unit 73, a decompression unit 74, a superposition unit 75, an 8B / 10B encoder 76, an output unit 77, a memory 78, an error correction coding unit 79, and a control unit 80.

The input unit 71 receives a compressed video signal (60 Hz video signal and 120 Hz interpolated data) in the MVC format from the broadcasting station 6 of FIG. 6 via a broadcast radio wave or a network, and the packet processing unit 72 receives the compressed video signal. The packetized video signal is depacketized. The separation unit 73 separates the 60 Hz video signal (odd number frame) and the 120 Hz conversion data (even number frame), and sends them to the decompression unit 74 and the memory 78, respectively. The decompression unit 74 decodes the odd frame video signal to obtain a 60 Hz uncompressed video signal.

On the other hand, the error correction encoding unit 79 performs error correction encoding on the interpolation data for 120 Hz conversion of even frames stored in the memory 78. The superimposing unit 75 superimposes the interpolation data for 120 Hz conversion of the even frame in the blanking period of the 60 Hz uncompressed video signal of the odd frame. The 8B / 10B encoder 76 serializes the 60 Hz uncompressed video signal with 120 Hz interpolated data (converts 8-bit data into 10-bit data), and outputs the video signal from the output unit 77 to the display device 2 as an HDMI format video signal, for example. Output.

The control unit 80 controls each unit, reads display scanning information (single / dual scan, scanning direction, backlight lighting timing) of the display device 2, and generates an optimal uncompressed video signal with interpolation data. Also, imaging device information (single / dual scan, scanning direction, line / frame shutter, opening time) added to the received video signal is added as metadata of the 60 Hz uncompressed video signal. Also, a function of extracting or converting data usable by the display device 2 from the interpolation data for 120 Hz and transmitting it to the display device, and a function of securing a voice data transmission band that requires simultaneous transmission during the retrace period You may have.

As described above, the STB 7 generates an optimal uncompressed video signal based on the imaging scanning information of the imaging device 3 and the display scanning information of the display device 2, and generates high frame rate interpolation data and imaging scanning information on this. And output to the display device 2. As a result, as described in the first embodiment, it is possible to realize a broadcasting system in which the delay time from the imaging timing to the display timing is short or the delay time variation is small.

In the third embodiment, a case of 3D video transmission by the frame packing method will be described. The frame packing method is a method of alternately transmitting 3D left-eye video (L video) and right-eye video (R video). When applied to the 120 Hz video transmission system of the first embodiment, for example, it is conceivable to assign odd frames to L video and even frames to R video. In this case, if the R video is always assigned to the even frame, the even frame is restored from the odd frame and the interpolation data as described above, so that the image quality is always lower than that of the L video assigned to the odd frame. there is a possibility. Further, since the uncompressed video signal in 60 Hz transmission is only L video, the display device without the interpolation data processing function performs only 2D display.

In the present embodiment, a case will be described in which a frame packing method including both L video and R video alternately even in uncompressed video transmission at 60 Hz is described. According to this embodiment, since an uncompressed video signal is also assigned to the R video, a display device that can process the interpolation data has an advantage that the image quality of the L video and the R video is uniform. In addition, a 3D image can be displayed even on a display device without an interpolation data processing function.

The imaging / display system has the same configuration as that of the first embodiment (FIG. 1). Time axis conversion in transmission and display is realized by the

memories

12, 16, 24, 30 and the scanning processing unit 25.

FIG. 9 is a diagram illustrating an example of imaging, transmission, and display timing of a 3D video signal. Here, it is assumed that the imaging scanning method performs a single scan at a frame rate of 120 Hz. A thick arrow (eg, reference numeral 401) indicates the scanning timing of the L video (corresponding to odd frames), and a thin arrow (eg, reference numeral 402) indicates the scanning timing of the R video (corresponding to even frames). Further, a solid line arrow (for example, reference numeral 411) indicates an uncompressed video signal, and a broken line arrow (for example, reference numeral 413) indicates interpolation data.

(A) is an imaging timing, the image sensor 11 performs a single scan at a frame rate of 120 Hz, and

L images

401, 403, and 405 and

R images

402, 404, and 406 are alternately output.

(B) is the transmission timing, and the

L images

401 and 405 and the R image 402 output from the image sensor 11 are alternately scanned and converted into 60 Hz single scan signals 411, 412 and 415 (time width 2T). Further, the L video 403 and the R video 404 sandwiched between them are converted into 120 Hz interpolated

data

413 and 414 and are superimposed on the 60 Hz single scan signals 411, 412 and 415 and output.

(C) (g) (e) (f) are display timings in various display scanning methods. (C) is a case of 120 Hz single scan, and the display element 26 alternately scans the

L images

421 and 423 and the

R images

422 and 424. In synchronization with the scanning of the L video and the R video, the shutter glasses are opened and closed synchronously, so that the viewer's left eye captures the L video and the right eye captures the R video, thereby reproducing the 3D video.

(G) (e) (f) is the case of 120 Hz dual scan, (g) is the same direction scan, (e) is the reverse direction scan (center start), and (f) is the reverse direction scan (up and down start). . In (g), the same frame image is displayed on the upper screen and the lower screen. For example, in (g), the upper screen of the display device 26 alternately scans the

L video

431, 433 and the

R video

432, 434, and the lower screen displays the

corresponding L video

441, 443 and

R video

442, 444 of the same frame. Scanning alternately. The same applies to the other scanning methods (e) and (f). If shutter glasses are used in synchronism with this, 3D video can be reproduced in the same manner as (c).

The transmission and display timing will be described in detail below. The delay time will be described in comparison with the case of the first embodiment (FIG. 3).

At the transmission timing of FIG. 9B, the

L image

401, 403, 405 and the

R image

402, 403, 406 captured by the image sensor 11 with a single scan of 120 Hz are converted to 60 Hz, and the L image and the R image are alternately displayed. Transmit side by side. Accordingly, there is an advantage that 3D video display with frame packing can be realized even in a 3D display device that does not perform interpolation data processing. For this reason, the captured L video 401 is transmitted as an immediate L video 411, and the R video 402 is started to be transmitted after being delayed by one frame (1T). The next L video 403 and R video 404 are compressed and transmitted as L video interpolation data 413 and R video 414, respectively. The transmission start of the L video interpolation data 413 is delayed by the compression or encoding processing time 418 compared to the start of the imaging of the L video 403. If the transmission of the R video interpolation data 414 starts immediately after the transmission of the L video interpolation data 413, the transmission periods of the two interpolation data do not overlap, and the transmission band can be used effectively. The transmission start of the R video interpolation data 414 may be delayed by one frame (2T) of 60 Hz from the transmission start of the L video interpolation data 413.

In the display timing of FIG. 9C, the display element 26 alternately displays the L video and the R video in a single scan. At the start of display of the L video 421,

delays

428 and 429 of approximately 2 frames (2T) are generated compared to the start of imaging of the L video 401. This is because the display end time of the next R video 422 is matched with the transmission end time of the R video 412, and the display interval between the L video 421 and the R video 422 is set to 1T. As a result, the display is delayed by about 1 frame (1T) as a whole as compared with the first embodiment (FIG. 3C).

FIG. 9 (g) shows a case where L video and R video are alternately displayed by the dual scan method in the same direction. In this case, it is necessary to synchronize the shutter glasses. That is, when scanning over two frames as shown in FIG. 3D is performed, the L video and the R video are displayed on the same screen and are difficult to separate with the shutter glasses. Therefore, in the same direction dual scan, the same frame is scanned on both the upper and lower screens. The delay time is almost the same as in the case of FIG. However, since the scanning timing at the center of the screen is time-shifted between the upper screen and the lower screen, the moving body displayed at the center of the screen may appear distorted as described with reference to FIG.

FIG. 9 (e) shows a dual scan method in which scanning is performed from the center to the upper and lower ends. In the lower screen, the scanning is the same as in FIG. 9G, but in the upper screen, the scanning direction is opposite to that in FIG. 9G. As a result, the delay time from the imaging timing has a delay 458 of about 3 frames at the upper end of the screen, a delay 469 of about 2 frames at the lower end of the screen, and a delay of about 1.5 frames at the center. Note that since the scanning timing in the center of the screen is the same, the moving body is not distorted.

FIG. 9F shows a dual scan method in which scanning is performed from the upper and lower ends to the center. Since scanning is performed from the upper and lower ends of the screen, video data at the upper and lower ends of the screen is required at the start of scanning. In particular, the display start of the R video 482 on the lower screen is after 417 at the end of transmission of the R video 412, and other displays are also adjusted to this. As a result, the delay 478 at the upper end of the screen is about 3 frames, the delay at the center of the screen is about 3.5 frames, and the delay 489 at the lower end of the screen is about 2 frames.

FIG. 10 is a diagram illustrating another example of imaging, transmission, and display timing of a 3D video signal. Here, it is assumed that the imaging scanning method performs dual scanning at a frame rate of 120 Hz.

(A) is an imaging timing, the image sensor 11 is dual-scanning at a frame rate of 120 Hz, and

L images

501, 503, 505 and

R images

502, 504, 506 are alternately output (time width 2T). . In this case, two image sensors may be arranged for the L video and the R video, and output with a phase difference of one frame of 120H.

(B) is the transmission timing, and the

L images

501 and 505 and the R image 502 output from the image sensor 11 are alternately scanned to obtain 60 Hz single scan signals 511, 512 and 515. Also, the L video 503 and the R video 504 sandwiched between them are converted into 120 Hz interpolated

data

513 and 514 and superimposed on the 60 Hz single scan signals 511, 512 and 515 for output. As a result, the transmission timing is the same as that in FIG.

(C) (g) (e) (f) are display timings in various display scanning methods. (C) is a case of 120 Hz single scan, and (g), (e), and (f) are cases of 120 Hz dual scan. Since these display timings are the same as those in FIGS. 9C, 9G, 9E, and 9F, the description thereof is omitted.

FIG. 11 is a table summarizing the delay times in each of the 3D display scanning methods described in FIGS. 9 and 10. This is compared with the display in Example 1 (2D display). What can be said from this is that the delay time is larger in the 3D display (FIG. 11) than in the case of the 2D display (FIG. 5). This is the case of the dual scan method (e) in which scanning is performed from the center of the screen to the upper and lower ends. In the 3D display using the shutter glasses method, it is difficult to adopt the dual scan display method (d) that scans in the same direction over two frames.

In the shutter glasses type 3D display device, in order to secure the opening / closing switching time of the shutter glasses, the backlight may blink in accordance with the opening / closing of the shutter glasses. In the case of using a display element that blinks on the entire screen at the same time, when combined with an image pickup element of a frame shutter, there is no distribution of delay time in the screen, and uniform 3D display becomes easy.

As described above, the 3D display device transmits display scanning information (single / dual scan, scanning direction, backlight lighting timing) to the 3D imaging device, or conversely, the 3D imaging device captures imaging scanning information (single / dual scan, scanning). (Direction, line / frame shutter, opening time) are given to the video data and transmitted to the 3D display device, so that each scanning method is selected to be optimum. Accordingly, it is possible to realize an imaging / display system with a short delay time from the imaging timing to the display timing or with little delay time variation.

Example 4 describes a configuration in which image quality enhancement processing is added to a display device, and the scanning rate is doubled while the frame rate of the display element remains 120 Hz.

FIG. 12 is a diagram illustrating a configuration example of the display device according to the fourth embodiment. The display device 8 includes an input unit 81, an image quality improvement processing unit 82, an interpolation data restoration unit 83, a simple image quality improvement processing unit 84, a scanning processing unit 85, and a display element 86. Note that a description of a control unit that transmits the display scanning information of the display element 86 to the imaging device side, detects the imaging scanning information, and controls the entire display device 8 is omitted.

The operation of the display device 8 will be described. The input unit 81 receives a 60 Hz uncompressed video signal and 120 Hz interpolation data. The input 60 Hz uncompressed video signal is sent to the image quality enhancement processing unit 82, and the video is analyzed to improve the brightness, hue, sense of resolution, and the like. On the other hand, the interpolation data restoration unit 83 restores a 120 Hz interpolated video signal from the input 120 Hz conversion data and the 60 Hz uncompressed video signal. Since the restored 120 Hz interpolated video signal does not include the input 60 Hz uncompressed video signal portion, it is a video signal equivalent to 60 Hz. That is, if the 60 Hz video signal input to the input unit 81 and the restored 120 Hz interpolated video signal are combined as they are, a 120 Hz dual scan video signal is obtained.

The 120 Hz interpolated video signal output from the interpolation data restoration unit 83 is subjected to image quality improvement processing by the simple image quality improvement processing unit 84. Since the 120 Hz interpolated video signal has a high correlation with the 60 Hz video signal, there is an advantage that the image quality improvement processing can be easily realized by using the result of the image analysis performed by the image quality improvement processing unit 82. The 60 Hz video signal output from the image quality improvement processing unit 82 and the 120 Hz interpolated video signal output from the simple image quality improvement processing unit 84 are input to the scanning processing unit 85 to match the imaging scanning information and the scanning method of the display element. The video signal is generated, and the display device 86 displays the video with high image quality.

On the other hand, when the display element 86 is a liquid crystal display element that blinks the backlight, the moving image response can be improved by reducing the scanning time to, for example, ½, and providing the time for turning on the backlight after the scanning is completed. . The video signal whose scanning time is shortened can be generated by the time axis compression processing (doubling the scanning speed) of the scanning processor 85. The backlight lighting time corresponds to the opening time of the image sensor, and it is preferable to set the backlight lighting time to be equivalent in order to make use of the photographer's intention. On the other hand, since the photographer can take an image while estimating the degree of blurring of the display element using the information on the frame rate of the display element and the backlight lighting time, it is easy to reflect the intention of the photographer. However, in video transmission formats where the display element method cannot be specified, such as video broadcasting, the imaging scanning information is transmitted to the display device side, and the display scanning method and backlight lighting time are set based on the photographer's intention on the display device side. It is good to choose.

FIG. 13 is a diagram showing the display timing when the scanning time of the display element is shortened. This will be described in comparison with Example 1 (FIG. 3).

(A) Imaging timing (120 Hz single scan) and (b) transmission timing are the same as those in FIG. (C), (g), (e), and (f) are display timings in various display scanning methods.

(C) is a case of single scan display, and scanning is started with a delay of half of the effective display period, that is, a period 829 of about 0.5 frame (0.5 T) from the start of scanning of each frame in FIG. The scanning end timing is matched with the scanning end timing in FIG. As a result, the scanning time is reduced to half (0.5 T). In this manner, it is preferable to secure a long period 829 from the end of scanning to the start of the next scanning, and turn on the backlight during that period. In the shutter glasses 3D display, by performing an open / close switching operation while the backlight is turned off, a good separation display of the L video and the R video can be realized, and a good stereoscopic display can be realized.

(G) is a case of dual scan display (same direction scanning), and the upper and lower screens are scanned in the latter half of the scanning period. In this case, as shown in FIG. 3D, if the upper screen and the lower screen are video signals of different frames, a time lag of about 0.5 frames occurs above and below the center of the screen. Therefore, it is desirable to use the upper screen and the lower screen as the video signals of the

same frames

831 and 841, and suppress the delay time variation within the screen to within 0.5 frames.

(E) is a case of dual scan display (reverse scanning, center start), and the upper and lower screens are scanned in the latter half of the scanning period of FIG. This is because the display end point of the double speed second frame 862 needs to come after the transmission end point 217 of the corresponding interpolation data 212.

(F) is a case of dual scan display (reverse scanning, upper and lower end start), and the upper and lower screens are scanned in the first half of the scanning period of FIG. 5 (f). This is also because the scanning start time of the double speed second frame 882 needs to come after the transmission end time 217 of the corresponding interpolation data 212. Compared with FIG. 3F, the delay time at the center of the screen becomes shorter.

In this embodiment, in the 3D video transmission, even when the L video is fixed to the 60 Hz video signal and the R video is fixed to the interpolation data for 120 Hz conversion, there is a period during which scanning is paused, so that shutter glasses 3D display is possible. It becomes. Of course, even in the 3D video transmission in which the L video and the R video are switched as described in the third embodiment (FIGS. 9 and 10), there is an effect that the opening / closing switching period of the shutter glasses can be secured by shortening the scanning time.

In the fifth embodiment, a configuration in which the frame rate of the display element is doubled to 240 Hz will be described.
FIG. 14 is a diagram illustrating a configuration example of the display device according to the fifth embodiment. The display device 9 includes an input unit 91, an image quality improvement processing unit 92, an interpolation data image quality improvement unit 93, a frame rate conversion unit 94, a scanning processing unit 95, and a display element 96.

The input unit 91 receives a 60 Hz uncompressed video signal and 120 Hz interpolation data. The input 60 Hz uncompressed video signal is sent to the image quality enhancement processing unit 92, and the video is analyzed to improve brightness, hue, resolution, and the like. On the other hand, the interpolation data image quality improving unit 93 reflects the video analysis result of the image quality improving processing unit 92 in the input 120 Hz conversion data. For example, when the interpolation data is difference information and luminance or color correction is performed in an area corresponding to the difference information, the correction amount is reflected in the difference information. The frame rate conversion unit 94 converts the 60 Hz video signal from the image quality improvement processing unit 92 into a 240-fold video signal of 4 times while referring to the 120-Hz interpolation data.

The 240 Hz video signal subjected to the high image quality processing output from the frame rate conversion unit 94 is sent to the scanning processing unit 95. The scanning processing unit 95 generates a video signal in accordance with the imaging scanning information and the scanning method of the display element, and performs video display at a high frame rate on the display element 96.

When the frame rate of the display element 96 is set to 240 Hz, the moving image characteristics of a hold type display element such as a liquid crystal that continues to be displayed from one scan to the next can be improved. However, if high-quality processing with a large amount of processing is performed after frame rate conversion, the circuit scale increases and power consumption increases. In order to avoid this, it is desirable to perform the image quality improvement processing on the 60 Hz video signal and perform the frame rate conversion after the image quality improvement of the 120 Hz conversion data.

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, although the case where the frame rate of the image sensor is set to 120 Hz has been described, it is needless to say that the present invention can be similarly applied to cases where the frame rate of the image sensor is other than that (such as 100 Hz and 96 Hz). The frame rate of the image sensor may be 240 Hz, the uncompressed video signal to be transmitted may be restored to a frame rate of 60 Hz once every four frames, and the interpolation data may be restored to 240 Hz at the time of imaging.

In the embodiment described above, the transmission target video is a video generated by imaging, but the transmission target of the present invention is not limited to this. A video that is different from a video generated by the imaging device, such as a video generated by computer graphics, may be a transmission target. In this case, in each of the above-described embodiments, “imaging device” may be replaced with “video output device”, and “imaging scanning information” may be replaced with “scanning information when generating a video”.

Also, in the example of “imaging device”, “imaging scanning information” can be said to be scanning information when generating the captured image. Therefore, “imaging scanning information” is included in “scanning information when generating image”. It can be said that

In the above-described embodiments, the configuration of each part is described in detail in order to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to one having all the configurations described. Further, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, or the configuration of another embodiment can be added to the configuration of a certain embodiment.

1, 3: an imaging device,
2, 8, 9: display device,
4: Encoder,
5: Server,
6: Broadcasting station
7: STB,
11: Image sensor
13, 75: Superimposition part,
14, 76: 8B / 10B encoder,
17, 43, 47, 74: compression section,
18, 79: Error correction encoding unit,
19, 27, 48, 80: control unit,
22: 10B / 8B decoder,
23, 73: separation part,
25, 85, 95: scanning processing unit,
26, 86, 96: display elements,
28: Error correction decoding unit,
29, 74: extension part,
31: Display signal processing unit,
44: Packetization processing unit,
72: Packet restoration processing unit,
82: High image quality processing unit,
83: Interpolation data restoration unit,
84: Simple high image quality processing unit,
93: Interpolation data image quality improvement section,
94: Frame rate conversion unit.

Claims

In video equipment that displays video output from the video output device,
An input unit for receiving, from the video output device, a first video signal, interpolation data for interpolating a frame of the first video signal, and scanning information for generating the video;
A display signal processing unit that generates a second video signal for interpolating a frame of the first video signal using the first video signal received by the input unit and the interpolation data;
A display element for displaying the first video signal and the second video signal generated by the display signal processing unit;
The display signal processing unit and a control unit for controlling the display element,
The control unit determines a display scanning method of the display element based on the scanning information received by the input unit,
The video apparatus, wherein the display signal processing unit generates the second video signal based on a display scanning method of the display element determined by the control unit.
The video device according to claim 1,
The scanning information includes at least one of the number of simultaneous scans, a scanning direction, and a line / frame shutter determination in an image sensor that acquires the video,
The display apparatus includes at least one of the number of simultaneous scans in the display element, a scanning direction, and a backlight lighting timing.
In video equipment that outputs video to a display device,
A first video signal output from the video source, a second video signal having a frame rate lower than the frame rate of the first video signal, and interpolation data for interpolating a frame of the second video signal; A video signal processing unit for generating
An output unit for transmitting the second video signal and the interpolation data to the display device;
A control unit for controlling the video signal processing unit,
The control unit acquires display scanning information of the display device, information on the second video signal that can be processed, and the interpolation data, and acquires the display scanning information, the second video signal that can be processed, and the interpolation. Determining the format of the second video signal output from the video signal processing unit and the interpolation data based on the data-related information;
The video apparatus, wherein the video signal processing unit generates the second video signal and the interpolation data based on the format determined by the control unit.
The video equipment according to claim 3,
An image sensor is provided as the video source,
The control unit determines an imaging scanning method of the imaging element based on the display scanning information, the second video signal that can be processed, and information on the interpolation data,
The imaging element scans with the imaging scanning method determined by the control unit,
The display scan information includes at least one of the number of simultaneous scans in the display element, the scan direction, and the lighting timing of the backlight,
The imaging apparatus includes at least one of the number of simultaneous scans, a scanning direction, and line / frame shutter discrimination in the imaging device.
The video equipment according to claim 3,
When the first video signal output from the video source is a 3D video signal including a left-eye video and a right-eye video,
The video apparatus, wherein the second video signal generated by the video signal processing unit is a frame packing method that alternately includes the left-eye video and the right-eye video.
The video device according to claim 1,
The display signal processing unit
An image quality improvement processing unit that performs an image quality improvement process on the first video signal received by the input unit;
An interpolation data restoration unit for generating the second video signal from the first video signal and the interpolation data received by the input unit;
A video apparatus comprising: a simple image quality improvement processing unit for improving an image quality of a video signal output from the interpolation data restoration unit using an image quality improvement parameter determined by the image quality improvement processing unit.
The video device according to claim 1,
The display signal processing unit
An image quality improvement processing unit that performs an image quality improvement process on the first video signal received by the input unit;
An interpolation data image quality improvement unit that corrects the interpolation data received by the input unit using a processing result of the image quality improvement processing unit;
Using the corrected interpolation data, the second video signal for interpolating a frame of the first video signal is generated, and the first video signal and the second video signal are combined to generate the first video signal. What is claimed is: 1. A video device comprising a frame rate conversion unit for converting into a third video signal having a frame rate twice or more that of a video signal.
The video equipment according to claim 2,
The controller is
When the scanning information indicates a line shutter mode, select a dual scan method for scanning the upper and lower screens in the same direction as the display scanning method of the display element,
When the scanning information indicates a frame shutter mode, a video apparatus characterized by selecting a dual scan method in which screen center scanning is started as the display scanning method of the display element.
The video equipment according to claim 4,
The controller is
When the display scanning information indicates a single scan method, select a single scan method in a line shutter mode as an imaging scan method of the image sensor,
When the display scanning information indicates a dual scan method, a video apparatus, wherein a line scan mode dual scan method is selected as an imaging scan method of the image sensor.
In a video processing method for displaying a video signal acquired by an image sensor on a display element,
Video that generates a second video signal having a lower frame rate than the first video signal and interpolation data for interpolating the frames of the second video signal from the first video signal output by the imaging device A signal generation step;
A transmission / reception step of transmitting / receiving the second video signal, the interpolation data, and imaging scanning information of the imaging device;
A display signal processing step of generating a third video signal for interpolating a frame of the second video signal using the received second video signal and the interpolation data;
A display step of displaying the second video signal and the third video signal on the display element;
The display scanning method of the display element is determined based on the received imaging scanning information,
Alternatively, the imaging scanning method of the imaging element is determined based on display scanning information of the display element.