WO2019003609A1

WO2019003609A1 - Image processing device and image processing method

Info

Publication number: WO2019003609A1
Application number: PCT/JP2018/016478
Authority: WO
Inventors: 啓文葛西; 一彰鳥羽; 市村　元; 山本　和夫
Original assignee: ソニー株式会社
Priority date: 2017-06-26
Filing date: 2018-04-23
Publication date: 2019-01-03

Abstract

Provided are an image processing device and an image processing method for processing omnidirectional image data and performing a process related to the transmission of a baseband signal of image data. The image processing device is provided with: a defining unit which defines latitude lines and longitude lines on a globe on which the omnidirectional image is represented; and an arrangement unit which arranges pixels of the omnidirectional image on each of the defined latitude lines. Moreover, the image processing device is further provided with: a processing unit which sequentially arranges each of the pixels arranged on each of the latitude lines of the spherical surface on a single line, and generates baseband signals having a different number of pixels for each of the lines; and a transmission unit which transmits the baseband signals to an external device via a predetermined transmission channel. The image processing device can operate as a source device.

Description

Image processing apparatus and image processing method

The technology disclosed in the present specification relates to an image processing apparatus and an image processing method for processing omnidirectional image data or performing processing related to transmission of a baseband signal of image data.

With the spread of wide-angle cameras and omnidirectional cameras, the content business dealing with omnidirectional images is also expanding. The omnidirectional image can be viewed using a display device such as a large screen display or a head mounted display, for example.

An omnidirectional image can also be defined as an image having information of 360 degrees in all directions vertically and horizontally around the viewpoint when the viewpoint is a certain point in space. The omnidirectional image is spherical and has a three-dimensional spread. For this reason, as image data, a spherical surface may be mapped and handled on a two-dimensional rectangle (for example, refer to patent documents 1).

JP, 2014-127001, A JP, 2013-229885, A

An object of the technology disclosed in the present specification is to provide an image processing apparatus and an image processing method for processing omnidirectional image data or performing processing related to transmission of a baseband signal of image data.

The first aspect of the technology disclosed herein is:
A definition unit that defines latitude lines and longitude lines on a sphere that represents an omnidirectional image;
An arrangement unit for arranging the pixels of the omnidirectional image on each of the defined latitude lines;
An image processing apparatus comprising

The definition unit determines the interval of the latitude line based on either an angle of the spherical surface from the equatorial plane or a height of the spherical surface in the ground axis direction. In addition, the placement unit places pixels on each latitude line at substantially equal intervals starting from the location of zero longitude.

The image processing apparatus sequentially arranges pixels on each of the latitudinal lines of the spherical surface sequentially in one line, and generates a baseband signal having a different number of pixels for each line, and a predetermined transmission path for the baseband signal. You may further provide the transmission part which transmits to an external device via it.

In this case, the image processing apparatus transmits the omnidirectional image data based on the acquisition unit that acquires transmission method information on the transmission method of the omnidirectional image data that can be handled by the external device, and the acquired transmission method information. The information processing apparatus may further include a selection unit that selects a scheme, and the processing unit may generate the baseband signal according to the selected transmission scheme. The image processing apparatus may further include a notification unit that notifies the external device of format information related to the format of the omnidirectional image data.

Alternatively, the image processing apparatus may generate baseband signals of different numbers of pixels for each line generated by sequentially arranging the pixels on each of the latitudinal lines of the spherical surface in one line from an external device through a predetermined transmission path. The image processing apparatus may further include a receiving unit that receives and a processing unit that performs processing for displaying an omnidirectional image based on the baseband signal.

The image processing apparatus in this case may further include a notification unit that notifies the external device of transmission method information on a transmission method of compatible omnidirectional image data. The image processing apparatus further includes an acquisition unit for acquiring format information on the format of the omnidirectional image data from the external device, and the processing unit displays the omnidirectional image based on the format information. Processing may be performed.

Also, a second aspect of the technology disclosed in the present specification is
Defining a latitude line and a longitude line on a sphere representing an omnidirectional image;
Arranging the pixels of the omnidirectional image on each of the defined latitude lines;
Image processing method.

According to the technology disclosed in the present specification, it is possible to provide an image processing apparatus and an image processing method for processing omnidirectional image data or performing processing relating to transmission of a baseband signal of image data.

The effects described in the present specification are merely examples, and the effects of the present invention are not limited thereto. In addition to the effects described above, the present invention may exhibit additional effects.

Other objects, features, and advantages of the technology disclosed herein will be apparent from the more detailed description based on the embodiments described below and the accompanying drawings.

FIG. 1 is a diagram showing a method of arranging pixels on a spherical surface. FIG. 2 is a diagram showing a configuration example of the AV system 200. As shown in FIG. FIG. 3 is a diagram showing a configuration example of the HDMI transmitting unit 218 and the HDMI receiving unit 252 in the AV system 200. FIG. 4 is a diagram showing a configuration example of the omnidirectional image data. FIG. 5 is a diagram showing an example of the configuration of the omnidirectional image data. FIG. 6 is a diagram showing an example of the configuration of the omnidirectional image data. FIG. 7 is a diagram showing an example of TMDS transmission data for transmitting omnidirectional image data represented as a spherical surface. FIG. 8 is a diagram showing an example of a packing format when transmitting image data through three TMDS channels # 0 to # 2 of HDMI. FIG. 9 is a diagram showing an example of the data structure of E-EDID. FIG. 10 is a diagram showing an example of the data structure of the Vender Specific area. FIG. 11 is a diagram showing an example data structure of an AVI InfoFrame packet. FIG. 12 is a flowchart showing the processing procedure performed by the disc player 210 when the display 250 is connected. FIG. 13 is a flowchart showing a processing procedure for the disc player 210 to determine the transmission method of the omnidirectional image data. FIG. 14 is a diagram illustrating how a video signal is transmitted in the transmission format shown in FIG. FIG. 15 is a diagram showing a modification of the transmission format of the omnidirectional image data. FIG. 16 is a view showing a method of mapping an omnidirectional image on a two-dimensional plane using equidistant cylindrical projection. FIG. 17 is a view showing a method of projecting an omnidirectional image on a polyhedron and mapping it on a two-dimensional plane.

Hereinafter, embodiments of the technology disclosed herein will be described in detail with reference to the drawings.

The omnidirectional image is an image having information of 360 degrees in all directions vertically and horizontally around a certain viewpoint in space, and has a three-dimensional spread. On the other hand, most of the image data currently used consists of rectangular information having a predetermined aspect ratio such as 4: 3 or 16: 9. Therefore, techniques for recording and transmitting rectangular image data are generally used. Therefore, the omnidirectional image may be mapped and handled on a two-dimensional rectangle.

For example, using equidistant cylindrical projection, all intersections of latitude and longitude lines on a sphere are defined as pixels and mapped on a two-dimensional rectangle (see FIG. 16), or a cube circumscribed to a sphere There may be mentioned a method of projecting and mapping an omnidirectional image on a polyhedron such as (regular hexahedron) (see FIG. 17) and the like.

In the case of the former mapping method, first, as shown in FIG. Next, as shown in FIG. 16B, the cylinder 1602 is developed into a flat surface as indicated by reference numeral 1603. The image data mapped to the plane 1603 of the two-dimensional coordinates is H.264. It can be compression encoded using a standard moving image data compression encoding scheme such as H.264, and can be transmitted and stored. Also, if the image data developed on the two-dimensional plane 1603 is mapped to a sphere based on the correspondence (index) between the two-dimensional coordinates and the original three-dimensional coordinates, the original omnidirectional image is reproduced. can do.

When the equidistant cylindrical projection is used, the upper and lower high latitude areas become high resolution areas where the number of pixels mapped per unit area of the original spherical surface is large, while the central low latitude areas are original This results in a low resolution area in which the number of pixels mapped per unit area of the spherical surface is small.

Further, in the case of the latter mapping method, first, as shown in FIG. 17A, the omnidirectional image is projected on the inner periphery of the cube 1702 circumscribing the omnidirectional object 1701. Next, the cube 1702 is expanded on a plane as shown in FIG. 17B, and the image data projected on the side surfaces # 1 to # 6 of the cube 1702 is further two-dimensionally shown in FIG. 17C. Map on the plane 1703 of coordinates. The image data mapped to the plane 1703 of the two-dimensional coordinates is as follows. It can be compression encoded using a standard moving image data compression encoding scheme such as H.264, and can be transmitted and stored. Also, if the image data developed on the two-dimensional plane 1703 is mapped to a sphere based on the correspondence (index) between the two-dimensional coordinates and the original three-dimensional coordinates, the original omnidirectional image is reproduced. can do.

On the other hand, a method of representing as a spherical surface without mapping the omnidirectional image to a two-dimensional rectangle or the like may be considered.

In order to represent an omnidirectional image as a spherical surface, it is necessary to arrange pixels on the spherical surface and to have an index that allows reference of the position on the spherical surface of each pixel. Therefore, in this specification, a method of arranging pixels on a spherical surface as shown in FIG. 1 is proposed.

In FIG. 1, first, latitude and longitude lines are defined on the spherical surface. Subsequently, on each latitude line, pixels are arranged at equal intervals starting from the location of zero longitude. Then, a latitude index is allocated to each pixel of each latitude line, and a longitude index starting from zero longitude is sequentially allocated to the pixels arranged on each latitude line. This makes it possible to refer to the positions of all the pixels arranged on the sphere by a combination of the latitude index and the longitude index.

In the pixel arrangement method on the spherical surface shown in FIG. 1, the number of pixels arranged at the poles corresponding to the north pole and the south pole is 1, and the number of pixels arranged on the latitude line corresponding to the equator is maximum.

When the pixel arrangement on the spherical surface shown in FIG. 1 is compared with the existing image data representing method, each latitude line corresponds to a so-called line (scanning line), and the total number of latitude lines or latitude line interval (or longitude) It can be said that the total number of pixels on the line or the pixel arrangement interval) corresponds to the vertical resolution. Also, the total number of pixels or the pixel arrangement interval on each latitude line can be said to correspond to so-called horizontal resolution.

When omnidirectional image data represented as a pixel arrangement on a spherical surface as shown in FIG. 1 is used as it is for transmission, transmission of mapped omnidirectional image data (see, eg, FIGS. 16 and 17) The following merits (1) and (2) can be mentioned in comparison with the above.

(1) The amount of data is small For example, in the equidistant cylindrical projection (see FIG. 16), the number of pixels is larger than that of the spherical representation as the latitude is closer to the high latitude, and the amount of data is also larger. Also, with the method of projecting on a cube (see FIG. 17), although the resolution does not become uneven depending on the latitude, the amount of data becomes large in proportion to the surface area ratio of the sphere. On the other hand, according to the pixel arrangement method on the spherical surface shown in FIG. 1, the pixels are arranged at equal intervals on each latitude line (that is, the pixels are arranged with almost the same density on the spherical surface). Is constant, and the resolution does not become uneven depending on the latitude.

(2) The hardware resources on the receiving device side can be very small. Assuming that the received omnidirectional image is displayed as it is on the receiving side using, for example, a display that can be displayed on a spherical surface, as shown in FIG. And, as shown in FIG. 17, when using a transmission format for mapping the omnidirectional image on a two-dimensional plane, it is necessary to carry out reverse mapping of the received image data on the sphere. On the other hand, according to the pixel arrangement method shown in FIG. 1, since the image data representing the spherical image is transmitted spherically, the receiving side does not need reverse mapping processing or an image using a very small memory. Data processing is sufficient. Thus, hardware resources of the receiver can be saved.

FIG. 2 shows a configuration example of an AV (Audio Visual) system 200 capable of transmitting image data. The AV system 200 includes a disk player 210 as a source device and a display 250 as a sink device. Image data of a spherically-represented omnidirectional image can be transmitted and received between the disk player 210 and the display 250. Thus, the disc player 210 is an all-sky signal reproduction device, and the display 250 is also an all-sky signal display device.

The disk player 210 and the display 250 are connected via, for example, a high-definition multimedia interface (HDMI) cable 350. The disc player 210 is provided with an HDMI terminal 219 to which an HDMI transmitting unit 218 and a high-speed data line interface (I / F) 215 are connected. Further, the display 250 is provided with an HDMI terminal 259 to which the HDMI receiving unit 252 and the high-speed data line interface 251 are connected. Then, one end of the HDMI cable 350 is connected to the HDMI terminal 219 of the disc player 210, and the other end of the HDMI cable 350 is connected to the HDMI terminal 259 of the display 250.

Uncompressed (baseband) image data obtained by reproduction on the disc player 210 side is transmitted to the display 250 via the HDMI cable 350. Then, the display 250 processes the received image data to display an image. In addition, uncompressed (baseband) audio data obtained by reproduction on the disc player 210 side is transmitted to the display 250 via the HDMI cable 350, and the display 250 side processes the received audio data. Audio is output.

When the image data transmitted from the disc player 210 is omnidirectional image data, the display 250 displays the omnidirectional image in order to provide the omnidirectional image to the user.

Here, an example of the display method of the omnidirectional image will be described.

As described above, since the omnidirectional image is an image having a three-dimensional spread, it needs to be displayed on the spherical surface in order to display correctly. For example, in the case of a display that can be correctly displayed on a spherical surface, the received omnidirectional image data can be displayed as it is. Further, for example, in the case of a rectangular display such as a television, an appropriate rectangular portion can be cut out and displayed from the received omnidirectional image.

The configuration of the disc player 210 shown in FIG. 2 will be described focusing on the case of transmitting omnidirectional image data.

The disc player 210 includes an HDMI terminal 219, an HDMI transmitting unit 218, and a high-speed data line interface 215. Also, the disk player 210 includes a central processing unit (CPU) 214, a memory 213, an Ethernet (registered trademark) interface 211, a disk drive 212, a general image processing unit 216, and an omnidirectional image processing unit 217. , And these components are interconnected via

internal buses

220, 221.

The CPU 214 executes control software stored in the memory 213 and centrally controls the operation of each component in the disk player 210 through the

internal buses

220 and 221. Data generated when the CPU 214 executes control software is also stored in the memory 213 as appropriate.

The HDMI transmitting unit 218 transmits baseband data (or video) and audio data from the HDMI terminal 219 by communication conforming to HDMI as an HDMI source. Image and audio data are transmitted on a Transition Minimized Differential Signaling (TMDS) channel of HDMI.

The high-speed data line interface 215 is an interface of a bidirectional communication path configured using a predetermined line (in the present embodiment, a reserve line and an HPD (Hot-Plug-Detect) line) configuring the HDMI cable 350. is there.

The high speed data line interface 215 is disposed between the Ethernet interface 211 and the HDMI terminal 219. The high-speed data line interface 215 transmits transmission data supplied from the CPU 214 from the HDMI terminal 219 via the HDMI cable 350 to the other device (ie, the display 250 as an HDMI sink device). Also, the high-speed data line interface 215 receives data from the other party's device from the HDMI cable 350 via the HDMI terminal 219 and supplies the received data to the CPU 214.

Content data recorded on a disk (not shown) loaded in the disk drive 212 (that is, content data reproduced from the disk by the disk drive 212) is sent to the general image processing unit 216 through the internal bus 220. Be The content data recorded in the disk drive 212 is, for example, a compressed image compressed in the MPEG (Moving Picture Experts Group) format.

The general image processing unit 216 decodes or decompresses the compressed image, and then sends it to the omnidirectional image processing unit 217. When transmitting the omnidirectional image data for displaying the omnidirectional image among the image data obtained from the general image processing unit 216 using the TMDS channel of HDMI, the omnidirectional image processing unit 217 transmits the transmission method. Process to the state according to

In the present embodiment, the omnidirectional image data is expressed as a pixel arrangement on a spherical surface. That is, latitude lines and longitude lines are defined on the spherical surface representing the omnidirectional image, and the pixels forming the omnidirectional image are arranged at predetermined intervals (for example, equally spaced) in the longitudinal direction on each latitudinal line. (See Figure 1). Then, when transmitting the omnidirectional image data expressed as the pixel arrangement on the spherical surface in this manner, the omnidirectional image processing unit 217 sets pixel data on each latitude line of the omnidirectional image data for each line. Generate a placed baseband image. The baseband image for transmitting the omnidirectional image data has a configuration in which the number of pixels is different for each line, but the details will be described later.

The omnidirectional image data processed by the omnidirectional image processing unit 217 is supplied to the HDMI transmission unit 218. The HDMI transmitting unit 218 packs the omnidirectional image data and outputs the packed data from the HDMI terminal 219. The HDMI transmitting unit 218 packs (described later) baseband signals of the omnidirectional image data according to a transmission method such as RGB 4: 4: 4, YCbCr 4: 4: 4, YCbCr 4: 2: 0, or the like.

If the image data is not omnidirectional image data, the processing of the omnidirectional image processing unit 217 is not performed, and the image data is supplied to the HDMI transmission unit 218. On the other hand, if the image data is omnidirectional image data, the omnidirectional image data is processed by the omnidirectional image processing unit 217 into a state according to the selected transmission method, and then the HDMI transmission unit 218 is performed. Supplied to

Also, when the content data reproduced from the disk by the disk drive 212 is sent out to the network instead of the HDMI cable 350, the content data is output to the network terminal 222 through the Ethernet interface 211. . Similarly, when the content data reproduced from the disk by the disk drive 212 is sent not to the TMDS channel of the HDMI cable 350 but to the bidirectional communication path (described above), the content data is transmitted to the Ethernet interface 211. And the high speed data line interface 215 to the HDMI terminal 219.

Subsequently, the configuration of the display 250 shown in FIG. 2 will be described focusing on the case of receiving the omnidirectional image data and displaying and outputting it.

The display 250 includes an HDMI terminal 259, an HDMI receiving unit 252, and a high-speed data line interface 251. In addition, the display 250 includes a CPU 253, an Ethernet interface 255, a display panel 258, a general image processing unit 257, and an omnidirectional image processing unit 256, and these components are mutually connected via the

internal buses

260 and 261. It is connected.

The CPU 253 executes control software stored in the memory 254, and centrally controls the operation of each component in the display 250 through the

internal buses

260 and 261. Data generated when the CPU 253 executes the control software is also stored in the memory 254 as appropriate.

The HDMI receiving unit 252 receives baseband image (or video) and audio data supplied to the HDMI terminal 259 through the HDMI cable 350 by communication conforming to HDMI as an HDMI sink. Image and audio data are transmitted on the TMDS channel of HDMI (same as above).

Similar to the high-speed data line interface 215 (described above) on the disc player 210 side, the high-speed data line interface 251 includes predetermined lines (reserve line and HPD line in this embodiment) which constitute the HDMI cable 350. Is an interface of a two-way communication path configured using

The high speed data line interface 251 is disposed between the Ethernet interface 255 and the HDMI terminal 259. The high-speed data line interface 251 transmits transmission data supplied from the CPU 253 from the HDMI terminal 259 to the other device (that is, the display 250 as an HDMI sink device) via the HDMI cable 350. Also, the high-speed data line interface 251 receives data from the other party's device from the HDMI cable 350 via the HDMI terminal 259, and supplies the received data to the CPU 253.

The HDMI receiving unit 252 receives a baseband image. The HDMI reception unit 252 depacks a baseband signal packed by a transmission method such as RGB 4: 4: 4, YCbCr 4: 4: 4, or YCbCr 4: 2: 0. If the image data received by the HDMI receiving unit 252 is omnidirectional image data, the omnidirectional image processing unit 256 processes the image data into a state suitable for the display panel 258, and then the general image processing unit 257 Sent. If the image data received by the HDMI receiving unit 252 is not omnidirectional image data, the omnidirectional image processing unit 256 does not perform processing, and the received image data is sent to the general image processing unit 257 as it is .

A process in the case of receiving the omnidirectional image data will be described. In the present embodiment, the omnidirectional image data is expressed as a pixel arrangement on a spherical surface. That is, latitude lines and longitude lines are defined on the spherical surface representing the omnidirectional image, and the pixels forming the omnidirectional image are arranged at predetermined intervals (for example, equally spaced) in the longitudinal direction on each latitudinal line. (See Figure 1). The omnidirectional image processing unit 256 maps each pixel arranged in each line of the baseband image on the spherical surface representing the omnidirectional image to reproduce the original omnidirectional image. carry out.

The omnidirectional image processing unit 256 performs processing of the received omnidirectional image data based on the format information of the omnidirectional image data notified from the disc player 210 side. For example, the omnidirectional image processing unit 256 calculates the number of pixels on each line of the baseband image based on the number of pixels on the equator and the total number of lines in the omnidirectional image data acquired as the format information. Also, the omnidirectional image processing unit 256 can perform display position correction when displaying the omnidirectional image, based on the origin position information acquired as the format information.

The general image processing unit 257 performs, for example, image quality improvement processing and superimposing processing of graphic data (such as OSD) on the image data. After such general image processing, the image data is sent to the display panel 258, and the image is presented to the user.

FIG. 3 shows a configuration example of the HDMI transmitting unit (HDMI source) 218 on the disc player 210 side and the HDMI receiving unit (HDMI sink) 252 on the display 250 side in the AV system 200.

The HDMI cable 350 connecting the HDMI transmitting unit 218 and the HDMI receiving unit 252 includes TMDS channels # 0 to # 2 and a TMDS clock channel. Image data and audio data are serially transmitted in one direction in synchronization with the pixel clock from the HDMI transmitting unit 218 to the HDMI receiving unit 252 using the TMDS channels # 0 to # 2. Also, the pixel clock is transmitted using the TMDS clock channel.

The HDMI transmitting unit 218 is an effective image period (hereinafter referred to as “active video period” which is a period obtained by removing the horizontal blanking period and the vertical blanking period from the interval from one vertical synchronization signal (Vsync) to the next vertical synchronization signal. , And transmits the differential signal corresponding to pixel data of a non-compressed (baseband) image for one screen to the HDMI receiving unit 252 in one direction. In addition, the HDMI transmitting unit 218 transmits a differential signal corresponding to at least audio data and control data associated with an image to the HDMI receiving unit 252 in one direction in the horizontal blanking interval or the vertical blanking interval. Send to

That is, the HDMI transmitter 81 included in the HDMI transmitting unit 218 converts pixel data of an uncompressed image into a corresponding differential signal, and the HDMI receiving unit is generated by the three TMDS channels # 0 to # 2 included in the HDMI cable 350. Serially transmit in one direction 252.

Also, the HDMI transmitter 81 converts audio data accompanying uncompressed images, and further necessary control data and other auxiliary data into corresponding differential signals, and the three TMDS channels included in the HDMI cable 350 Serial transmission is performed in one direction to the HDMI reception unit 252 at 0 to # 2.

Further, the HDMI transmitter 81 transmits a pixel clock synchronized with pixel data transmitted on the three TMDS channels # 0 to # 2 to the HDMI receiving unit 252 on the TMDS clock channel included in the HDMI cable 350. For example, in each TMDS channel # 0 to # 2, 10 bits of pixel data are transmitted during one clock of the pixel clock.

The HDMI reception unit 252 receives a differential signal corresponding to pixel data transmitted in one direction from the HDMI transmission unit 218 in a plurality of channels in the active video period. Also, the HDMI receiving unit 252 receives a differential signal corresponding to audio data or control data transmitted in one direction from the HDMI transmitting unit 218 on a plurality of channels in the horizontal blanking interval or the vertical blanking interval. Do.

That is, the HDMI receiver 82 included in the HDMI receiving unit 252 is a differential signal corresponding to pixel data transmitted in one direction from the HDMI transmitting unit 218 through the three TMDS channels # 0 to # 2 included in the HDMI cable 350. And differential signals corresponding to audio data and control data in synchronization with the pixel clock transmitted from the HDMI transmitting unit 218 on the TMDS clock channel.

The HDMI cable 350 also includes transmission channels called Display Data Channel (DDC) 83 and Consumer Electronics Control (CEC) line 84 in addition to TMDS channels # 0 to # 2 and TMDS clock channel.

The DDC 83 includes two signal lines (not shown) included in the HDMI cable 350. The HDMI transmitting unit 218 uses the DDC 83 in order to read E-EDID (Enhanced Extended Display Identification Data) from the HDMI receiving unit 252 connected via the HDMI cable 350.

The HDMI receiving unit 252 has an EDID ROM (Read Only Memory) 85 in addition to the HDMI receiver 82, and stores E-EDID, which is performance information related to its own performance (Configuration or Capability), in the EDID ROM 85. ing. The HDMI transmitting unit 218 can read out the E-EDID of the HDMI receiving unit 252 connected via the HDMI cable 350 from the EDID ROM 85 via the DDC 83. Then, the HDMI transmitting unit 218 sets the performance of the HDMI receiving unit 252 based on the E-EDID. For example, the HDMI transmitting unit 218 recognizes the format (profile) of an image supported by the electronic device (display 250) having the HDMI receiving unit 252, for example, RGB, YCbCr 4: 4: 4, YCbCr 4: 2: 0, etc. Do.

The CEC line 84 is formed of a single signal line (not shown) included in the HDMI cable 350. The CEC line 84 is used to perform bi-directional communication of control data between the HDMI transmitting unit 218 and the HDMI receiving unit 252.

The HDMI cable 350 further includes an HPD line 86 connected to an HPD pin consisting of 19 pins. The HDMI transmitting unit 218 can detect that the HDMI receiving unit 252 is connected via the HDMI cable 350 using the HPD line 86. Also, the HDMI cable 350 includes a line (+5 V power supply line 87) consisting of 18 pins used to supply +5 V power from the source device to the sink device. Further, the HDMI cable 350 includes a 14-pin reserve line 88.

Subsequently, a transmission method of omnidirectional image data will be described.

FIGS. 4 and 5 show an example of the configuration of the omnidirectional image data of the original signal. Define latitude and longitude lines on the sphere. FIG. 4 shows a side view of a spherical surface. Latitude lines are defined every angular interval θ from the equatorial plane of the sphere. FIG. 5A shows a state in which a latitude line is added on the spherical surface. Further, FIG. 5B shows the latitude line viewed from the top. Then, as described with reference to FIG. 1, the omnidirectional image is expressed as a spherical surface by arranging the pixels at equal intervals starting from the location of the zero longitude on each latitude line. In such a pixel arrangement method on a spherical surface, the number of pixels arranged at the poles corresponding to the north pole and the south pole is 1, and the number of pixels arranged on the latitude line corresponding to the equator becomes maximum.

For example, it is assumed that the number of pixels on the latitudinal line corresponding to the equator is 4,000, and the angle between the latitudinal lines is constant at 0.00107471 radians, that is, the total number of latitudinal lines corresponding to the vertical resolution is 3001. The number of pixels arranged for each latitude line (line number) corresponding to the scanning line is summarized in the following Table 1 (however, the display of information of some lines is omitted).

As shown in FIGS. 4 and 5, the latitude lines are not defined at equal angular intervals θ from the equatorial plane of the sphere, but as shown in FIG. It is also possible to define all celestial sphere image data represented as a spherical surface by defining every interval h. However, it is thought that the image quality will be higher in the transmission method in which the latitude lines are defined at equal angular intervals θ from the equatorial plane of the sphere.

FIG. 7 shows an example of TMDS transmission data in the case of transmitting the omnidirectional image data represented as a spherical surface as shown in FIG. 5 and FIG. 6 by the TMDS channel of HDMI. The figure shows sections of various transmission data in the case where image data of 4000 pixels in width x height of 3001 lines is transmitted in the TMDS channels # 0 to # 2.

Depending on the type of transmission data, video data period (Video Data period) and data island period are in the video field (Video Field) or TMDS period in which the transmission data is transmitted by three TMDS channels # 0 to # 2 of HDMI. There are three types of intervals: (Data Island period) and control period (Control period).

A video field interval is an interval from a rising edge (active edge) of one vertical synchronization signal to a rising edge of the next vertical synchronization signal, and a horizontal blanking period (horizontal blanking period), a vertical blanking period (vertical blanking period), and The video field period is divided into an active video period (Active Video) which is a period obtained by removing the horizontal blanking period and the vertical blanking period.

Video data intervals are assigned to active video intervals. This video data section consists of 3001 lines, and it is possible to transmit data of 4000 pixels (active pixels) of each line. That is, in the video data section, it is possible to transmit data of effective pixels of 4000 pixels (pixels) × 3001 lines constituting image data of one non-compressed screen.

The data island period and the control period are assigned to the horizontal blanking period and the vertical blanking period. Auxiliary data (Auxiliary data) is transmitted in the data island period and the control period.

In the data island section, a dark gray area in FIG. 7 is a data island section. It is allocated to a part of the horizontal blanking period and the vertical blanking period. In the data island period, among the auxiliary data, data not related to control, for example, a packet of audio data is transmitted.

The control period is assigned to the horizontal blanking period and other parts of the vertical blanking period. In FIG. 7, an area indicated by hatching is a control section. In the control section, of the auxiliary data, data relating to control, for example, a vertical synchronization signal, a horizontal synchronization signal (Hsync), a control packet and the like are transmitted.

In the configuration example of the omnidirectional image data expressed as a spherical surface shown in FIG. 4 and FIG. 5, the total number of latitude lines corresponding to the vertical resolution is 3001, and the number of pixels of each latitude line varies. That is, the number of pixels disposed at the poles corresponding to the north pole and the south pole is 1, and the number of pixels disposed on the latitude line corresponding to the equator is at most 4000 (see, for example, FIG. 1).

In the case of TMDS transmission data shown in FIG. 7, the pixels for each latitude line of the omnidirectional image data as shown in FIGS. 4 and 5 are sequentially arranged in each line of the active video section consisting of 4000 pixels (pixels) × 3001 lines. By arranging, a video data section consisting of 3001 lines is configured. In this case, the number of effective pixels of each line is different for each line. This is because only a minimum of one pixel is arranged on the scan line corresponding to the latitude lines of the north pole and the south pole, but a maximum of 4000 pixels is arranged on the scan line corresponding to the equator.

The TMDS transmission data shown in FIG. 7 can be said to have a data format in which the number of pixels is different for each line (or the number of pixels is not constant in each line). The gray area on each line of the video data section in FIG. 7 is a video data section. Since the number of pixels is different for each line, the length of the video data section also differs for each line. One line can also be defined as a section separated by a synchronization signal. In each line, when the video data section displayed in gray ends, the next horizontal synchronization signal is inserted and returned to the beginning of the line, and transmission of the signal of the next line is started. Therefore, when the number of pixels is different for each line, the interval of the horizontal synchronization signal is not the same for each line. For reference, FIG. 14 exemplifies how a video signal is transmitted in a section from one vertical synchronization signal to the next vertical synchronization signal in the transmission format shown in FIG.

Note that when transmitting image data in which the number of effective pixels differs for each line, the video data section for each line is not changed, but for the line with few effective pixels, A transmission format may also be considered in which the video data section of each line is equalized by filling in invalid pixels (indicated by 0). FIG. 15 shows an example of a transmission format of omnidirectional image data in which the length of each line of the video data section is made uniform. In the figure, it is assumed that invalid pixels are filled in a portion surrounded by a dashed square. By filling in with invalid pixels, it is possible to convert spherical image data represented as a whole spherical image into rectangular image data, and it is possible to perform compression processing using an existing compression method such as MPEG. However, if such a transmission format is used, the amount of transmission data increases more than the transmission format shown in FIG.

When transmitting the omnidirectional image data represented as the pixel arrangement on the spherical surface as shown in FIGS. 4 and 5 from the disc player 210 to the display 250, the omnidirectional image processing unit 217 The pixel data on each latitudinal line of the omnidirectional image data is arranged to generate a baseband image as shown in FIG.

According to the format of TMDS transmission data shown in FIG. 7, image data representing an omnidirectional image as a spherical surface is defined as a baseband image having a different number of pixels for each line (or the number of pixels is not constant in each line) It can be transmitted by the TMDS channel of HDMI. When such a TMDS transmission data is transmitted from the disc player 210 in the AV system 200 shown in FIG. 2, the display 250 does not need to perform a process of demapping the received image data on the sphere, or Image data processing using a small amount of memory is sufficient. For example, in the case of a display that can be correctly displayed on a spherical surface, the received omnidirectional image data can be displayed as it is. Further, for example, in the case of a rectangular display such as a television, an appropriate rectangular portion can be cut out and displayed from the received omnidirectional image.

Of course, as shown in FIG. 6, even in the case of omnidirectional image data in which latitude lines are defined for each height interval h in the direction of the earth axis of the sphere and expressed as a sphere, each line as shown in FIG. , Or different in the number of pixels (or the number of pixels in each line is not constant) can be transmitted on the AV system 200 as TMDS transmission data, that is, a baseband image. Also in this case, the display 250 side has an advantage that the process of reversely mapping the received image data on the spherical surface is not necessary.

FIG. 8 shows an example of packing format when transmitting image data through three TMDS channels # 0 to # 2 of HDMI. However, the relationship between the TMDS clock and the pixel clock in the same figure is the relationship between TMDS clock = pixel clock.

In the RGB 4: 4: 4 scheme, 8-bit blue (B) data, 8-bit green (G) data, 8-bit red (in red) in the data area of each pixel in the TMDS channels # 0 to # 2. R) Data is arranged. Although FIG. 8 exemplifies only one of RGB 4: 4: 4 as a transmission method of image data, other transmission methods such as YCbCr 4: 4: 4 and YCbCr 4: 2: 0 may of course be used.

In the present embodiment, when transmitting the omnidirectional image data from the disk player 210 to the display 250, the HDMI transmitting unit 218 generates a baseband having a different number of pixels for each line, generated by the omnidirectional image processing unit 217. The signal is packed in a packing format as shown in FIG. 8 and output from the HDMI terminal 219.

As described above, the HDMI receiving unit 252 on the display 250 side has the EDID ROM 85 that stores the E-EDID related to its own performance. Also, the HDMI transmitting unit 218 on the disc player 210 side reads out the E-EDID of the HDMI receiving unit 252 from the EDID ROM 85 through the DDC 83 included in the HDMI cable 350, and the HDMI receiving unit 252 based on the E-EDID. The performance of can be set.

In the AV system 200 according to the present embodiment, the CPU 214 of the disc player 210 transmits all omnidirectional image data compatible with the display 250 based on the E-EDID read from the HDMI receiving unit 252 of the display 250. It also recognizes the method. Specifically, the CPU 214 of the disc player 210 recognizes whether or not the display 250 corresponds to a baseband image in which the number of pixels is different for each line (or the number of pixels is not constant in each line).

FIG. 9 shows an example of the data structure of E-EDID. The illustrated E-EDID is composed of a basic block and an extension lock.

At the top of the basic block, data defined by the E-EDID1.3 standard represented by "E-EDID1.3 Basic Structure" is placed, and then the conventional EDID represented by "Preferred timing" and The timing information different from the "Preferred timing" for maintaining the compatibility with the conventional EDID represented by "2nd timing" is disposed.

In addition, in the basic block, following “2nd timing”, information indicating the name of the display device represented by “Monitor NAME”, and aspect ratio 4: 3 and 16 represented by “Monitor Range Limits” The information which shows the number of displayable pixels about the case of: 9 is arrange | positioned in order.

On the other hand, at the beginning of the extension block, information such as a displayable image size (resolution), frame rate, information indicating interlace or progressive, aspect ratio, etc., described by “Short Video Descriptor” is described Data, data represented by "Short Audio Descriptor", data in which information such as reproducible audio codec method, sampling frequency, cutoff band, codec bit number, etc. are described, and left and right represented by "Speaker Allocation" Information on the speakers of is arranged in order.

In addition, in the extension block, following "Speaker Allocation", compatibility with the conventional EDID represented by "3rd timing" and data uniquely defined for each maker represented by "Vender Specific" is maintained. Timing information for maintaining compatibility with the conventional EDID represented by "4th timing".

In this embodiment, in this Vender Specific area, a data area to be expanded to store the omnidirectional image information is defined.

FIG. 10 shows an example of the data structure of the Vender Specific area in the extension block of E-EDID. In the illustrated Vendor Specific area, blocks 0 to N which are blocks of 1 byte are provided. Then, in the 12th byte to the 16th byte following the 0th byte to the 11th byte already defined, the data area of the omnidirectional image information to be stored by the sink device (in the present embodiment, the display 250) Define.

First, the 0th byte to the 7th byte will be described. The 0th byte placed at the beginning of the data represented by "Vender Specific" has a header indicating the data area of the data "Vender Specific" represented by "Vendor-Specific tag code (= 3)", Information indicating the length of data "Vender Specific" represented by Length (= N) "is arranged.

Further, in the first to third bytes, information indicating the number “0x000C03” registered for the HDMI, which is represented by “24 bit IEEE Registration Identifier (0x000C03) LSB first”, is arranged. Further, in the fourth byte and the fifth byte, information indicating the physical address of the 24-bit sink device, which is represented by "A", "B", "C", and "D" respectively, is arranged.

The sixth byte is a flag indicating the function supported by the sink device represented by "Supports-AI", represented by "DC-48bit", "DC-36bit", and "DC-30bit", respectively. A flag indicating whether or not the sink device corresponds to the transmission of the YCbCr 4: 4: 4 image, which is represented by “DC-Y 444”, is disposed in each of the information specifying the number of bits per pixel.

Further, in the seventh byte, information indicating the maximum frequency of the TMDS pixel clock represented by "Max-TMDS-Clock" is placed.

The 8th byte to the 11th byte are information used for transmitting and receiving a stereoscopic image (see, for example, Patent Document 2). Information on a stereoscopic image is stored in the eighth byte to the tenth byte. Also, in the 11th byte, information related to stereophonic sound is stored. Since stereoscopic image transmission and reception is not directly related to the technology disclosed herein, detailed description of the eighth to eleventh bytes is omitted.

Next, the 12th to 16th bytes will be described. The 12th byte to the 16th byte store information on the omnidirectional image.

In the seventh bit and the sixth bit of the twelfth byte, data indicating the method of the omnidirectional image data supported by the sink device is written. Specifically, as shown in FIG. 4 and FIG. 5, the seventh bit is the distance from the equatorial plane of the spherical surface defined by the latitude lines defined when arranging each pixel of the omnidirectional image data on the spherical surface. This is a method determined by the angle (SLBA: Sphere Latitude by Angle). In the sixth bit, as shown in FIG. 6, the distance between the latitude lines defined when arranging each pixel of the omnidirectional image data on the spherical surface is determined by the height in the direction of the earth axis of the spherical surface It is a system (SLBH: Sphere Latitude by Height). However, it is considered that the SLBA in which the seventh bit is set is a transmission method with higher image quality.

In the 13th byte and the 14th byte, the maximum number of pixels on the equator (Pixel Number on Equator) that can be handled by the sink device is written. In the pixel arrangement method shown in FIGS. 5 and 6, the maximum number of pixels 4000 is written in the 13th byte and the 14th byte.

In the 15th byte and the 16th byte, the maximum number of latitude lines (or the total number of lines in the video data section) that can be handled by the sink device (Total Latitude Line Number) is written.

Regarding the 13th byte to 16th byte, instead of writing the information about the number of pixels and the number of lines, a common combination is defined as VIC (Video format Identification Code) to assign bits, and the sink device can respond It is also possible to write data indicating a specific VIC.

According to the Veodor Specific area shown in FIG. 10, the sink device determines the interval of the latitude line for arranging the pixels of the omnidirectional image data (angle or height) as the omnidirectional image data method that the sink device can handle. Can be specified, the maximum number of pixels to be placed on the equator, the maximum number of latitudes that can be supported (the maximum number of lines), and so on.

In the AV system 200 shown in FIG. 2, when transmitting the omnidirectional image data to the display 250, the disc player 210 is in the Vendor Specific area of the E-EDID read out from the HDMI receiving unit 252 on the display 250 side. Based on the information mainly described in the 12th byte to the 16th byte, it is possible to identify the type of the omnidirectional image data that can be supported by the display 250. Then, the disc player 210 selects any method that can be supported by the display 250 of the transmission destination, and transmits the omnidirectional image data. The information on the omnidirectional image as described above may be arranged not in the Vendor Specific region but in an information storage region newly provided for the omnidirectional image transmission. In short, the means or method for storing information on the omnidirectional image in the sink device, and the means or method for transmitting the information on the omnidirectional image from the sink device to the source device are not limited to specific ones.

Also, when transmitting the omnidirectional image data, the disc player 210 transmits information on the currently transmitted image format to the display 250.

Here, the information on the image format includes a method (SLBA or SLBH) of determining the interval of the latitude line defined when arranging each pixel of the omnidirectional image data on the spherical surface, and the arrangement on the equator Maximum number of pixels to be displayed (Pixel Number on Equator), number of latitude lines (or total number of lines in video data section) defined when arranging each pixel of omnidirectional image data on a sphere (Total Latitude Line Number) , Pixel number of origin position, line number of origin position, etc. are included.

For example, the disc player 210 inserts information on the image format of the omnidirectional image data being transmitted during the blanking period of the omnidirectional image data (non-compressed (baseband) video signal) to be transmitted to the display 250 Can be sent to the display 250.

Specifically, the disc player 210 uses the AVI (Auxiliary Video Information) InfoFrame packet or the like in the HDMI signal to transmit information regarding the image format of the omnidirectional image data being transmitted as a block of the omnidirectional image data. It can be inserted in the ranking period.

The AVI InfoFrame packet is placed in a data island period (see FIG. 7) in TMDS transmission data. FIG. 11 shows an example data structure of an AVI InfoFrame packet. According to the HDMI standard, it is possible to transmit incidental information on an image being transmitted from a source device to a sink device using an AVI InfoFrame packet.

In the 0th byte, "Packet Type" indicating the type of data packet is defined. The “Packet Type” of the AVI InfoFrame packet is “0x82”. The first byte describes version information of the packet data definition. The AVI InfoFrame packet is currently "0x03", but when the transmission method of the omnidirectional image data is defined by the 18th to 26th bytes in this embodiment, the version is "0x04" as shown in the figure. It will be updated.

The second byte describes information indicating the packet length. The packet length of the AVI InfoFrame is currently "0x0E", but when the omnidirectional image output format information is defined in the 18th to 26th bytes in this embodiment, it is "0x17", as shown in FIG. become. Each piece of information described in the AVI InfoFrame is defined in CEA-861-D Section 6-4. Therefore, detailed description of the third to seventeenth bytes which are not particularly modified or added in the present embodiment will be omitted.

The 18th to 26th bytes contain information on the image format of the omnidirectional image data being transmitted.

In the eighteenth byte, any one of the transmission methods of the omnidirectional image data selected by the source device is designated. Specifically, data indicating the method of the omnidirectional image data selected by the source device is written in the seventh bit and the sixth bit of the eighteenth byte. Among them, the seventh bit is a method in which the interval of the latitude line defined when arranging each pixel of the omnidirectional image data on the spherical surface as shown in FIG. 4 and FIG. SLBA). In the sixth bit, as shown in FIG. 6, the distance between the latitude lines defined when arranging each pixel of the omnidirectional image data on the spherical surface is determined by the height in the direction of the earth axis of the spherical surface It is a system (SLBH).

In the 19th byte and the 20th byte, the maximum number of pixels on the equator (Pixel Number on Equator) in the transmitted omnidirectional image data is set.

In the 21st and 22nd bytes, the number of latitude lines (or the total number of lines in the video data section) (Total Latitude Line Number) in the transmitted omnidirectional image data is set.

In the 23rd byte through the 26th byte, the origin position information of the transmitted omnidirectional image data is set. The origin position is the standard position of the image. For example, the sink device can use this information to present the origin always in front of the viewer. The pixel number (Pixel Number) of the origin position is set in the 23rd byte and the 24th byte, and the line number (Line Number) of the origin position is set in the 25th byte and the 26th byte.

The display 250 as a sink device can control the processing of the image data transmitted from the disc player 210 based on the information on the image format notified using the AVI InfoFrame packet.

Specifically, the display 250 receives all celestial sphere image data from the disc player 210 when the seventh bit or the sixth bit of the eighteenth byte of the received AVI InfoFrame packet is set. It can be determined that it is being transmitted. Furthermore, the display 250 can obtain the number of pixels on the equator and the total number of lines in the transmitted omnidirectional image data based on the values set in the 19th byte to the 22nd byte of the same packet. The number of pixels on each line can be calculated and obtained based on that.

In addition, packing formats such as RGB 4: 4: 4, YCbCr 4: 4: 4, YCbCr 4: 2: 0, etc. are designated by the sixth bit and the fifth bit of the fourth byte of the AVI InfoFrame packet.

Furthermore, the display 250 can perform display position correction at the time of presentation on the display panel 258 based on the origin position information of the transmission image, which is set to the 23rd byte to the 26th byte of the AVI InfoFrame packet.

FIG. 12 shows, in the form of a flowchart, a processing procedure performed when the disc player 210 connects to the display 250 via the HDMI cable 350 in the AV system 200 shown in FIG. The illustrated procedure is basically implemented mainly by the CPU 214 in the disk player 210.

First, the disc player 210 checks whether the HPD signal of the HDMI cable 350 connected to its own HDMI terminal 219 is at high level (step S1201).

When the HPD signal of the HDMI cable 350 is at the low level (No in step S1201), the disc player 210 determines that the HDMI sink device is not connected via the HDMI cable 350, and the process ends. Do.

On the other hand, when the HPD signal of the HDMI cable 350 is at the high level (Yes in step S1201), the disc player 210 uses the DDC 83 of the HDMI cable 350 to connect the HDMI sink device (display 250). The E-EDID is read out from the EDID ROM 85 in the HDMI receiving unit 252 (step S1202).

The data structure of E-EDID is as illustrated in FIG. Further, as shown in FIG. 10, in the 12th to 16th bytes of the Vendor Specific area included in the extension block in E-EDID, the transmission method of the omnidirectional image compatible with the HDMI sink device is related. Information is stored. Among them, the seventh bit of the 12th byte indicates that the system of the latitude lines where the pixels of the omnidirectional image data are arranged is determined by the angle from the equatorial plane of the spherical surface (SLBA), and the sixth The bit indicates that the interval of the same latitude line is a scheme (SLBH) determined by the height in the direction of the ground axis of the sphere. Therefore, the disc player 210 checks whether the seventh bit or the sixth bit of the 12th byte is set, and the connected HDMI sink device corresponds to the display of the omnidirectional image. It can be confirmed that In the processing procedure shown in FIG. 12, it is assumed that the connected HDMI sink device is a device (display 250) compatible with the display of the omnidirectional image.

Subsequently, the disc player 210 checks whether there is a omnidirectional image to be transmitted to the connected HDMI sink device (display 250) (step S1203).

If there is no omnidirectional image to be transmitted (No in step S1203), the disc player 210 sets data indicating non-transmission of the omnidirectional image in the AVI InfoFrame packet to be inserted in the blanking period of the image data. Then (step S1207), the process ends. For example, non-transmission of the omnidirectional image can be indicated by setting 0 to any of the seventh bit and the sixth bit of the 18th byte of the AVI InfoFrame packet.

If there is a omnidirectional image to be transmitted to the display 250 (Yes in step S1203), the disc player 210 starts processing for transmitting the omnidirectional image data to the display 250.

Here, the disc player 210 determines the transmission method of the omnidirectional image data to be transmitted to the display 250 (step S1204). At this time, the disc player 210 selects the omnidirectional image in consideration of the transmission method of the omnidirectional image compatible with the display 250 described in the Vendor Specific area of the EDID read from the display 250 in step S1202. Determine the data transmission method. The disc player 210 may use, for example, a method of determining an interval of latitude lines, a maximum number of pixels arranged in one line, a total number of latitude lines (or a total number of lines), and an omnidirectional sphere Origin position information of the image data is determined.

Then, the disc player 210 determines whether the transmission of the omnidirectional image data is started (step S1205). If transmission of the omnidirectional image data is not started (No in step S1205), the disc player 210 sets data indicating non-transmission of the omnidirectional image in the AVI InfoFrame packet (step S1207), and ends this processing. Do.

When the transmission of the omnidirectional image data is started (Yes in step S1205), the disc player 210 indicates the transmission method of the omnidirectional image data in the 18th to 26th bytes of the AVI InfoFrame packet. Data is set (step S1206), and the process ends.

FIG. 13 shows, in the form of a flowchart, a detailed processing procedure for the disc player 210 to determine the transmission method of the omnidirectional image data, which is performed in step S1205 in the flowchart shown in FIG. .

First, the disc player 210 checks whether or not the seventh bit of the twelfth byte of the Vendor Specific area included in the extension block of the E-EDID acquired from the EDID ROM 85 of the HDMI sink device (display 250) is set. (Step S1301).

In the present embodiment, the method of determining the spacing of the latitude lines when arranging each pixel of the omnidirectional image data on the sphere includes the SLBA method of determining the angle from the equatorial plane of the sphere and the height of the sphere in the ground axis direction. There are two types of SLBH method determined by the above (described above). Among them, the above-mentioned person's SLBA method is a transmission method with higher image quality.

Therefore, when the seventh bit of the twelfth byte of the Vendor Specific area is set, and the display 250 designates the SLBA method (Yes in step S1301), the disc player 210 preferentially gives priority to the latitude line. The SLBA method in which the distance is determined by the angle of the spherical surface from the equatorial plane is selected (step S1302), and the present process ends.

If the seventh bit of the 12th byte of the Vendor Specific area is not set (No in step S1301), the disc player 210 subsequently sets the 6th bit of the 12th byte of the Vendor Specific area. It is checked whether it has been done (step S1303).

Here, when the sixth bit of the twelfth byte of the Vendor Specific area is set, and the display 250 designates the SLBH method (Yes in step S1303), the disc player 210 sets the latitude line to the spherical surface. The SLBH method determined by the height in the ground axis direction is selected (step S1304), and the process ends.

When neither the seventh bit nor the sixth bit of the twelfth byte of the Vendor Specific area is set (No in step S1303), the disc player 210 transmits the omnidirectional image data to the display 250. It is determined that there is no possible method, the output non-selection of the omnidirectional image is set (step S1305), and the process ends.

As described above, in the AV system 200 shown in FIG. 2, when transmitting the omnidirectional image data from the disc player 210 to the display 250, the disc player 210 is an omnidirectional sphere that the display 250 can handle. The transmission method information of the image data is acquired, and the omnidirectional image data is transmitted by the transmission method compatible with the display 250. Also, when transmitting the omnidirectional image data, the disc player 210 transmits the transmission method information to the display 250.

Therefore, it should be fully understood that, in the AV system 200, transmission of omnidirectional image data can be well performed between the disc player 210 and the display 250.

Further, in the present embodiment, the disc player 210 describes transmission method information of omnidirectional image data in an AVI InfoFrame packet, and transmits it to the display 250 using the blanking period of the image data (video signal). I am trying to do it.

However, the method for the disc player 210 to transmit the transmission method information of the omnidirectional image data to the display 250 is not limited to the above. For example, the disc player 210 may transmit transmission method information of omnidirectional image data to the display 250 via the CEC line 84 of the HDMI cable 350. Alternatively, the disc player 210 transmits the transmission method information of the omnidirectional image data to the display 250 via the bidirectional communication path (described above) configured by the reserve line 88 of the HDMI cable 350 and the HPD line 86. You may do so.

FIG. 2 exemplifies the AV system 200 using the HDMI transmission path, but as a base band digital interface used for transmission of omnidirectional image data, DVI (Digital Visual Interface), other than HDMI, Examples include DP (Display Port) interface and wireless interface using 60 MHz millimeter wave. The technique disclosed in the present specification can be similarly applied to transmission of omnidirectional image data using any of these digital interfaces.

Also, in the case of transmitting omnidirectional image data between a source device and a sink device, which are combinations of a disc player and a display, the technology disclosed in the present specification can be applied similarly.

The technology disclosed herein has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the scope of the technology disclosed herein.

In the present specification, an AV system adopting the HDMI standard has been described focusing on an embodiment in which all celestial sphere image data represented as a spherical surface is transmitted as TMDS transmission data, but the gist of the technology disclosed in the present specification. Is not limited to this. The techniques disclosed in the present specification can be similarly applied to systems based on transmission standards other than HDMI and various systems that transmit baseband images in a system other than TMDS.

As a base band digital interface used for transmission of omnidirectional image data, a DVI interface, a DP interface, a wireless interface using 60 MHz millimeter waves, and the like can be mentioned besides HDMI. The technique disclosed in the present specification can be similarly applied to transmission of omnidirectional image data using any of these digital interfaces.

Further, the present specification describes a pixel arrangement method for representing all celestial sphere image data as a spherical surface by arranging each pixel at predetermined intervals in the longitudinal direction on each of the latitudinal lines on which the latitude line and the longitude line are defined. As described above, such a pixel layout method can be applied not only to transmission of omnidirectional image data but also to various processing such as recording.

In addition, even in the case of image data mapped to a curved surface other than a spherical surface, a method of transmitting as a baseband image in which the number of pixels differs for each line can be applied.

In short, although the technology disclosed herein has been described in the form of exemplification, the contents of the specification should not be interpreted in a limited manner. In order to determine the scope of the technology disclosed herein, the claims should be referred to.

Note that the technology disclosed in the present specification can also be configured as follows.
(1) A definition unit that defines latitude lines and longitude lines on a sphere that represents an omnidirectional image;
An arrangement unit for arranging the pixels of the omnidirectional image on each of the defined latitude lines;
An image processing apparatus equipped with
(1-1) An index assignment unit is further provided, which assigns a latitude index to each latitude line and assigns a longitude index starting from zero longitude to pixels arranged on each latitude line.
The image processing apparatus as described in said (1).
(2) The arrangement unit arranges pixels at approximately equal intervals starting from the location of zero longitude on each latitude line.
The image processing apparatus as described in said (1).
(3) The definition unit determines the distance between the latitude lines based on either the angle from the equatorial plane of the spherical surface or the height of the spherical surface in the ground axis direction.
The image processing apparatus according to any one of the above (1) and (2).
(4) A processing unit for sequentially arranging pixels on each of the latitudinal lines of the spherical surface in one line to generate baseband signals having different numbers of pixels for each line;
A transmitter configured to transmit the baseband signal to an external device via a predetermined transmission path;
The image processing apparatus according to any one of (1) to (3), further comprising:
(5) an acquisition unit for acquiring transmission method information on a transmission method of the omnidirectional image data compatible with the external device;
A selection unit that selects a transmission method of the omnidirectional image data based on the acquired transmission method information;
And further
The processing unit generates the baseband signal according to the selected transmission scheme.
The image processing apparatus as described in said (4).
(6) The acquisition unit acquires the transmission method information from the external device via the predetermined transmission path.
The image processing device according to (5).
(7) The predetermined transmission path is a transmission path based on the HDMI standard,
The acquisition unit acquires the transmission method information from an E-EDID stored in an EDID ROM included in the external device.
The image processing apparatus as described in said (6).
(8) The acquisition unit determines, as the transmission method information, at least one of a method of determining an interval of the latitude lines, a maximum number of pixels arranged in one line, and a maximum number of maximum latitude lines (maximum number of lines) Get
The selection unit is a method of determining the interval of the latitude line when the image processing unit generates the baseband signal based on the acquired transmission method information, the maximum number of pixels arranged in one line, correspondence Select at least one of the maximum possible latitudes (maximum number of lines),
The image processing apparatus according to any one of the above (5) to (7).
(9) The information processing apparatus further comprises a notification unit that notifies the external device of format information related to the format of the omnidirectional image data.
The image processing apparatus according to any one of the above (4) to (8).
(10) The notification unit transmits the format information to the external device via the predetermined transmission path.
The image processing apparatus according to (9).
(11) The predetermined transmission path is a transmission path based on the HDMI standard,
The notification unit transmits the format information using a blanking period when the transmission unit transmits the baseband signal.
The image processing apparatus according to any one of the above (9) or (10).
(12) The notification unit determines, as the format information, a method of determining an interval of the latitude lines, a maximum number of pixels arranged in one line, a total number of the latitude lines or a total number of lines included in the baseband signal Informing at least one of the origin position information of the omnidirectional image data,
The image processing apparatus according to any one of the above (9) to (11).
(13) A receiving unit for receiving baseband signals of different numbers of pixels for each line, which are generated by sequentially arranging the pixels on each latitude line of the spherical surface in one line, from an external device through a predetermined transmission path When,
A processing unit that performs processing for displaying an omnidirectional image based on the baseband signal;
The image processing apparatus according to any one of (1) to (3), further comprising:
(14) The information processing apparatus further includes a notification unit that notifies the external device of transmission method information on a transmission method of compatible omnidirectional image data.
The image processing apparatus according to (13).
(15) The notification unit determines, as the transmission method information, at least one of a method of determining an interval of the latitude lines, a maximum number of pixels arranged in one line, and a maximum number of maximum latitude lines (maximum number of lines) To notify,
The image processing apparatus as described in said (14).
(16) The predetermined transmission path is a transmission path based on the HDMI standard,
An EDID ROM storing an E-EDID including the transmission method information,
The image processing apparatus according to any one of the above (14) or (15).
(17) The information processing apparatus further comprises an acquisition unit for acquiring format information on a format of the omnidirectional image data from the external device,
The processing unit performs processing for displaying an omnidirectional image based on the format information.
The image processing apparatus according to any one of the above (13) to (16).
(18) The acquisition unit may use, as the format information, a method of determining an interval of the latitude lines, a maximum number of pixels arranged in one line, a total number of the latitude lines or a total number of lines included in the baseband signal. Acquiring at least one of origin position information of the omnidirectional image data from the external device;
The image processing apparatus according to (17).
(19) The predetermined transmission path is a transmission path based on the HDMI standard,
The acquisition unit acquires the format information using a blanking period when the transmission unit transmits the baseband signal.
The image processing apparatus according to any one of the above (18) or (19).
(20) defining the latitude line and the longitude line on the sphere that represents the omnidirectional image;
Arranging the pixels of the omnidirectional image on each of the defined latitude lines;
An image processing method having:
(21) An image processing unit that generates baseband signals having different numbers of pixels for each line from omnidirectional image data configured by arranging pixels on a spherical surface;
A transmitter configured to transmit the baseband signal to an external device via a predetermined transmission path;
A transmitter equipped with:
(22) The image processing unit arranges, for each line, pixels on each of the latitudinal lines of the spherical surface in which the latitude line and the longitude line are defined, and generates the baseband signal.
The transmitter according to (21).
(23) an acquisition unit for acquiring transmission method information on a transmission method of all celestial sphere image data that can be handled by the external device;
A selection unit that selects a transmission method of the omnidirectional image data based on the acquired transmission method information;
And further
The image processing unit generates the baseband signal according to the selected transmission scheme.
The transmitter according to any one of the above (21) or (22).
(24) The acquisition unit acquires the transmission scheme information from the external device via the predetermined transmission path.
The transmitter according to (23).
(25) The predetermined transmission path is a transmission path based on the HDMI standard,
The acquisition unit acquires the transmission method information from an E-EDID stored in an EDID ROM included in the external device.
The transmitter according to any one of the above (23) or (24).
(26) The omnidirectional image data is configured by arranging each pixel at a predetermined interval in the longitude direction on each latitude line of the sphere on which the latitude line and the longitude line are defined,
The acquisition unit acquires, as the transmission method information, at least one of a method of determining an interval of the latitude lines, a maximum number of pixels arranged in one line, and a maximum number of maximum latitude lines (maximum number of lines) that can be supported. ,
The selection unit is a method of determining the interval of the latitude line when the image processing unit generates the baseband signal based on the acquired transmission method information, the maximum number of pixels arranged in one line, correspondence Select at least one of the maximum possible latitudes (maximum number of lines),
The transmitter according to any one of (23) to (25).
(27) The information processing apparatus further comprises a notification unit that notifies the external device of format information related to the format of the omnidirectional image data.
The transmitter according to any one of (21) to (26).
(28) The notification unit transmits the format information to the external device via the predetermined transmission path.
The transmitter according to (27).
(29) The predetermined transmission path is a transmission path based on the HDMI standard,
The notification unit transmits the format information using a blanking period when the transmission unit transmits the baseband signal.
The transmitter according to any one of the above (27) or (28).
(30) The omnidirectional image data is configured by arranging each pixel at a predetermined interval in the longitude direction on each latitude line of the spherical surface on which the latitude line and the longitude line are defined,
The notification unit determines, as the format information, a method of determining an interval of the latitude lines, a maximum number of pixels arranged in one line, a total number of the latitude lines or a total number of lines included in the baseband signal At least one of origin position information of image data is notified,
The transmitter according to any one of (27) to (29).
(31) an image processing step of generating baseband signals having different numbers of pixels for each line from omnidirectional image data configured by arranging pixels on a spherical surface;
Transmitting the baseband signal to an external device through a predetermined transmission path;
Transmission method.
(32) A receiving unit for receiving baseband signals of different numbers of pixels for each line, which are generated from omnidirectional image data configured by arranging pixels on a spherical surface, from an external device through a predetermined transmission path When,
A processing unit that performs processing for displaying an omnidirectional image based on the baseband signal;
A receiver comprising:
(33) The information processing apparatus further comprises a notification unit for notifying the external device of transmission method information on a transmission method of compatible omnidirectional image data.
The receiving device according to (32).
(34) The omnidirectional image data is configured by arranging each pixel at a predetermined interval in the longitude direction on each latitude line of the spherical surface on which the latitude line and the longitude line are defined,
The notification unit notifies at least one of the method of determining the interval of the latitude lines, the maximum number of pixels arranged in one line, and the maximum number of maximum latitude lines (maximum number of lines) that can be supported as the transmission method information. ,
The receiving device according to (33).
(35) The image processing apparatus further comprises an acquisition unit for acquiring format information on the format of the omnidirectional image data from the external device,
The processing unit performs processing for displaying an omnidirectional image based on the format information.
The receiver according to any one of (32) to (34).
(36) The acquisition unit determines, as the format information, a method of determining the interval of the latitude lines, the maximum number of pixels arranged in one line, the total number of the latitude lines or the total number of lines included in the baseband signal Acquire at least one of origin position information of the omnidirectional image data;
The receiving device according to (35).
(37) a receiving step of receiving, from an external device, baseband signals having different numbers of pixels for each line, generated from omnidirectional image data configured by arranging pixels on a spherical surface;
A processing step of performing processing for displaying an omnidirectional image based on the baseband signal;
Receiving method.

81 ... HDMI transmitter, 82 ... HDMI receiver 83 ... DDC, 84 ... CEC line 85 ... EDID ROM, 86 ... HPD line 87 ... Power supply line, 88 ... Reserved line 200 ... AV system 210 ... Disk player 211 ... Ethernet interface 212: Disk drive, 213: Memory, 214: CPU
215: High-speed data line interface 216: General image processing unit, 217: All-sky image processing unit 218: HDMI transmission unit 219: HDMI terminal, 220, 221: Internal bus 222: Network terminal 250: Display 251: High-speed data Line interface 252: HDMI receiver, 253: CPU, 254: memory 255: Ethernet interface 256: all-sky image processor, 257: general image processor 258: display panel, 259:

HDMI terminal

260, 261 ... Internal bus, 262 ... Network terminal 350 ... HDMI cable

Claims

A definition unit that defines latitude lines and longitude lines on a sphere that represents an omnidirectional image;
An arrangement unit for arranging the pixels of the omnidirectional image on each of the defined latitude lines;
An image processing apparatus equipped with
The arrangement unit arranges pixels at substantially equal intervals starting from the location of zero longitude on each latitude line.
The image processing apparatus according to claim 1.
The definition unit determines the distance between the latitude lines based on either the angle from the equatorial plane of the spherical surface or the height of the spherical surface in the ground axis direction.
The image processing apparatus according to claim 1.
A processing unit that sequentially arranges pixels on each of the latitudinal lines of the spherical surface in one line, and generates baseband signals having different numbers of pixels for each line;
A transmitter configured to transmit the baseband signal to an external device via a predetermined transmission path;
The image processing apparatus according to claim 1, further comprising:
An acquisition unit configured to acquire transmission method information on a transmission method of all-sky image data compatible with the external device;
A selection unit that selects a transmission method of the omnidirectional image data based on the acquired transmission method information;
And further
The processing unit generates the baseband signal according to the selected transmission scheme.
The image processing apparatus according to claim 4.
The acquisition unit acquires the transmission method information from the external device via the predetermined transmission path.
The image processing apparatus according to claim 5.
The predetermined transmission path is a transmission path based on the HDMI standard,
The acquisition unit acquires the transmission method information from an E-EDID stored in an EDID ROM included in the external device.
The image processing apparatus according to claim 6.
The acquisition unit acquires, as the transmission method information, at least one of a method of determining an interval of the latitude lines, a maximum number of pixels arranged in one line, and a maximum number of maximum latitude lines (maximum number of lines) that can be supported. ,
The selection unit is a method of determining the interval of the latitude line when the image processing unit generates the baseband signal based on the acquired transmission method information, the maximum number of pixels arranged in one line, correspondence Select at least one of the maximum possible latitudes (maximum number of lines),
The image processing apparatus according to claim 5.
And a notification unit for notifying the external device of format information on the format of the omnidirectional image data.
The image processing apparatus according to claim 4.
The notification unit transmits the format information to the external device via the predetermined transmission path.
The image processing apparatus according to claim 9.
The predetermined transmission path is a transmission path based on the HDMI standard,
The notification unit transmits the format information using a blanking period when the transmission unit transmits the baseband signal.
The image processing apparatus according to claim 9.
The notification unit determines, as the format information, a method of determining an interval of the latitude lines, a maximum number of pixels arranged in one line, a total number of the latitude lines or a total number of lines included in the baseband signal At least one of origin position information of image data is notified,
The image processing apparatus according to claim 9.
A receiving unit that receives baseband signals of different numbers of pixels for each line, which are generated by sequentially arranging the pixels on each of the spherical latitude lines in one line, from an external device via a predetermined transmission path;
A processing unit that performs processing for displaying an omnidirectional image based on the baseband signal;
The image processing apparatus according to claim 1, further comprising:
The information processing apparatus further comprises a notification unit for notifying the external device of transmission method information on a transmission method of the compatible omnidirectional image data.
The image processing apparatus according to claim 13.
The notification unit notifies at least one of the method of determining the interval of the latitude lines, the maximum number of pixels arranged in one line, and the maximum number of maximum latitude lines (maximum number of lines) that can be supported as the transmission method information. ,
The image processing apparatus according to claim 14.
The predetermined transmission path is a transmission path based on the HDMI standard,
An EDID ROM storing an E-EDID including the transmission method information,
The image processing apparatus according to claim 14.
It further comprises an acquisition unit for acquiring format information on the format of the omnidirectional image data from the external device
The processing unit performs processing for displaying an omnidirectional image based on the format information.
The image processing apparatus according to claim 13.
The acquisition unit determines the interval of the latitude line as the format information, the maximum number of pixels arranged in one line, the total number of the latitude lines or the total number of lines included in the baseband signal, the omnidirectional sphere Acquiring at least one of origin position information of image data from the external device;
The image processing apparatus according to claim 17.
The predetermined transmission path is a transmission path based on the HDMI standard,
The acquisition unit acquires the format information using a blanking period when the transmission unit transmits the baseband signal.
The image processing apparatus according to claim 18.
Defining a latitude line and a longitude line on a sphere representing an omnidirectional image;
Arranging the pixels of the omnidirectional image on each of the defined latitude lines;
An image processing method having: