JP2003319351A - Service providing system, broadcast apparatus and method, receiving apparatus and method, recording medium, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To display a video image with higher resolution in the case of displaying the video image with magnification. <P>SOLUTION: A broadcast video frame includes an omnidirectional video image 251 and zooming images C1 to C5. For example, areas a and h of the omnidirectional video image 251 correspond to the zooming image C4. An receiving apparatus segments a designated area from the omnidirectional video image 251 and displays the image of the area on a display apparatus of a television receiver (TV). When a user enters an instruction of magnifying the displayed image and the instructed area includes 80 percentages of the area a or h, the receiving apparatus displays the zooming image C4 with higher resolution in place of segmenting the omnidirectional video image 251 and displaying the segmented image. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a service providing system, a broadcasting apparatus and method, a receiving apparatus and method, a recording medium, and a program, and particularly, when displaying an enlarged image, a higher resolution image is displayed. The present invention relates to a service providing system, a broadcasting device and method, a receiving device and method, a recording medium, and a program capable of performing the above.

[0002]

2. Description of the Related Art An omnidirectional camera system is known as a device for providing a 360-degree landscape image to a user. The omnidirectional camera is configured, for example, by arranging a plurality of cameras radially. The images photographed by the plurality of cameras arranged in a radial pattern are pasted so that their boundaries are matched by a predetermined process. When the user views the image thus pasted (hereinafter, this image is referred to as an omnidirectional image or an omnidirectional image), the user views 360 degrees around himself.
It is possible to view an image in a desired direction from among landscapes in all directions. As described above, in the omnidirectional camera system, a region much wider than the field of view of each camera can be generated as one image.

[0003]

[Problems to be Solved by the Invention]
For example, in the case of broadcasting an omnidirectional video, when a user who receives the broadcast views a part of the omnidirectional video by enlarging (zooming up), the resolution of the video is low, so that details cannot be clearly displayed. There was a problem.

It is possible to improve the resolution of the omnidirectional video by improving the shooting resolution of each camera constituting the omnidirectional camera, but in that case, the amount of data to be broadcast becomes large. was there.

There is also a method of transmitting a high-resolution video by using a plurality of channels and switching the channel according to the range of the video viewed by the user, but in this case, there is a problem that the cost becomes high. It was

The present invention has been made in view of such a situation, and an object thereof is to display a video of higher resolution when the video is enlarged and displayed.

[0007]

In the service providing system of the present invention, the broadcasting device expands the first image taken by the first photographing device and a part of the photographing range of the first photographing device. An acquisition unit that acquires the second image captured by the second image capturing apparatus that captures an image; a creation unit that creates position information indicating the position of the second image with respect to the first image acquired by the acquisition unit; The position information created by the creating unit is provided with a broadcasting unit that broadcasts the position information together with the first image and the second image, and the receiving device receives the broadcast from the broadcasting device and the receiving unit receives the broadcast. The display means for displaying the first image included in the broadcast, the receiving means for receiving the input of the instruction to enlarge and display a part of the first image displayed by the display means, and the receiving means,
When the input of the instruction to enlarge and display a part of the first image is received, whether or not the second image corresponding to the range in which the input of the instruction to enlarge and display is received is included in the broadcast is displayed. When it is determined that the determination unit that determines based on the position information and the second image corresponding to the range in which the input of the instruction to enlarge and display is accepted are included in the broadcast by the determination unit, the display unit displays the second image. And a control unit for displaying the second image.

The broadcasting apparatus of the present invention is photographed by the first image photographed by the first photographing apparatus and the second photographing apparatus which magnifies a part of the photographing range of the first photographing apparatus. The acquisition unit that acquires the second image, the creation unit that creates the position information indicating the position of the second image with respect to the first image acquired by the acquisition unit, and the position information created by the creation unit And a broadcasting means for broadcasting together with the first image and the second image.

The first photographing device may be an omnidirectional camera, and the first image may be an omnidirectional image.

[0010] The broadcasting means is caused to broadcast the information indicating that the first image is the omnidirectional video together with the first image, the second image, and the position information. You can

The creating means receives the input of an operation for designating the position of the second image corresponding to the first image, and the first means based on the operation received by the receiving means. A calculation means for calculating the coordinates of the second image corresponding to the first image may be provided, and the position information in a predetermined format including the coordinates calculated by the calculation means may be created. it can.

The acquisition means may be configured to acquire a plurality of the second images photographed by a plurality of the second photographing devices that magnify and photograph a part of the first image. it can.

The creating means may create the position information including the positional relationship of the plurality of second images with respect to the first image.

In the broadcasting method of the present invention, the first image captured by the first image capturing device and the second image capturing device capturing a part of the image capturing range of the first image capturing device by enlarging the image are captured. An acquisition step of acquiring the second image and a second step of acquiring the first image by the processing of the acquisition step.
The creation step of creating the position information indicating the position of the image of, and the position information created by the processing of the creation step,
A broadcast step of broadcasting with the first image and the second image.

The program of the first recording medium of the present invention is
The first image captured by the first image capturing device, and the first image
Second part of the shooting device to enlarge the shooting range
An acquisition step of acquiring a second image captured by the image capturing apparatus and a first image acquired by the processing of the acquisition step.
Including a creating step for creating position information indicating the position of the second image with respect to the image, and a broadcasting step for broadcasting the position information created by the process of the creating step together with the first image and the second image. Is characterized by.

The first program of the present invention is the first image captured by the first image capturing device and the second image capturing device for enlarging and capturing a part of the image capturing range of the first image capturing device. An acquisition step of acquiring the acquired second image, a creation step of creating position information indicating the position of the second image with respect to the first image acquired by the processing of the acquisition step, and a creation step of the creation step. And causing the computer to execute a step of broadcasting the location information together with the first image and the second image.

The receiving apparatus of the present invention has the first image, the second image which is an enlarged image of a part of the first image, and the position information indicating the position of the second image with respect to the first image. Receiving means for receiving the included broadcast, display means for displaying the first image included in the broadcast received by the receiving means, and instruction for enlarging and displaying a part of the first image displayed by the display means If the input of the instruction to display the portion of the first image is received by the first receiving unit that receives the input of the, the range of the input of the instruction to display the enlarged image is received. A first determination unit that determines whether or not the corresponding second image is included in the broadcast based on the position information, and a range in which the input of the instruction to enlarge the display is accepted by the first determination unit. The second image corresponding to If it is determined that the rare,
And a control unit that controls the display unit to display the second image.

The first image may be an omnidirectional image.

In the broadcast, the first image includes information indicating that the first image is an omnidirectional image, and the first image is omnidirectional according to the presence or absence of information indicating that the first image is the omnidirectional image. A second determination means for determining whether or not the image is a azimuth image can be further provided.

The control means causes the display means to display a part of the omnidirectional image when the second determining means determines that the first image is the omnidirectional image. Can be made to.

A second receiving means for receiving an input of an instruction to move the range of the omnidirectional image displayed on the display means is further provided, and the control means receives the input by the second receiving means. The range of the omnidirectional image displayed on the display unit can be moved according to the instruction.

In the first determination means, the range of the first image in which the input of the instruction to enlarge and display is accepted is set in the second image by a predetermined preset value. When the range of the ratio is included, it is possible to determine that the second image corresponding to the range in which the input of the instruction for the enlarged display is accepted is included in the broadcast.

The broadcast received by the receiving means may include a plurality of the second images.

If it is determined by the first determination means that the second image corresponding to the range in which the input of the instruction to enlarge the display is accepted is not included in the broadcast, the control is performed. The means includes the display means and the first
It is possible to control so as to display a part of the image of FIG.

According to the receiving method of the present invention, the first image, the second image which is an enlarged image of a part of the first image, and the position information indicating the position of the second image with respect to the first image are stored. A reception step of receiving the included broadcast, a display step of displaying the first image included in the broadcast received by the processing of the reception step, and a part of the first image displayed by the processing of the display step. When the input of the instruction to display the enlarged image is received by the receiving step of receiving the input of the instruction to display the enlarged image and the processing of the receiving step, the range of the input of the instruction to display the enlarged image is received. The determination step of determining whether the corresponding second image is included in the broadcast based on the position information, and the processing of the determination step, the range in which the input of the enlarged display instruction is accepted. Second image corresponding to the, if it is determined to be included in the broadcast, characterized in that it comprises a control step of controlling so as to display the second image.

The program of the second recording medium of the present invention is
Reception control for controlling reception of a broadcast including a first image, a second image that is an enlarged image of a part of the first image, and position information indicating the position of the second image with respect to the first image And a first display control step for controlling the display of the first image included in the broadcast whose reception is controlled by the processing of the reception control step, and the display is controlled by the processing of the first display control step. An accepting step of accepting an input of an instruction to magnify and display a part of the first image, and an instruction to magnify and display an input of an instruction to magnify and display a part of the first image by the processing of the accepting step The second image corresponding to the range where the input of
The determination step of determining whether or not it is included in the broadcast based on the position information, and the second image corresponding to the range in which the input of the instruction to enlarge the display is accepted is included in the broadcast by the processing of the determination step. If it is determined that the second image is displayed, a second display control step of controlling the second image to be displayed is included.

A second program of the present invention is a position indicating the position of the second image with respect to the first image, the second image which is an enlarged image of a part of the first image, and the position indicating the position of the second image with respect to the first image. A reception control step for controlling reception of a broadcast including information; a first display control step for controlling display of a first image included in the broadcast whose reception is controlled by the processing of the reception control step; An acceptance step of accepting an input of an instruction to magnify and display a part of the first image whose display is controlled by the processing of the display control step, and an instruction to magnify and display a part of the first image by the processing of the acceptance step. And a determination step of determining, based on the position information, whether or not the second image corresponding to the range in which the input of the enlarged display instruction is received is included in the broadcast. Su When it is determined that the second image corresponding to the range in which the input of the instruction to magnify the display is received is included in the broadcast by the processing of the pop-up, the second image is controlled to be displayed. And causing the computer to execute the two display control steps.

In the service providing system of the present invention, in the broadcasting device, the first image captured by the first image capturing device is used.
Of the second image captured by the second image capturing device that captures a part of the image capturing range of the first image capturing device in an enlarged manner, and the second image of the second image with respect to the acquired first image is acquired. Position information indicating the position is created. Then, the created position information is broadcast together with the first image and the second image. Further, in the receiving device, the broadcast is received from the broadcasting device, and first, the first image included in the received broadcast is displayed. When the input of the instruction to enlarge and display a part of the displayed first image is accepted, the second image corresponding to the range in which the input of the instruction to enlarge and receive is accepted is included in the broadcast. If it is determined that the second image corresponding to the range in which the input of the instruction to enlarge and display is accepted is included in the broadcast, the second image is determined. Is displayed.

Broadcasting apparatus and method of the present invention, recording medium,
In the program, the first image captured by the first image capturing device and the second image captured by the second image capturing device that enlarges and captures a part of the image capturing range of the first image capturing device are recorded. The acquired position information indicating the position of the second image with respect to the acquired first image is created. Then, the created position information is broadcast together with the first image and the second image.

Receiving apparatus and method of the present invention, recording medium,
In addition, the program receives the broadcast including the first image, the second image that is an enlarged image of a part of the first image, and position information indicating the position of the second image with respect to the first image. Then, the first image included in the received broadcast is displayed. Then, when the input of the instruction to enlarge and display a part of the displayed first image is accepted,
Whether or not the second image corresponding to the range in which the input of the enlarged display is accepted is included in the broadcast is determined based on the position information, and the second image is included in the range in which the input of the enlarged display instruction is accepted. When it is determined that the corresponding second image is included in the broadcast, the second image is controlled to be displayed.

[0031]

1 shows an example of the configuration of a service providing system to which the present invention is applied. In this service providing system, an image captured using the omnidirectional camera 31 and five cameras 32-1 to 32-5 owned by the broadcasting station 11 is subjected to a predetermined process in the broadcasting device 33 (details will be described later). Then, the data is broadcast via the artificial satellite 12.

The receiving device 13 receives the satellite broadcast via the artificial satellite 12, displays an image which has been subjected to a predetermined process based on the received data on the display of the television receiver 14, and causes the user 15 to display the image. Make them watch.

Next, FIG. 2 shows an omnidirectional camera 31 and cameras 32-1 to 32-5 (hereinafter, if it is not necessary to distinguish the cameras 32-1 to 32-5 from each other, they are collectively referred to as the camera 32). This is also the case with other devices or reference numerals indicating parts), and is a diagram showing an arrangement example of each camera when a surrounding landscape is photographed. In FIG. 2, with the omnidirectional camera 31 as the center, the cameras 32-1 to 32-5 are concentrically arranged around the omnidirectional camera 31. In the example of FIG. 2, the camera 32-1 and the camera 32-2 are arranged close to each other, and the cameras 32-3 to 32
-5 are placed in close proximity. Note that the camera 32-1
32 to 32-5 are not limited to the arrangement as shown in FIG. The intervals between the cameras 32 can be set arbitrarily as long as they are arranged radially around the omnidirectional camera 31.

Next, FIG. 3 is a diagram showing an external configuration of the omnidirectional camera 31. In FIG. 3, the omnidirectional camera 31
8 cameras 51-1 to 5-5 as shown at the top of the
1-8 are arranged at substantially equal intervals on the same circumference.
The lenses of the cameras 51-1 to 51-8 for capturing images are directed downward, and the cameras 5-1 to 51-8 are located below the cameras 5-1 to 51-8.
1-1 to 51-8, each of which corresponds to eight plane mirrors, is provided with a reflecting portion 5 in which an eight-sided pyramid is arranged at a predetermined angle.
The cameras 51-1 to 51-8, in which No. 2 is installed, photograph the landscape reflected by the reflecting section 52.

Each of the cameras 51-1 to 51-8 captures the surrounding landscape reflected by the corresponding plane mirror, and the omnidirectional camera 31 as a whole captures a mirror image of a 360-degree image in the horizontal direction. It becomes possible.

An image obtained by combining the images photographed by the omnidirectional camera 31 by a predetermined process is shown in FIG.
Is viewed by the user as shown in FIG. In FIG. 4, an omnidirectional image 71 is an image captured by the cameras 51-1 to 51-8 forming the omnidirectional camera 31 and joined by a predetermined process.

That is, the image 71 forming an omnidirectional image
-1 is an image taken by the camera 51-1 and
The image 71-2 is an image captured by the camera 51-2, the image 71-3 is an image captured by the camera 51-3, and the image 71-4 is captured by the camera 51-4. The image 71-5 is an image.
-5 is an image captured by the camera 51-6, and an image 71-6 is an image captured by the camera 51-6.
Reference numeral 7 is an image captured by the camera 51-7, and image 71-8 is an image captured by the camera 51-8.

As shown in FIG. 4, each camera 51-1
The images captured by the cameras 51 through 51-8 are stitched together so as to maintain consistency. From this omnidirectional image 71, the user 15 can display an image within the range surrounded by the frame of the display area 72, for example, on the display of the television receiver 14 and watch it.

Further, the user 15 is the remote commander 2
By operating the cursor 21 (see FIG. 7), the display area 72 displayed on the television receiver 14 can be arbitrarily moved in the directions indicated by arrows A to D in the figure. .

Further, the user 15 is the remote commander 2
By operating the predetermined button installed in 21,
The display area 72 can be enlarged and reduced. For example, in FIG. 4, the upper and lower edges of the display area are set to be narrower than the upper and lower edges of the omnidirectional image 71, but the upper and lower edges of the display area are set as the upper and lower edges of the omnidirectional image 71. Can be matched. in this case,
A wider range of images are displayed on the television receiver 14. On the contrary, the user 15 is the remote commander 2
By operating the predetermined button installed in 21,
The area of the display region 72 shown in FIG. 4 can be made smaller. In this case, the television receiver 14
The narrower area is zoomed up and displayed.

As described above, as shown in FIG. 4, the user 15 selects the remote commander 2 from the omnidirectional image 71.
By 21, the desired display area 72 can be cut out and displayed on the display of the television receiver 14.

Next, FIG. 5 shows an example of a detailed configuration inside the broadcasting station 11. As shown in FIG. 5, each of the cameras 51-1 to 51-8 forming the omnidirectional camera 31 includes
VTR (Video tape Recorder) 102-1 to 10 respectively
It is connected to 2-8. VTR 102-1 to 102-
8 records the video streams photographed by the cameras 51-1 to 51-8, respectively. Also, VTR102
A microphone 101 is also connected to -1, and VT
The R 102-1 also records the audio input from the microphone 101 together with the video stream.

Further, the cameras 32-1 to 32-5 have
VTRs 102-9 to 102-13 are respectively connected. The VTRs 102-9 to 102-13 are respectively
The video stream captured by the cameras 32-1 to 32-5 is recorded.

The video stream recorded by the VTRs 102-1 to 102-13 is sent to the switch 12 of the broadcasting device 33.
1 to the computer 122. At that time, the switching device 121 is operated by the VTRs 102-1 to 102-13.
By sequentially switching the connection with, each VTR102
-1 to 102-13 supplies the video stream to the computer 122.

The computer 122 video-captures the video data input from the VTRs 102-1 to 102-13 via the switch 121, and, for example, Window
Creates image data in the bitmap (BMP) file format, which is a standard image data storage file format for s (registered trademark) and OS / 2. Then, the computer 122
Among the image data converted into the bitmap file, the data corresponding to the image captured by the omnidirectional camera 31 is subjected to rotation processing, inversion processing, etc., and then subjected to predetermined processing to be described later to display the image. Laminate and generate omnidirectional image data.

Further, the computer 122 uses the camera 32-- among the image data converted into the bit map file.
A coordinate correspondence table that represents the correspondence between the image data corresponding to the images captured by 1 to 32-5 and the omnidirectional image data is created, and BML (Broadcast markup Language) is created.
It is supplied to the input device 125.

Further, the computer 122 generates a distribution video frame in which the omnidirectional video data and the image data corresponding to the images photographed by the cameras 32-1 to 32-5 are arranged at a predetermined position, and the MPEG for MPEG is generated. (Moving Pic
ture Experts Group) to the encoder 123.

The MPEG encoder 123 is the computer 1
The HD (High De
finition) -A video stream is created by performing an encoding process in the MPEG system and supplied to the multiplexer (MUX) 126.

A PSI (Program Service Information) input device 124 sets a flag indicating that the distribution video frame is a viewpoint free-form omnidirectional video and transmits it to the multiplexer 126. In existing satellite broadcasting, since there is PSI data that defines the type of broadcasted data, a flag or the like indicating that the broadcast program is a 360-degree omnidirectional video can be added thereto. Receiver 1
On the 3rd side, the cursor key on the remote commander 221 is activated by determining this flag,
Viewing of free-viewpoint omnidirectional images can be permitted. A portion of the transmission control signal standardized by ISO / IEC13818-1 is called PSI. PSI data consists of four types of tables (PAT, CAT, PMT, NIT) and is transmitted in section format.

The BML input device 125 supplies to the multiplexer 126 the content for data broadcasting described in the BML format which has been edited in advance. The BML content includes a script describing a movement rule of the omnidirectional video display area 72 and a scaling rule. The scaling rule includes a coordinate correspondence table supplied from the computer 122. That is, the computer 122
Thus, when the coordinate correspondence table is provided, the BML input device 125 creates a script that describes the scaling rule including the coordinate correspondence table, and creates BML content including the script. The created BML content is supplied to the multiplexer 126.

The multiplexer 126 receives the video stream encoded by the HD-MPEG system supplied from the MPEG encoder 123, the PSI data supplied from the PSI input device 124, and the BML content supplied from the BML input device 125. The multiplexed data is supplied to the transmission unit (TX) 127.

The transmitting section 127 modulates and A / D-converts the video stream in which the PSI data and the BML content are supplied, which is supplied from the multiplexer 126, up-converts the baseband signal into a satellite wave, and Is sent. The sent video stream is received by the receiving device 13 via the artificial satellite 12.

Next, FIG. 6 is a block diagram showing an example of the internal structure of the computer 122. In FIG. 6, the CPU
The 151 executes various processes according to the program stored in the ROM 152 or the program loaded from the storage unit 161 to the RAM 153. The RAM 153 also appropriately stores data necessary for the CPU 151 to execute various processes.

CPU 151, ROM 152, and RAM 153
Are mutually connected via a bus 154. An input / output interface 155 is also connected to the bus 154.

The input / output interface 155 includes an operation unit 156 composed of a plurality of buttons for receiving an operation input from a user, a voice output unit 157 for outputting a voice, a CRT (Cathode-Ray Tube), an LCD (Liquid). Cryst
via a display 158 such as an al display) and a switching device 121, a predetermined data is input to an input unit 159 that receives supply of video data from the VTRs 102-1 to 102-13, an MPEG encoder 123, a BML input device 125, and the like. An output unit 160 for outputting, a storage unit 161 including a hard disk, and a communication unit 162 including a modem and a terminal adapter are connected. The communication unit 162 performs communication processing via a network including the Internet.

The input / output interface 155 also has
A drive 163 is connected as necessary, a magnetic disk 181, an optical disk 182, a magneto-optical disk 183, a semiconductor memory 184, or the like is appropriately mounted, and a computer program read from them is stored in the storage unit 161 as necessary. Installed.

Next, FIG. 7 is a block diagram showing an example of the internal structure of the receiving device 13.

The control unit 201 receives a command signal input from the operation unit 202 or the infrared receiving unit 203, or
The entire processing of the receiving device 13 is controlled according to a program stored in advance. For example, the control unit 201 notifies the BML engine 207 of a command received from the remote commander 221 that is related to movement of the display area 72 to be cut out from the omnidirectional image 71 and displayed, and enlargement / reduction.

The operation unit 202 is composed of a plurality of buttons and the like, receives an operation input from the user 15, and transmits a corresponding signal to the control unit 201.

The infrared receiver 203 receives the infrared signal transmitted by the remote commander 221, and sends a corresponding signal to the controller 201.

The receiver (RX) 204 receives satellite waves from the artificial satellite 12, down-converts the satellite waves into baseband signals, and performs A / D conversion and demodulation processing. Receiver 2
04 supplies the demodulated data to the frame buffer 205. The frame buffer 205 stores the data supplied from the reception unit 204, and supplies the data to the demultiplexer (DEMUX) 206 at a predetermined stage.

The demultiplexer 206 separates the data supplied from the frame buffer 205 into an MPEG system video stream and BML content, and the BML content is BM.
It is supplied to the L engine 207 and the video stream is supplied to the MPEG decoder 208.

The BML engine 207 receives the BML content supplied from the demultiplexer 206, and executes the processing corresponding to the BML content. The BML engine 207 analyzes, for example, BML content for data broadcasting, and superimposes display data obtained from the BML content on the omnidirectional video frame in the frame buffer 209. In addition, it analyzes the omnidirectional video movement rules and scaling rules described as scripts. The BML engine 207 uses the omnidirectional image 7 based on the notification from the control unit 201, for example.
The process of moving the display area 72 to be cut out from 1 and displayed, and the scaling is executed. Further, the BML engine 207 executes zoom processing described later.

The MPEG decoder 208 MPEG-decodes the video stream supplied from the demultiplexer 206 to reproduce the original omnidirectional video frame, and supplies this to the frame buffer 209.

The frame buffer 209 stores the omnidirectional video frame supplied from the MPEG decoder 208 and supplies it to the clipping driver 210 at a predetermined timing.

The clipping driver 210 extracts the omnidirectional video frame stored in the frame buffer 209 from the omnidirectional video frame.
The angle of view designated to be displayed by the BML engine 207 is cut out and displayed on the display of the television receiver 14.

The remote commander 221 is provided with a plurality of buttons for receiving inputs of various operations from the user and a cursor key for receiving movement of the display area 72 in the omnidirectional image 71. The buttons include a button for accepting an instruction for enlarging or reducing the display area 72 from the user. The remote commander 221 outputs an infrared signal corresponding to the operation input to the button and the cursor key.

Next, referring to the flow chart of FIG.
The broadcast processing of the broadcast device 33 will be described.

In step S11, the broadcasting device 33
Executes omnidirectional video material data processing. Here, the process of step S11 of FIG. 8 will be described in detail with reference to the flowchart of FIG. The process of the flowchart of FIG. 9 is performed by the CPU 15 of the computer 122.
It is executed by 1.

In step S31 of FIG. 9, the CPU 151
Stores a value n = 1 indicating the number of cameras in the RAM 153. In step S32, the CPU 151 stores m = 1 indicating the number of frames. In step S33, the CPU 151 performs capturing on the mth frame image of the nth camera.

In step S34, the CPU 151
Lens distortion correction is performed on the bitmap video frame captured in step S33 based on the distortion parameter of the nth camera. Then, in step S35, the CPU 151 rotates the lens distortion-corrected bitmap image frame by 90 degrees as shown in FIG. 10, performs mirror inversion in step S36, and in step S37, mirror inverts in step S36. The stored image data is stored in the storage unit 161.

In step S38, the CPU 151
Increment m by 1 and in step S39, the CPU 151 determines whether or not the frame number m has reached the maximum frame number M. If the frame number m has not reached the maximum frame number M, the process proceeds to step S33. Returning to step S33, the processing for processing the video frame data is repeated. When the CPU 151 determines in step S39 that the frame number m has reached the maximum frame number M, the process proceeds to step S40, and the CPU 151 increments n by 1.

In step S41, the CPU 151
It is determined whether or not n has reached 8. If n has not reached 8, the process returns to step S32 and the above-described processing from step S32 is repeated for the video frame of the next camera. In step S41, the CPU 151 determines that n is 8
If it is determined that the value has reached, the process proceeds to step S42. In step S42, the CPU 151 saves the data of all eight cameras 51-1 to 51-8, and ends the processing shown in FIG.

Thereafter, the process proceeds to step S12 in FIG. In step S12, the computer 122
Executes a distribution omnidirectional video frame creation process. In step S12, the CPU 151 of the computer 122 combines the eight synchronized frames of the cameras 51-1 to 51-8 as shown in FIG. 11 to generate one frame. At this time, as the configuration of the omnidirectional camera 31, by arranging the video frames on the cylindrical side surface in the order in which the cameras 51-1 to 51-8 are installed (see FIG. 3), the bonded video frames are , 360 ° omnidirectional images captured by the omnidirectional camera 31 can be displayed.

When such an omnidirectional video frame is distributed by terrestrial broadcasting, satellite broadcasting, cable television broadcasting, high-definition broadcasting, etc., on the receiving device 13 side, the user 15 selects from the received omnidirectional video frames. By cutting out and displaying the specified angle of view, any angle of view can be seamlessly selected from the omnidirectional scene without being aware of which camera 51-1 to 51-8 originally captured the image. You can watch.

For example, in the omnidirectional video frame shown in FIG. 11, when the angle of view F1 is designated by the user 15, the video frame corresponding to this is cut out and displayed on the receiving device 13, but the camera 51 is displayed. -1 and the video frame captured by the camera 51-2 are connected to each other so that the user 15 cannot identify them. Similarly, even when the angle of view is moved to F2, the video frame corresponding to this is cut out and displayed on the receiving device 13, but the camera 51
-3 and the video frame captured by the camera 51-4 are connected to each other, the user cannot identify them.

When a viewpoint free-form video frame such as an omnidirectional video is distributed by terrestrial broadcasting, satellite broadcasting, cable television broadcasting, high-definition broadcasting, etc., since the video frame has an enormous amount of data, for example, HD-MPEG2 Image compression is performed according to the method. At this time, it is necessary to make the aspect ratio closer to the image compression method of HD-MPEG2 because of the request to fit the omnidirectional video in the size of HD-MPEG2.

For this reason, as shown in FIG. 11, an omnidirectional image originally formed by arranging the image frames of the cameras 51-1 to 51-8 on the cylindrical side surface is divided into two equal parts, and the latter half is divided into two parts. The frame-divided part of is arranged below the first half of the divided frame. As a result of such frame division and rearrangement, the camera 51-1 and the camera 5
The video frames 1-8 and the video frames of the cameras 51-4 and 51-5 are essentially continuous, but on the omnidirectional video frame that is data-distributed, as shown in FIG.
As shown in FIG. 1, they are arranged at distant positions.

In such a case, when the angle of view of the omnidirectional image is moved on the receiving device 13 side, the camera 51-8
It is difficult to seamlessly switch the area extending over the video frame of the camera 51-1 and the area extending over the video frame of the camera 51-4 and the camera 51-5.

Therefore, in the present embodiment, when dividing and rearranging the omnidirectional video frame having a cylindrical side surface, the video frames of the cameras adjacent to one of the divided edges are excessively pasted together. I chose That is, near the both edges of the omnidirectional video, the video frames of the attached cameras are rearranged so that they overlap.

In FIG. 12, when dividing and rearranging a cylindrical side-view omnidirectional video frame, the video frames of the cameras adjacent to one of the divided edges are superposed together. 3 shows an omnidirectional video frame.

In the example shown in the figure, the video frame of the camera 51-8 adjacent to the camera 51-1 and the video frame of the camera 51-7 adjacent to the camera 51-8 are provided at the head of the first half omnidirectional video frame divided portion. And the camera 5 at the beginning of the omnidirectional video frame division in the latter half.
Camera 51-4 adjacent to 1-5, and camera 51-
The video frames of the camera 51-3 adjacent to No. 4 are sequentially laminated.

Therefore, in the illustrated omnidirectional video frame,
Video frames of the camera 51-3 and the camera 51-4 overlap in the first half and the second half, and
Video frames of -7 and the camera 51-8 are duplicated in the first half and the second half.

Next, with reference to the flow chart of FIG. 13, the processing of step S12 of FIG. 8, that is, the distribution omnidirectional video frame creation processing will be described. The process shown in the flowchart of FIG. 13 is executed by the CPU 151 of the computer 122.

In step S61, the CPU 151
The initial value 1 is substituted for the variable n. In step S62, the CPU 151 reads the video data of the mth frame of the nth camera from the video data stored in the storage unit 161 by the processing of step S11 of FIG.

In step S63, the CPU 151
The image data of the mth frame of the nth camera read in step S62 is pasted on the side surface of the cylinder in the three-dimensional coordinate system. This is realized by performing a plane curved surface conversion.

In step S64, the CPU 151
The image data pasted on the side surface of the cylinder in step S63 is arranged according to the camera direction.

In step S65, the CPU 151
Increment n by 1. In step S66, the CPU 151 determines whether n has reached 8, and n
Is less than 8, the process returns to step S62, and the process of step S62 and subsequent steps described above is repeatedly executed for the video frame of the next camera.

In step S65, the CPU 151
n has reached 8 (all eight cameras 51-1 to 51-
8, it is determined that the process of attaching the video frame to the side surface of the cylinder is completed), the process proceeds to step S67.

In step S67, the CPU 151
The cylinder radius and other parameters are calculated while adjusting the matching between adjacent images.

In step S68, the CPU 151
Cut the side of the cylinder and transform it into a rectangular plane. In step S69, the CPU 151 divides the rectangular plane at half of the horizontal direction (arrangement direction of the camera 51),
The right half is placed below the left half.

Subsequently, in step S70 of FIG. 14, the CPU 151 duplicates the lower right quarter area of the arranged rectangular plane and arranges it at the left end of the upper left portion, and at the same time, the upper right quarter of the rectangular plane. Duplicate the area and place it on the left edge of the lower left.

Next, in step S71, the CPU 15
1 executes the matching process. Here, the matching process will be described with reference to the flowchart of FIG. 15 and FIG.

In step S91 of FIG. 15, the CPU1
51 is the omnidirectional image 25 created in step S70 in the center of the display 158, as shown in FIG.
1 is displayed, and a zoom image 2 captured by the cameras 32-1 to 32-5 is displayed below the display.
52-1 to 252-5 are displayed. The zoom image 252 displayed individually is the omnidirectional image 251.
It is based on the frame captured by the camera 32 at the same timing as.

In step S92, the CPU 151
The selection of one zoom image is received from the zoom images 252-1 to 252-5. A person who performs the matching process (hereinafter referred to as an editor) can select a zoom image by operating the operation unit 156.

In step S93, the CPU 151
The designation of the area in the omnidirectional image 251 corresponding to the zoom image selected in step S92 is accepted. The editor operates step S9 by operating the operation unit 156.
Omnidirectional image 25 corresponding to the zoom image selected in 2
The area in 1 can be designated.

In step S94, the CPU 151
For the zoom in the omnidirectional image 251, the correlation between the zoom image 252 and the omnidirectional image 251 is highest around the area in the omnidirectional image 251 corresponding to the zoom image whose designation is received in step S93. The coordinates of the image 252 are searched and the coordinates of the zoom image 252 in the omnidirectional image 251 are specified.

For example, in FIG. 16, the editor selects the zoom image 252-3 in step S92, and in step S93 the omnidirectional image in which the same landscape as the zoom image 252-3 is displayed. Specify the position in 251
This can be done by using the frame 253 displayed on the display 158. When the editor specifies the approximate position of the zoom image 252-3 in the omnidirectional image 251, the CPU 151 determines in step S93.
By searching the periphery of the frame 253 designated by, the omnidirectional image 251 and the zoom image 252-3 have the highest image correlation. The coordinates are specified by the pattern matching process.

By the processing of step S94, the omnidirectional images 251 of all the zoom images 252-1 to 252-5.
When the inside coordinates are specified, the process proceeds to step S in FIG.
Proceed to 72.

In step S72, the CPU 151
A coordinate correspondence table as shown in FIG. 17 is created.
The file size of the coordinate correspondence table shown in FIG. 17 is “header 301 (48 bits) + table unit 302 (96 bits) × table unit number (n)”.
Is. It requires 24 bits for the time code and 24 bits for the number of zooming points. Number of zooming points 31
2 is the zoom image 25 in the omnidirectional image 251.
2 indicates the number of corresponding areas.

That is, FIG. 18 shows an omnidirectional image 251.
The area of the zoom image 252 corresponding to is indicated by the frames a to i. As shown in FIG. 18, for example, in the omnidirectional image 251, the image in the frame indicated by c and e corresponds to the zoom image 252-1 (hereinafter, this image is also referred to as C1). . Further, in the omnidirectional video image 251, the image in the frame indicated by d and f is the zoom image 252-2 (hereinafter, this image is C2.
Also referred to as). In the omnidirectional image 251, the image in the frame indicated by g is the zoom image 2
52-3 (hereinafter, this image is also referred to as C3) corresponds. In the omnidirectional image 251, the image in the frame indicated by a and h corresponds to the zoom image 252-4 (hereinafter, this image is also referred to as C4). Further, in the omnidirectional image 251, the image in the frame indicated by b and i corresponds to the zoom image 252-5 (hereinafter, this image is also referred to as C5).

As described above, in the omnidirectional image 251,
9 from a to i corresponding to the zoom images C1 to C5
When there are two areas, “9” is recorded in the number of zooming points 312 in FIG. The nine areas from a to i in FIG. 18 and the coordinate correspondences of the zoom images C1 to C5 are recorded in the table units 302-1 to 302-9 in FIG.

That is, for example, the coordinate correspondence between the area a in the omnidirectional image 251 and the zoom image C4 is recorded in the table unit 302-1. Table unit 3
In 02-1, the coordinates (X, Y) of the upper left corner of the area a in the omnidirectional image 251 are recorded at the master area start point 321. Each recording of X and Y requires 12 bits (X and Y below)
Is also the same). In the master area endpoint 322, the coordinates (X, Y) of the lower right corner of the area a in the omnidirectional image 251 are recorded.

In the table unit 302-1, the zooming area start point 323 is
Zoom image C corresponding to area a of omnidirectional image 251
The coordinates (X, Y) of the upper left corner of 4 are recorded. In addition, in the table unit 302-1, the zooming area
At the end point 324, the coordinates (X, X, at the lower right corner of the zoom image C4 corresponding to the area a of the omnidirectional image 251 are displayed.
Y) is recorded.

On the table unit 302-2, the coordinate correspondence between the area b in the omnidirectional video image 251 and the zoom image C5 is recorded. Although detailed recording contents are not shown, they are the same as the case of the table unit 302-1 described above. Hereinafter, similarly, the table unit 302-
3, the coordinate correspondence between the area c in the omnidirectional video 251 and the zoom image C1 is recorded.
In -4, the coordinate correspondence between the area d in the omnidirectional image 251 and the zoom image C2 is recorded, and the table unit 30
In 2-5, the coordinate correspondence between the area e in the omnidirectional image 251 and the zoom image C1 is recorded, and the table unit 3
In 02-6, the coordinate correspondence between the area f in the omnidirectional video 251 and the zoom image C2 is recorded, and in the table unit 302-7, the area g in the omnidirectional video 251 and the zoom image C3 are recorded. The coordinate correspondence is recorded, the coordinate correspondence between the area h in the omnidirectional image 251 and the zoom image C4 is recorded in the table unit 302-8, and the area in the omnidirectional image 251 is recorded in the table unit 302-9. The coordinate correspondence between i and the zoom image C5 is recorded.

The CPU 151 creates the coordinate correspondence table as described above in step S72 of FIG. Next, in step S73, the CPU 151 stores the coordinate correspondence table created in step S72 in the storage unit 161 in association with the frame. In step S74, the CPU 151 displays the omnidirectional image 251 at 1386.
The size is reduced to 684 pixels.

At step S75, the CPU 151
An omnidirectional image 251, whose size has been reduced in step S74,
And zoom images C1 to C5 are arranged as shown in FIG. 19, and the size of the entire frame is 1920 × 1.
The size is 080 pixels. FIG. 19 shows a video frame for distribution. As shown in FIG. 19, in the distribution video frame, an omnidirectional video 251 is arranged on the left side, and a zoom image C1, a zoom image C2, and a zoom image C3 are displayed on the right side in order from the top. The zoom image C5 and the zoom image C2 are arranged to the left of the zoom image C3.

In step S76, the storage unit 161 stores various geometrical parameters created by the processes of steps S62 to S75 shown in the flowcharts of FIGS. Above,
The distribution omnidirectional video frame creation process ends. Then, the process proceeds to step S13 in FIG.

In step S13, the computer 1
The CPU 151 of the 22 renders the omnidirectional video frame made into the bit map serial number file by the omnidirectional video frame creation process for distribution, and HD-SDI (Serial D
signal is output to the MPEG encoder 123.

In step S14, the MPEG encoder 123 encodes the video stream supplied from the computer 122 by the HD-MPEG system and supplies it to the multiplexer 126.

In step S15, the PSI input device 1
Reference numeral 24 sets a flag indicating that the video stream is a viewpoint free-form omnidirectional video.

In step S16, the computer 1
22 supplies the coordinate correspondence table created in step S72 of FIG. 14 to the BML input device 125. The BML input device 125 executes a predetermined process based on the supplied coordinate correspondence table, creates BML content including the coordinate correspondence table, and supplies the BML content to the multiplexer 126.

In step S17, the multiplexer 126 multiplexes the video stream supplied from the MPEG encoder 123, the PSI data input from the PSI input device 124, and the BML content supplied from the BML input device 125, and the transmission unit Supply to 127. Transmitter 1
When the video stream in which the PSI data and the BML content are multiplexed is supplied from the multiplexer 126, the reference numeral 27 modulates this, performs A / D conversion, and up-converts the baseband signal into a satellite wave, and then transmits the satellite wave.

This is the end of the description of the broadcasting process of the broadcasting device 33.

Next, referring to the flowchart of FIG. 20,
The display control process of the receiving device 13 will be described.

In step S121, the receiving section 204
Receives the satellite wave from the artificial satellite 12, down-converts it to a baseband signal, A / D converts it, demodulates it,
It is stored in the frame buffer 205.

In step S122, the control unit 201
Extracts PSI data flag data indicating that the received video stream is a viewpoint free-form omnidirectional video. In step S123, the control unit 201 determines whether the received data is a viewpoint free-form video stream based on the flag data extracted in step S122, and when the received data is a viewpoint free-form video stream. In step S124, the control unit 201 controls the demultiplexer 206, the BML engine 207, the MPEG decoder 208, and the cutout driver 210 to display a viewpoint-free image on the display of the television receiver 14. Run.

That is, the data stored in the frame buffer 205 is sent to the BML by the demultiplexer 206.
The content and the MPEG stream are separated, the BML content is supplied to the BML engine 207, and the MPEG stream is supplied to the MPEG decoder 208. BML engine 20
7 is a process of moving and enlarging / reducing the display area 72 which is cut out from the omnidirectional image 71 and displayed according to the BML content.
(Zoom processing) is executed.

The MPEG decoder 208 outputs the MPEG stream.
MPEG decoding is performed to reproduce the original viewpoint free-form video frame, and this is stored in the frame buffer 209.

The clipping driver 210 clips the image in the display area 72 from the viewpoint free-form video frame stored in the frame buffer 209 in accordance with the BML script executed by the BML engine 207 and supplies it to the television receiver 14. . The television receiver 14 displays the supplied image on the display.

If the control unit 201 determines in step S123 of FIG. 20 that the received data is not a viewpoint free-form video stream, the process proceeds to step S125, and the received data is treated as normal broadcast data by MPEG.
The video stream is decoded and the decoded video stream is supplied to the television receiver 14. Television receiver 14
Displays the supplied image on the display.

In step S126, the control unit 201
Determines whether or not an instruction to end viewing is input from the remote commander 221 (or the operation unit 202) while executing the processing in step S124 or step S125, until the instruction to end viewing is input. Step S1
24 or the viewing process of step S125 is executed. When an instruction to end viewing is input from the remote commander 221 (or the operation unit 202), the control unit 201
Stops the supply of images to the television receiver 14,
The process ends.

Next, in step S124 of FIG. 20, referring to the flowchart of FIG. 21, for the zoom process of the receiving device 13 when the receiving device 13 causes the television receiver 14 to display a free-viewpoint image. While explaining.

The control unit 201 always monitors the signal received by the infrared receiving unit 203 from the remote commander 221, and inputs an instruction for enlarging or reducing an image to be displayed on the display of the television receiver 14 from the remote commander 221. It keeps judging whether it was done.

Therefore, in step S201, the control unit 201 continues to determine whether or not an image enlargement instruction has been input via the remote commander 221, and if an image enlargement instruction has been input, step S202. Then, the BML engine 207 is notified of the image enlargement command from the remote commander 221. BML engine 207
First, based on the notification from the control unit 201, it is first determined whether 80% of the display area instructed to be enlarged includes the display area c or the display area e.

As shown in FIG. 18, the display area c and the display area e are areas corresponding to the same image, and the display area c and the display area e are included in 80% of the display area instructed to be enlarged. If it is included,
The process proceeds to step S203, and BML engine 207
Instructs the cutout driver 210 to cut out the zoom image C1. The cutting driver 210 is a BM
Image C1 for zooming according to the command from the L engine 207
Is cut out and supplied to the television receiver 14. The television receiver 14 displays the supplied zoom image C1.
Is displayed on the display. Then, a process progresses to step S213.

In step S202, when the BML engine 207 determines that 80% of the display area instructed to be enlarged does not include the display area c or the display area e, the process proceeds to step S204.

In step S204, the BML engine 207 determines whether 80% of the display area instructed to be enlarged includes the display area d or the display area f.

As shown in FIG. 18, the display area d and the display area f are areas corresponding to the same image, and the display area d and the display area f are included in 80% of the display area instructed to be enlarged. If it is included,
The process proceeds to step S205, and BML engine 207
Instructs the cutout driver 210 to cut out the zoom image C2. The cutting driver 210 is a BM
Image C2 for zooming according to the command from the L engine 207
Is cut out and supplied to the television receiver 14. The television receiver 14 displays the supplied zoom image C2.
Is displayed on the display. Then, a process progresses to step S213.

When the BML engine 207 determines in step S204 that 80% of the display area instructed to be enlarged does not include the display area d or the display area f, the process proceeds to step S206.

In step S206, the BML engine 207 determines whether the display area g is included in 80% of the display area instructed to be enlarged.

If the display area g is included in 80% of the display area instructed to be enlarged, the process proceeds to step S20.
7, the BML engine 207 determines the cutout driver 2
10 is instructed to cut out the zoom image C3.
The cut-out driver 210 cuts out the zoom image C3 according to a command from the BML engine 207 and supplies the zoom image C3 to the television receiver 14. Television receiver 14
Displays the supplied zoom image C3 on the display. Then, a process progresses to step S213.

If the BML engine 207 determines in step S206 that the display area g is not included in 80% of the display area instructed to be enlarged, the process proceeds to step S208.

In step S208, the BML engine 207 determines whether 80% of the display area instructed to be enlarged includes the display area a or the display area h.

As shown in FIG. 18, the display area a and the display area h are areas corresponding to the same image, and the display area a and the display area h are included in 80% of the display area instructed to be enlarged. If it is included,
The process proceeds to step S209, and BML engine 207
Instructs the cutout driver 210 to cut out the zoom image C4. The cutting driver 210 is a BM
Image C4 for zooming according to the command from the L engine 207
Is cut out and supplied to the television receiver 14. The television receiver 14 displays the supplied zoom image C4.
Is displayed on the display. Then, a process progresses to step S213.

If the BML engine 207 determines in step S208 that 80% of the display area instructed to be expanded does not include the display area a or the display area h, the process proceeds to step S210.

In step S210, the BML engine 207 determines whether 80% of the display area instructed to be enlarged includes the display area b or the display area i.

As shown in FIG. 18, the display area b and the display area i are areas corresponding to the same image, and the display area b and the display area i are included in 80% of the display area instructed to be enlarged. If it is included,
The process proceeds to step S211, and the BML engine 207
Instructs the cutout driver 210 to cut out the zoom image C5. The cutting driver 210 is a BM
Image C5 for zooming according to the command from the L engine 207
Is cut out and supplied to the television receiver 14. The television receiver 14 displays the supplied zoom image C5.
Is displayed on the display. Then, a process progresses to step S213.

In step S210, if the BML engine 207 determines that 80% of the display area instructed to be enlarged does not include the display area b or the display area i, the process proceeds to step S212.

In step S212, the BML engine 207 instructs the cut-out driver 210 to cut out and display the area instructed to be enlarged from the omnidirectional image 251. The cutout driver 210 is
Omnidirectional image 2 according to the command from BML engine 207
The image of the area instructed to be enlarged in 51 is cut out and supplied to the television receiver 14. The television receiver 14 displays the supplied image on the display. Then, a process progresses to step S213.

In step S213, the control unit 201
If the signal input from the infrared receiver 203 includes a command to change the display area 72, the BML engine 207 is notified of this. The BML engine 207 determines whether or not an instruction to move the display area 72 to be displayed on the display of the television receiver 14 has been input from the remote commander 221 based on the notification supplied from the control unit 201. When it is determined that the instruction to move the display area 72 to be displayed on the display of the John receiver 14 is input, the process returns to step S202, and the processes of step S202 and subsequent steps described above are repeated. If the BML engine 207 determines in step S213 that the remote commander 221 has not input an instruction to move the display area 72 to be displayed on the display of the television receiver 14, the process proceeds to step S214.

In step S214, the BML engine 207 determines from the remote commander 221 whether or not the enlargement of the display area 72 to be displayed on the display of the television receiver 14 has been canceled. If the enlargement command has been canceled, The process returns to step S201, and step S2
The processing from 01 onward is repeated. When the BML engine 207 determines in step S214 that the enlargement command has not been released, the processing returns to step S213,
The processing from step S213 onward is repeated.

The zoom process of the receiving device 13 is executed as described above.

In the above description with reference to FIG. 21, it is determined whether 80% of the display area instructed to be enlarged includes any of the display areas a to i. However, the criterion for judgment is not limited to 80%. The criterion of determination can be set to an arbitrary value by the manufacturer of the receiving device 13. The user 15 may optionally make this setting.

As described above, when the zoom image 252 corresponding to the display area instructed to be enlarged exists, the omnidirectional image 251 to the zoom image 252 are displayed.
Further, by switching the display, it becomes possible to display a higher resolution image.

In particular, as shown in FIG. 18, by preparing a zoom image 252 corresponding to a point such as a person's face where it is easy to receive an instruction for enlargement, a high rate of user 15 can be obtained. It becomes possible to display the zoom image 252 corresponding to the point for which the enlargement instruction is given. Therefore, the user 15 can view a clearer image at a higher rate.

In the above description, the omnidirectional image is taken as an example, but this does not mean that the present invention is limited to the omnidirectional image. For example, if a zoom image corresponding to one or more ranges in a conventional image that is not an omnidirectional image is prepared in advance and the user inputs an instruction to enlarge the desired range, the range is expanded. The zoom image corresponding to may be displayed.

In the above description, the receiving device 1
3 and the television receiver 14 have been described as different devices, but the receiver 13 and the television receiver 14 can of course be integrally configured.

Further, in the above description, the broadcasting device 3
Although it has been described that the computer 122 executes a part of the processing executed by the computer 3, the computer 122 may of course execute the entire processing of the broadcasting device 33.

Also, in the above description, the artificial satellite 1
Although the description has been made by taking the broadcasting via 2 as an example, the distribution method can be any method other than the satellite broadcasting. For example, the distribution may be performed using a network such as terrestrial broadcasting, cable television broadcasting, or the Internet. Further, the video data may be recorded in a recording medium such as a DVD and sold or distributed.

The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processes is executed by software, a program (for example, CPU 151) in which a program configuring the software is installed in dedicated hardware or various programs are installed to perform various functions. It is installed from a program storage medium into a general-purpose personal computer or the like that can be executed.

A program storage medium for storing a program installed in a computer and made executable by the computer is, for example, the magnetic disk 1.
81 (including flexible disk), optical disk 18
2 (including CD-ROM (Compact Disk-Read Only Memory) and DVD), a magneto-optical disk 183, a package medium including a semiconductor memory 184, and the like. Storing the program in the program storage medium stores the program in a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via an interface such as a router or a modem as necessary. Be seen.

Further, in the present specification, the steps for writing the program provided by the medium may be performed in parallel in the order described, not necessarily in the time series. Alternatively, it also includes processes that are individually executed.

In the present specification, the system means
It represents the entire apparatus composed of a plurality of devices.

[0155]

As described above, according to the present invention, it is possible to enlarge and display an image.

Further, according to the present invention, it is possible to display a higher resolution video when the video is enlarged and displayed.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of an embodiment of a service providing system to which the present invention is applied.

FIG. 2 is a diagram showing an example of arrangement of omnidirectional cameras and other cameras.

FIG. 3 is a diagram showing an external configuration of an omnidirectional camera.

FIG. 4 is a diagram illustrating a display example of an omnidirectional image.

FIG. 5 is a block diagram showing a configuration example of an omnidirectional camera, a camera, and a broadcasting device.

FIG. 6 is a block diagram illustrating a configuration example of a computer.

FIG. 7 is a block diagram illustrating a configuration example of a receiving device.

FIG. 8 is a flowchart illustrating a broadcast process of a broadcast device.

9 is a flowchart illustrating in detail the process of step S11 of FIG.

FIG. 10 is a diagram illustrating image processing of a computer.

FIG. 11 is a diagram illustrating image processing of an omnidirectional image.

FIG. 12 is another diagram illustrating image processing of an omnidirectional image.

FIG. 13 is a flowchart illustrating in detail step S12 of FIG.

FIG. 14 is a diagram for explaining step S12 in FIG. 8 in detail.
It is a flowchart following 3.

FIG. 15 is a flowchart illustrating in detail step S71 of FIG.

FIG. 16 is a diagram showing an example of a screen displayed on a display of a computer.

FIG. 17 is a diagram illustrating a format of a coordinate correspondence table.

FIG. 18 is a diagram illustrating correspondence between an omnidirectional video image and a zoom image.

FIG. 19 is a diagram showing an example of a distribution video frame broadcast from a broadcasting device.

FIG. 20 is a flowchart illustrating a display control process of the receiving device.

FIG. 21 is a flowchart illustrating zoom processing of the reception device.

[Explanation of symbols]

11 broadcasting stations, 12 artificial satellites, 13 receivers,
14 television receiver, 31 omnidirectional camera,
32 cameras, 33 broadcasting equipment, 51 cameras,
52 reflector, 71 omnidirectional image, 72 display area, 101 microphone, 102 VTR, 12
1 switching device, 122 computer, 123 MPEG
Encoder, 124 PSI input device, 125 BML input device, 126 multiplexer, 127 transmission unit, 151 CPU, 201 control unit, 202 operation unit, 203 infrared reception unit, 204 reception unit, 2
05 frame buffer, 206 demultiplexer, 207 BML engine, 208 MPEG coder, 209 frame buffer, 210 clip driver, 221 remote commander

Claims (21)

[Claims] 1. A service providing system comprising a broadcasting device for broadcasting a video and a receiving device for receiving the broadcasting, wherein the broadcasting device comprises a first image taken by a first imaging device, and An acquisition unit for acquiring a second image captured by the second image capturing apparatus for enlarging and capturing a portion of the image capturing range of the first image capturing apparatus; and the acquisition unit for acquiring the second image for the first image. And a broadcasting unit configured to broadcast position information indicating the position of the second image, and the position information generated by the creating unit together with the first image and the second image. A receiving device for receiving a broadcast from the broadcasting device; a displaying device for displaying the first image included in the broadcasting received by the receiving device; and a displaying device. And a reception unit that receives an input of an instruction to enlarge and display a part of the first image that is being displayed, and an input of an instruction to enlarge and display a portion of the first image by the reception unit. In the case of the second area corresponding to the range in which the input of the instruction to enlarge the display is accepted.
Of the image is included in the broadcast based on the position information, the determining unit, the determining unit, the first corresponding to the range in which the input of the instruction to enlarge the display is accepted. A service providing system comprising: a control unit that causes the display unit to display the second image when it is determined that the second image is included in the broadcast. 2. A first image photographed by a first photographing device, and a second image photographed by a second photographing device for enlarging and photographing a part of a photographing range of the first photographing device. An acquisition unit that acquires position information, a creation unit that creates position information indicating a position of the second image with respect to the first image acquired by the acquisition unit, and the position information created by the creation unit, A broadcasting device comprising: a broadcasting unit that broadcasts together with the first image and the second image. 3. The broadcasting apparatus according to claim 2, wherein the first image capturing device is an omnidirectional camera, and the first image is an omnidirectional video image. 4. The broadcasting means broadcasts information indicating that the first image is the omnidirectional video together with the first image, the second image, and the position information. The broadcasting device according to claim 3. 5. The creating means is based on an accepting means for accepting an input of an operation for designating a position of the second image corresponding to the first image, and based on the operation accepted by the accepting means. Calculating means for calculating coordinates of the second image corresponding to the first image, and creating the position information in a predetermined format including the coordinates calculated by the calculating means. The broadcasting device according to claim 2. 6. The acquisition means acquires a plurality of the second images photographed by a plurality of the second photographing devices that magnify and photograph a part of the first image. The broadcasting device according to claim 2. 7. The broadcasting apparatus according to claim 6, wherein the creating unit creates the position information including a positional relationship of the plurality of second images with respect to the first image. 8. A first image photographed by a first photographing device and a second image photographed by a second photographing device for enlarging and photographing a part of a photographing range of the first photographing device. An acquisition step of acquiring a position information indicating a position of the second image with respect to the first image acquired by the processing of the acquisition step; A broadcast step of broadcasting position information together with the first image and the second image. 9. A first image photographed by a first photographing device, and a second image photographed by a second photographing device which magnifies and photographs a part of the photographing range of the first photographing device. An acquisition step of acquiring a position information indicating a position of the second image with respect to the first image acquired by the processing of the acquisition step; A recording medium having a computer-readable program recorded thereon, which comprises a step of broadcasting position information together with the first image and the second image. 10. A first image photographed by a first photographing device, and a second image photographed by a second photographing device for enlarging and photographing a part of a photographing range of the first photographing device.
An acquisition step of acquiring the image of the image, a creation step of creating position information indicating the position of the second image with respect to the first image acquired by the processing of the acquisition step, and a creation step of the processing of the creation step. And a broadcast step of broadcasting the position information together with the first image and the second image. 11. A first image, a second image which is an enlarged image of a part of the first image, and position information indicating a position of the second image with respect to the first image are included. Receiving means for receiving the broadcast, display means for displaying the first image included in the broadcast received by the receiving means, and a part of the first image displayed by the display means First accepting means for accepting an input of an instruction to be displayed; and, if the first accepting means accepts an instruction of an enlarged display of a part of the first image, an input of the instruction of an enlarged display. Is determined by the first determining means and the first determining means to determine whether or not the second image corresponding to the received range is included in the broadcast. Input of enlarged display instruction And a control unit that controls the display unit to display the second image when it is determined that the second image corresponding to the received range is included in the broadcast. A receiver characterized by. 12. The receiving device according to claim 11, wherein the first image is an omnidirectional image. 13. The broadcast includes information indicating that the first image is an omnidirectional video, and whether or not there is information indicating that the first image is the omnidirectional video,
The receiving device according to claim 12, further comprising a second determining unit that determines whether or not the first image is the omnidirectional image. 14. The control unit displays a part of the omnidirectional image on the display unit when the second determining unit determines that the first image is the omnidirectional image. The receiving device according to claim 13, wherein the receiving device is configured to: 15. The apparatus further comprises second receiving means for receiving an input of an instruction to move the range of the omnidirectional image displayed on the display means, wherein the control means receives the input by the second receiving means. 15. The receiving apparatus according to claim 14, wherein the range of the omnidirectional image displayed on the display unit is moved according to the instruction. 16. The first determination means is configured such that, in the first image, the range in which the input of the instruction to enlarge the display is accepted is a preset predetermined value in the second image. If the range of the ratio is included, the second number corresponding to the range in which the input of the instruction to enlarge the display is accepted.
The receiving device according to claim 11, wherein the image is determined to be included in the broadcast. 17. The broadcast received by the receiving means is
2. A plurality of said second images are included.
1. The receiving device according to 1. 18. When the first determination unit determines that the second image corresponding to the range in which the input of the instruction to enlarge the display is accepted is not included in the broadcast, The receiving device according to claim 11, wherein the control unit controls the display unit to display a part of the first image in an enlarged manner. 19. A first image, a second image which is an enlarged image of a part of the first image, and position information indicating a position of the second image with respect to the first image are included. A reception step of receiving a broadcast; a display step of displaying the first image included in the broadcast received by the processing of the reception step; and a first step displayed by the processing of the display step.
A receiving step of receiving an input of an instruction to enlarge and display a part of the image, and the processing of the receiving step causes the enlarged display when an input of an instruction to display a portion of the first image is received. A determination step of determining whether or not the second image corresponding to the range in which the input of the instruction is accepted is included in the broadcast based on the position information; When it is determined that the second image corresponding to the range in which the input of the instruction to display is accepted is included in the broadcast, a control step of controlling the second image to be displayed. Receiving method characterized by including. 20. A first image, a second image which is an enlarged image of a part of the first image, and position information indicating a position of the second image with respect to the first image are included. A reception control step of controlling reception of a broadcast, and a first control of displaying the first image included in the broadcast whose reception is controlled by the processing of the reception control step.
A display control step, a reception step of receiving an input of an instruction to enlarge and display a part of the first image whose display is controlled by the processing of the first display control step, and the processing of the reception step, When the input of the instruction to enlarge and display a part of the first image is accepted, whether the second image corresponding to the range in which the input of the instruction to enlarge and receive is accepted is included in the broadcast. The determination step of determining whether or not the position information is based on the position information, and the processing of the determination step includes the second image corresponding to the range in which the input of the instruction to enlarge the display is accepted, A second display control step of controlling the second image to be displayed when it is determined that the computer-readable process is performed. Recording medium grams is recorded. 21. A first image, a second image that is an enlarged image of a part of the first image, and position information indicating a position of the second image with respect to the first image are included. A reception control step of controlling reception of a broadcast, and a first control of displaying the first image included in the broadcast whose reception is controlled by the processing of the reception control step.
A display control step, a reception step of receiving an input of an instruction to enlarge and display a part of the first image whose display is controlled by the processing of the first display control step, and the processing of the reception step, When the input of the instruction to enlarge and display a part of the first image is accepted, whether the second image corresponding to the range in which the input of the instruction to enlarge and receive is accepted is included in the broadcast. The determination step of determining whether or not the position information is based on the position information, and the processing of the determination step includes the second image corresponding to the range in which the input of the instruction to enlarge the display is accepted, And a second display control step of controlling the second image to be displayed when it is determined that the second image is displayed.