JP2021078114A - Photographing device, distribution system, distribution method, and recording medium - Google Patents

Abstract

To simultaneously distribute image data of a plurality of streams by using a photographing device having limitation on the number of hardware encoders.SOLUTION: A photographing device 10 distributes image data of a plurality of streams, and comprises: a photographing processing section 15 (an example acquisition means) that acquires photographed image data photographed by an imaging unit 101 (an example of photographing means); a determination section 14 (an example of determination means) that determines a method for encoding the acquired photographed image data according to an encoding parameter for each of the streams; an encoding processing section 18 (an example of generation means) that generates streaming image data for each of the streams by using the determined encoding method; and a communication section 11 (an example of distribution means) that distributes the plurality of pieces of generated streaming image data.SELECTED DRAWING: Figure 2

Description

The present invention relates to a photographing device, a distribution system, a distribution method, and a recording medium.

In recent years, a technique capable of streaming and distributing a photographed image photographed by a photographing device using a distribution service on the Internet has become widespread. Further, there is also known a system capable of performing streaming distribution by directly connecting a photographing device to the Internet without going through a terminal such as a PC or a smartphone.

Further, in such streaming distribution, a technique for simultaneously distributing images of a plurality of streams having different image quality is known so that a stream to be received by a viewing terminal can be selected. In this case, the processing capacity for encoding the image data to be distributed is important in the device on the distribution side. For example, in Patent Document 1, the processing is allocated to the software encoder and the hardware encoder to allocate the processing to the video data. Is disclosed, and the contents of synthesizing the encoded data into one stream are disclosed.

However, the conventional method has a problem that it is difficult to simultaneously deliver image data of a plurality of streams because the encoding process for each stream cannot be performed in parallel in a device having a limited number of hardware encoders. was there.

In order to solve the above-mentioned problems, the invention according to claim 1 is an imaging device that distributes image data of a plurality of streams, and is an acquisition means for acquiring captured image data captured by the photographing means, and each of the streams. A determination means for determining the encoding method of the acquired captured image data according to the encoding parameters of the above, a generation means for generating streaming image data for each stream using the determined encoding method, and generation means. It is a photographing device including a distribution means for distributing a plurality of streaming image data.

According to the present invention, it is possible to simultaneously distribute image data of a plurality of streams by using a photographing device in which the number of hardware encoders is limited.

It is a figure which shows an example of the system configuration of the distribution system which concerns on 1st Embodiment. It is a figure which shows an outline example of the process of the distribution system which concerns on 1st Embodiment. It is a figure which shows an example of the hardware composition of the photographing apparatus which concerns on 1st Embodiment. It is a figure which shows an example of the hardware composition of the communication terminal which concerns on 1st Embodiment. It is a figure which shows an example of the functional structure of the distribution system which concerns on 1st Embodiment. It is a conceptual diagram which shows an example of the encoding parameter management table which concerns on 1st Embodiment. It is a sequence diagram which shows an example of the image distribution processing in the distribution system which concerns on 1st Embodiment. It is a flowchart which shows an example of the encoding process of the image data which concerns on 1st Embodiment. (A), (B) It is a conceptual diagram which shows another example of the encoding parameter management table which concerns on 1st Embodiment which concerns on 1st Embodiment. It is a flowchart which shows an example of the encoding process of the image data which concerns on the modification of 1st Embodiment. It is a conceptual diagram for demonstrating an example of the encoding parameter management table when the target encoder is changed in the photographing apparatus which concerns on the modification of 1st Embodiment. It is a figure which shows an example of the system configuration of the distribution system which concerns on 2nd Embodiment. It is a figure which shows an outline example of the process of the distribution system which concerns on 2nd Embodiment. It is a figure which shows an example of the functional structure of the distribution system which concerns on 2nd Embodiment. It is a sequence diagram which shows an example of the image distribution processing in the distribution system which concerns on 2nd Embodiment.

Hereinafter, modes for carrying out the invention will be described with reference to the drawings. In the description of the drawings, the same elements are designated by the same reference numerals, and duplicate description will be omitted.

● First embodiment ●
● System Configuration FIG. 1 is a diagram showing an example of the system configuration of the distribution system according to the first embodiment. The distribution system 1 shown in FIG. 1 is a system capable of streaming distribution of image data captured and acquired by the photographing device 10 to the communication terminal 50 used by the user.

As shown in FIG. 1, the distribution system 1 is composed of a photographing device 10 and a communication terminal 50. The photographing device 10 and the communication terminal 50 are communicably connected via the communication network 5. The communication network 5 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), or the like. The communication network 5 includes not only wired communication but also 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity (registered trademark)), and WiMAX (Worldwide Interoperability for Microwave). A network by wireless communication such as Access) or LTE (Long Term Evolution) may be included.

The photographing device 10 is a digital camera capable of photographing a subject and acquiring a photographed image. The photographing device 10 is, for example, a special digital camera for obtaining a spherical (360 °) panoramic image. The photographing device 10 may be a general digital camera (single-lens reflex camera, compact digital camera, surveillance camera, video (video) camera, etc.). The photographing device 10 performs streaming distribution of image data related to the captured image via a communication network 5 such as the Internet. The captured image may be a moving image or a still image, and may be both a moving image and a still image. Further, the captured image may include sound together with the image.

The communication terminal 50 is a terminal device such as a tablet terminal used by the user. The communication terminal 50 performs data communication with the photographing device 10 by accessing a communication network 5 such as the Internet. The communication terminal 50 can receive the streaming image data distributed from the photographing device 10. The communication terminal 50 is not limited to a tablet terminal, but is not limited to a tablet terminal, but is a PC (Personal Computer), a smartphone, a mobile phone, a wearable terminal, a game machine, a video conferencing (video conferencing) terminal, and an IWB (Interactive White Board: electronic capable of mutual communication). It may be a white board having a type of blackboard function), a telepresence robot, or the like.

● Outline Here, an outline of the processing of the distribution system according to the first embodiment will be described. Note that FIG. 2 briefly describes the outline of the distribution system according to the first embodiment, and details of the functions and the like realized by the distribution system 1 will be described with reference to the drawings and the like described later.

FIG. 2 is a diagram showing an outline example of processing of the distribution system according to the first embodiment. When the distribution system 1 distributes the captured image data captured and acquired by the photographing device 10 in a plurality of streams, the encoder of the photographing device 10 is assigned to each stream and parallel processing is performed, so that the images of the plurality of streams are processed in parallel. It is a system that can deliver data.

First, the photographing device 10 acquires photographed image data to be image-distributed. Next, the photographing device 10 prepares a plurality of streams having different image quality so that the communication terminal 50 can receive data of a desired image quality according to, for example, the state of the network band of the communication network 5. The photographing device 10 determines a method of encoding image data for each stream based on the number of streams to be streamed and the encoding parameters for each stream. The encoding parameter is information indicating the image quality of the image data processed by the encoder. Then, the photographing device 10 performs the encoding process for each frame (for example, frame X) of the photographed image data by using the determined encoding method.

In the example of FIG. 2, the photographing apparatus 10 performs encoding processing on stream 1 using a hardware encoder (HW), and encode processing on stream 2 and stream 3 using a software encoder (SW). Do. Frame X, which is one frame of captured image data, is input to three encoders in parallel after undergoing predetermined image processing, and each encoder performs encoding processing in parallel according to the set encoding parameters.

Then, the photographing device 10 distributes the plurality of encoded streaming image data as independent streams to the communication terminal 50 via the communication network 5. In this case, for example, the streaming image data delivered in stream 1 is high-quality data, the streaming image data delivered in stream 2 is medium-quality data, and the streaming image data delivered in stream 3 is. Is low quality data.

When simultaneously distributing image data of a plurality of streams having different image quality, the device on the distribution side often uses a dedicated hardware encoder. However, for example, in a device such as a photographing device such as a normal digital camera, which has physical restrictions such as the device size or layout in the device, or restrictions such as price, the number of hardware encoders installed is limited. Therefore, it was difficult to deliver multiple streams at the same time. Therefore, when the distribution system 1 distributes image data using a plurality of storms having different image quality, either the hardware encoder or the software encoder provided in the photographing device 10 for each stream depends on the encoding parameter for each stream. Is assigned and the encoding process is performed. As a result, the distribution system 1 can distribute image data of a plurality of streams having different image quality by using the photographing device 10 in which the number of hardware encoders is limited.

● Hardware Configuration Next, the hardware configuration of the photographing device 10 and the communication terminal 50 in the embodiment will be described with reference to FIGS. 3 and 4. In the hardware configuration shown in FIGS. 3 and 4, components may be added or deleted as necessary.

○ Hardware configuration of the shooting device ○
First, the hardware configuration of the photographing apparatus 10 will be described with reference to FIG. FIG. 3 is a diagram showing an example of the hardware configuration of the photographing apparatus according to the first embodiment. In the following, the photographing device 10 is an omnidirectional (omnidirectional) photographing device using two image pickup elements, but the number of image pickup elements may be two or more. Further, the photographing device 10 does not necessarily have to be a device dedicated to omnidirectional photography, and by attaching a retrofitted omnidirectional imaging unit to a normal digital camera, smartphone, or the like, substantially the same function as the photographing device 10 can be obtained. You may have it.

As shown in FIG. 3, the photographing device 10 includes an imaging unit 101, an image processing unit 104, an imaging control unit 105, a hardware encoder 106, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, and a ROM (Read). By Only Memory) 112, SRAM (Static Random Access Memory) 113, DRAM (Dynamic Random Access Memory) 114, Operation Unit 115, External Storage I / F116, Input / Output I / F117, Network I / F118 and Acceleration / Direction Sensor 119 It is configured.

Of these, the image sensor 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b each having an angle of view of 180 ° or more for forming a hemispherical image, and two images corresponding to each wide-angle lens. It includes image pickup elements 103a and 103b. The image sensors 103a and 103b are image sensors such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and a CCD (Charge Coupled Device) sensor that convert an optical image obtained by the wide-angle lenses 102a and 102b into image data of an electric signal and output the image. It has a timing generation circuit that generates a horizontal or vertical synchronization signal of the sensor, a pixel clock, etc., a register group in which various commands and parameters necessary for the operation of this image sensor are set, and the like.

The image sensors 103a and 103b are connected to the image processing unit 104 by a parallel I / F bus, respectively. Further, the image pickup devices 103a and 103b are connected to the image pickup control unit 105 by a serial I / F bus (I2C bus or the like), respectively. The image processing unit 104, the image pickup control unit 105, and the sound processing unit 109 are connected to the CPU 111 via the bus line 110. Further, the bus line 110 is connected to the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the external storage I / F116, the input / output I / F117, the network I / F118, and the acceleration / orientation sensor 119.

The image processing unit 104 takes in the image data output from the image pickup devices 103a and 103b through the parallel I / F bus. Then, the image processing unit 104 performs predetermined processing on each image data, and then synthesizes these image data to create equirectangular projected image (Equirectangular) data.

The image pickup control unit 105 generally uses the image pickup control unit 105 as a master device and the image pickup elements 103a and 103b as slave devices, and sets commands and the like in the register group of the image pickup elements 103a and 103b by using the I2C bus. Necessary commands and the like are received from the CPU 111. Further, the image pickup control unit 105 also uses the I2C bus to take in the status data of the register group of the image pickup elements 103a and 103b and send it to the CPU 111.

Further, the image pickup control unit 105 instructs the image pickup devices 103a and 103b to output image data at the timing when the shutter button of the operation unit 115 is pressed. The photographing device 10 may have a preview display function on the display and a function corresponding to the display of a still image or a moving image. In the case of moving images, the output of image data from the image sensors 103a and 103b is continuously performed at a predetermined frame rate (frames / minute).

Further, as will be described later, the image pickup control unit 105 also functions as a synchronization control means for synchronizing the output timings of the image data of the image pickup elements 103a and 103b in cooperation with the CPU 111. In the present embodiment, the photographing device 10 is not provided with a display, but may be provided with a display unit.

The hardware encoder 106 is an encoder that performs encoding processing (compression processing) using a coding method such as H.264 or MPEG-4. The hardware encoder 106 has a limited number of encoder processes that can be executed simultaneously.

The microphone 108 collects sound from the surrounding environment of the photographing device 10 and converts the collected sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 through the I / F bus and performs predetermined processing on the sound data. Each microphone 108 constitutes a sound collecting means for collecting sound from the surrounding environment. Further, although FIG. 3 shows only the case where the microphone 108 is built in the photographing device 10, the microphone 108 may be externally attached to the photographing device 10. Further, the number of microphones 108 is not limited to one, and a plurality of microphones having a predetermined arrangement configuration may be provided. Further, the microphone 108 may be an Ambisonics microphone.

The CPU 111 controls the overall operation of the photographing device 10 and executes necessary processing. The ROM 112 is a non-volatile memory that stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are volatile memories that are used as a work area of the CPU 111 and store programs executed by the CPU 111, data in the middle of processing, and the like. In particular, the DRAM 114 stores, for example, image data during processing by the image processing unit 104 and data of the processed equirectangular projection image.

The operation unit 115 is a general term for operation buttons such as a shutter button. By operating the operation unit 115, the user inputs various shooting modes, shooting conditions, and the like. The external storage I / F 116 controls reading or writing (storage) of data to an external storage such as a recording medium. The encoded streaming image data may be stored in the external storage as an image file (moving image file).

The input / output I / F 117 is an interface for connecting to an external device in order to input / output various data or information to / from various external devices. The input / output I / F 117 is configured by, for example, a USB (Universal Serial Bus) I / F regardless of whether it is wireless or wired. The network I / F 118 is an interface for performing data communication via the communication network 5. The network I / F 118 is composed of a wired NIC (Network Interface Card), a wireless NIC or a USB I / F.

The acceleration / direction sensor 119 calculates the direction of the photographing device 10 from the magnetism of the earth and outputs the direction information. The orientation information is an example of related information (metadata) according to Exif (Exchangeable image file format), and is used for image processing such as image correction of a captured image. The related information includes each data of the shooting date and time of the image and the data capacity of the image data.

The photographing device 10 is not limited to the omnidirectional (omnidirectional) photographing device, and may be a photographing device such as a normal digital camera, a surveillance camera, or a video camera. In this case, the image pickup unit 101 as a photographing means is composed of a set of lenses and an image pickup element.

〇Hardware configuration of communication terminal ○
Subsequently, the hardware configuration of the communication terminal 50 will be described with reference to FIG. FIG. 4 is a diagram showing an example of the hardware configuration of the communication terminal according to the embodiment. The communication terminal 50 includes a CPU 501, a ROM 502, a RAM 503, an input device 504, a display 505, an external storage I / F 506, an input / output I / F 507, and a network I / F 508.

The CPU 501 controls the operation of the entire communication terminal 50. The ROM 502 is a non-volatile memory that stores a program used to drive the CPU 501. The RAM 503 is a volatile memory used as a work area of the CPU 501. The input device 504 is a kind of input means for receiving an operation input by a user. The input device 504 is, for example, a keyboard, a mouse, a touch panel, buttons, a dial, a microphone for voice input, and the like.

The display 505 is a kind of display means such as a liquid crystal or an organic EL that displays various information such as a cursor, a menu, a window, characters, an icon, or an image. The external storage I / F 506 controls reading or writing (storage) of data to an external storage such as a recording medium. The input / output I / F 507 is an interface for connecting to an external device in order to input / output various data or information to / from various external devices. The network I / F 508 is an interface for performing data communication via the communication network 5. Further, the communication terminal 50 includes a bus line 510. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501.

Each of the above programs may be a file in an installable format or an executable format, and may be recorded and distributed on a computer-readable recording medium. Examples of recording media include CD-R (Compact Disc Recordable), DVD (Digital Versatile Disk), Blu-ray (registered trademark) disc, SD card, USB memory and the like. In addition, the recording medium can be provided domestically or internationally as a program product. For example, the photographing apparatus 10 realizes the distribution method according to the present invention by executing the program according to the present invention.

● Functional configuration Next, the functional configuration of the distribution system according to the first embodiment will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram showing an example of the functional configuration of the distribution system according to the embodiment. Note that FIG. 5 shows the devices and terminals shown in FIG. 1 that are related to the processing or operation described later.

○ Functional configuration of the imaging device ○
First, the functional configuration of the photographing apparatus 10 will be described with reference to FIG. The photographing device 10 includes a communication unit 11, a reception unit 12, an input / output unit 13, a determination unit 14, an imaging processing unit 15, a parameter setting unit 16, an image processing unit 17, an encoding processing unit 18, and a storage / reading unit 19. ing. Each of these parts has a function or functions realized by operating any of the components shown in FIG. 3 by an instruction from the CPU 111 according to a program developed from the ROM 112 or the SRAM 113 on the DRAM 114. It is a means. Further, the photographing apparatus 10 has a storage unit 1000 constructed by the ROM 112, the SRAM 113, or the DRAM 114 shown in FIG.

The communication unit 11 is realized by the command from the CPU 111 shown in FIG. 3 and the network I / F 118, and transmits / receives various data (or information) to / from the communication terminal 50 via the communication network 5. It is a function. For example, the communication unit 11 distributes the image data photographed and acquired by the photographing processing unit 15 to the communication terminal 50 via the communication network 5. The communication unit 11 is an example of the distribution means.

The reception unit 12 is a function realized by an instruction from the CPU 111 shown in FIG. 3 and an input means such as an operation unit 115, and receives various inputs from the user. The determination unit 14 is a function realized by an instruction from the CPU 111 shown in FIG. 3 and performing various determinations. The input / output unit 13 is a function realized by an instruction from the CPU 111 shown in FIG. 3 and an input / output I / F 117, and inputs / outputs various data or information to / from an external device. The determination unit 14 determines, for example, the encoding method of the captured image data according to the encoding parameters for each stream. Specifically, the determination unit 14 determines whether the captured image data for each stream is encoded by the hardware encoder or the software encoder. The determination unit 14 is an example of the determination means.

The shooting processing unit 15 is realized by a command from the CPU 111 shown in FIG. 3, an image pickup unit 101, an image processing unit 104, an image pickup control unit 105, a microphone 108, a sound processing unit 109, and the like, and captures a subject, a landscape image, and the like. It is a function to take a picture and acquire the shot image data. The photographing processing unit 15 is an example of the acquisition means.

The parameter setting unit 16 is a function realized by an instruction from the CPU 111 shown in FIG. 3 and sets an encoding parameter for each stream. The parameter setting unit 16 is an example of the setting means. The image processing unit 17 is a function realized by a command from the CPU 111 shown in FIG. 3 and performs various processing on the photographed image data acquired by the photographing processing unit 15.

The encoding processing unit 18 is a function realized by an instruction from the CPU 111 shown in FIG. 4 and performs encoding processing of captured image data by using a determined encoding method. The encoding processing unit 18 performs encoding processing using, for example, a determined encoding method, and generates streaming image data for each stream. When the encoding method is a method using a hardware encoder, the encoding processing unit 18 performs encoding processing of captured image data using the hardware encoder 106 shown in FIG. On the other hand, when the encoding method is a method using a software encoder, the encoding processing unit 18 uses a software encoder in which the CPU 111 shown in FIG. 3 executes the dedicated software to operate the captured image. Performs data encoding processing. The encoding processing unit 18 is an example of the generation means.

The storage / reading unit 19 is realized by an instruction from the CPU 111 shown in FIG. 3, and is a function of storing various data in the storage unit 1000 or reading various data from the storage unit 1000.

〇 Encoding parameter management table FIG. 6 is a conceptual diagram showing an example of the encoding parameter management table according to the first embodiment. In the storage unit 1000, an encoding parameter management DB 1001 configured by an encoding parameter management table as shown in FIG. 6 is constructed. The encoding parameter management table stores and manages the encoding method for each stream using the encoding processing unit 18. The encoding parameter management table shown in FIG. 6 is an example in which one hardware encoder 106 as shown in FIG. 3 is provided in the photographing device 10. Further, the number of encoding processes that can be simultaneously executed by the hardware encoder 106 in this case is one.

In the encoding parameter management table, an encoding parameter and a target encoder indicating an encoder used for encoding processing are stored and managed in association with each stream for delivering image data. Of these, the encoding parameter shows information indicating the image quality of the image data for each stream. The information indicated by the image quality includes, for example, the resolution and frame rate of the image data. The encoding parameter may include the bit rate of the image data. When the image data is moving image data, the encoding parameter is treated as information indicating the sound quality of the audio data as well as the image quality. Further, the encoding parameter may include a coding method used for the encoding process. As the coding method, for example, in the case of a still image, JPEG (Joint Photographic Experts Group) or an advanced system thereof (JPEG2000). , GIF (Graphics Interchange Format) and the like, and in the case of moving images, MPEG (Moving Picture Experts Group) or its development system (H.264, H.265) and the like can be mentioned.

As the target encoder, for example, the hardware encoder 106 is preferentially assigned to the streams 1 to 3 in descending order of the processing load of the encoding parameters. In the encoding parameter management table, for example, the hardware encoder 106 is assigned to the stream 1 having the highest resolution, and the software encoder is assigned to the streams 2 and 3.

○ Functional configuration of communication terminals ○
Subsequently, the functional configuration of the communication terminal 50 will be described with reference to FIG. The communication terminal 50 has a communication unit 51, a reception unit 52, a display control unit 53, a determination unit 54, and a storage / reading unit 59. Each of these parts is a function realized or a means for functioning by operating any of the components shown in FIG. 4 by an instruction from the CPU 501 according to a program developed on the RAM 503. Further, the communication terminal 50 has a storage unit 5000 constructed by the ROM 502 shown in FIG. 4 or the external storage.

The communication unit 51 is realized by a command from the CPU 501 shown in FIG. 4 and a network I / F 508, and is a function of transmitting and receiving various data or information to and from the photographing device 10 via the communication network 5. is there.

The reception unit 52 is a function realized by the instruction from the CPU 501 and the input device 504 shown in FIG. 4 and accepts various input operations from the user. The display control unit 53 is a function realized by the instruction from the CPU 501 shown in FIG. 3 and the display 505, and causes the display 505 to display various screens such as an operation screen that accepts an input operation by the user. The determination unit 54 is a function realized by an instruction from the CPU 501 shown in FIG. 4 and performing various determinations. The storage / reading unit 59 is a function executed by an instruction from the CPU 501 shown in FIG. 4 to store various data in the storage unit 5000 or read various data from the storage unit 5000.

● Processing or operation of the first embodiment Next, the processing or operation of the distribution system according to the first embodiment will be described with reference to FIGS. 7 to 11. First, the overall flow of processing or operation in the distribution system 1 will be described with reference to FIG. 7. FIG. 7 is a sequence diagram showing an example of image distribution processing in the distribution system according to the first embodiment.

First, the reception unit 52 of the communication terminal 50 accepts the distribution start request of the streaming image by accepting a predetermined input operation of the user to the input device 504 (step S11). The communication unit 51 transmits an image distribution request indicating that the distribution of the streaming image is requested to the photographing device 10 (step S12). As a result, the communication unit 11 of the photographing device 10 receives the image distribution request transmitted from the communication terminal 50. The image distribution request transmitted from the communication terminal 50 may include information on the number of streams requesting image distribution and encoding parameters for each stream. In this case, the number of streams requesting image distribution and the information of the encoding parameter for each stream included in the image distribution request are received by the reception unit 52 when the user input to the input device 504 is performed. Further, the photographing device 10 may start a process for automatically distributing data by using a user operation on the operation unit 115 or turning on the power of the photographing device 10 as a trigger.

Next, the shooting processing unit 15 of the shooting device 10 executes shooting processing of the subject and acquires shot image data (step S13). Then, the photographing device 10 executes an encoding process on the photographed image data acquired by the photographing processing unit 15 (step S14).

Here, the details of the encoding process of the photographed image data by the photographing apparatus 10 will be described with reference to FIG. FIG. 8 is a flowchart showing an example of image data encoding processing according to the first embodiment.

First, the parameter setting unit 16 sets the number of streams to be streamed and the encoding parameters for each stream (step S101). Specifically, the parameter setting unit 16 determines the number of streams and streams based on the information included in the image distribution request received by the communication unit 11 or the information input from the external device to the input / output unit 13. The encoding parameter for each is set in the encoding parameter management DB 1001 (see FIG. 6). The parameter setting unit 16 may set information stored in advance for streaming distribution as an encoding parameter. In the case of the example shown in FIG. 6, the parameter setting unit 16 sets three streams and stores the information of the encoding parameter for each stream in the encoding parameter management DB 1001.

Next, the determination unit 14 determines the number of hardware encoders 106 that can be processed simultaneously (step S102). The determination unit 14 determines the number of processes that can be simultaneously executed by the hardware encoder 106, for example, based on the processing capacity of the hardware encoder 106 or the value of the encoding parameter set in step S101. In the case of the encoding parameter management table shown in FIG. 6, the determination unit 14 determines that the number of simultaneous processes that can be executed is one.

Next, the determination unit 14 determines the target encoder to perform the encoding process of each stream as the encoding method of the captured image data for each stream set in step S101 (step S103). Specifically, the determination unit 14 determines whether to perform the encoding process using the hardware encoder 106 or the software encoder for each stream using the encoding parameters set in step S101. The determination unit 14 determines, for example, the target encoder so that the hardware encoder 106 is assigned to the stream having a large processing load. The number of streams that can be allocated by the hardware encoder 106 corresponds to the number of concurrent processes that can be executed determined in step S102.

The determination unit 14 allocates the hardware encoder 106 to the high-resolution stream having a large processing load according to the resolution value of the image data, which is one of the encoding parameters, and assigns the software encoder to the other streams. assign. When using the encoding parameter management table shown in FIG. 6, the determination unit 14 assigns the hardware encoder 106 to the stream 1 having the highest resolution, and assigns the software encoder to the streams 2 and 3.

The photographing processing unit 15 captures one frame of captured image data among the captured image data captured and acquired by using the imaging unit 101, which is an example of the photographing means (step S104). Then, the image processing unit 17 executes image processing of the captured captured image data (step S105). Then, the encoding processing unit 18 executes the encoding processing of the image data processed in step S105 using the target encoder determined in step S103 based on the encoding parameters set in each stream (step S106). .. Then, the encoding processing unit 18 performs encoding processing using the target encoder for each stream, and generates streaming image data for each stream. The encoding processing unit 18 processes the encoding processing using the target encoders related to the plurality of streams and the generation of the streaming image data in parallel using the hardware encoder and the software encoder.

Then, the communication unit 11 distributes the streaming image data, which is the captured image data encoded in step S106, to the communication terminal 50 (step S107). The streaming image data may be delivered directly to the communication terminal 50 by using the input / output unit 13 without going through the communication network 5.

Then, when the distribution of the captured image data for all the frames acquired by the photographing processing unit 15 is completed (YES in step S108), the photographing device 10 ends the process. On the other hand, when the captured image data acquired by the photographing processing unit 15 still has frames for which distribution has not been completed (NO in step S108), the photographing device 10 repeats the processing from step S104. Continue to deliver streaming image data.

As a result, when delivering image data of a plurality of streams having different image quality, the photographing device 10 determines an appropriate encoding method (whether a hard or soft encoder is used) according to the image quality of the image data for each stream. can do.

Returning to FIG. 7, as shown in step S107, the communication unit 11 of the photographing apparatus 10 transmits the plurality of streaming image data encoded and generated in step S14 to the communication terminal via the communication network 5. It is transmitted (delivered) to 50 (step S15).

Next, the communication terminal 50 selects the data to be received from the plurality of streaming image data distributed from the photographing device 10 (step S16). Specifically, the reception unit 52 of the communication terminal 50 receives the selection of the streaming image data to be received by the user's predetermined input operation to the input device 504. Further, the determination unit 54 of the communication terminal 50 selects the streaming image data to be received according to the information of the network band by the communication unit 51. In this case, the communication unit 11 monitors the communication state of the communication network 5 depending on, for example, the presence or absence of a communication delay. The determination unit 54 selects streaming image data corresponding to the current communication state of the communication network 5. Then, the display control unit 53 of the communication terminal 50 displays the image related to the streaming image data selected and received in step S16 on the display 505 (step S17).

Here, another example of the method of encoding the captured image data determined by the determination unit 14 described above will be described. First, unlike the example of FIG. 6, the encoding parameter management table shown in FIG. 9A shows the number of encoding processes that can be simultaneously executed by one hardware encoder 106 as shown in FIG. This is an example when there are two. In the encoding parameter management table shown in FIG. 9A, for example, the hardware encoder is assigned to the stream 1 having the highest resolution and the stream 2 having the second highest resolution, and the software encoder is assigned to the stream 3. Has been done. In this way, the photographing device 10 can allocate the hardware encoder to the encoding processing of a plurality of streams according to the number of hardware encoders 106 or the number of processing that can be simultaneously executed by one hardware encoder 106. is there.

Further, the encoding parameter management table shown in FIG. 9B is an example in which the frame rate of the encoding parameters differs for each stream, unlike the example of FIG. In the encoding parameter management table shown in FIG. 9B, for example, a hardware encoder is assigned to stream 1 having the highest frame rate, and a software encoder is assigned to stream 2 and stream 3. In this way, the photographing device 10 determines the target encoder to be used for the encoding processing for each stream by determining the processing load of the encoder using not only the resolution of the captured image data but also the frame rate of the image data as the encoding parameter. can do.

The type of encode parameter used to determine the target encoder (encoding method) for each stream can be set arbitrarily. For example, the determination unit 54 may determine the target encoder by giving priority to the resolution value among the plurality of encoding parameters, or may determine the target encoder by giving priority to the frame rate value.

● Effect of the first embodiment Therefore, the distribution system according to the first embodiment generates streaming image data by using an encoding method determined according to the encoding parameters for each stream having different image quality. Image data of multiple streams can be distributed at the same time.

● Modified Example of First Embodiment Next, the distribution system according to the modified example of the first embodiment will be described with reference to FIGS. 10 and 11. The same configurations and the same functions as those in the above-described embodiment are designated by the same reference numerals, and the description thereof will be omitted. The distribution system 1a according to the modification of the first embodiment is a system capable of changing the encoding parameters of the image data while the streaming image data is being distributed from the photographing device 10 by the distribution system 1 described above.

FIG. 10 is a flowchart showing an example of image data encoding processing according to a modified example of the first embodiment. Since the processing of steps S201 to S208 shown in FIG. 10 is the same as the processing of steps S101 to S108 shown in FIG. 8, the description thereof will be omitted. When the distribution of the streaming image data is continuing (NO in step S208), the photographing device 10 shifts the process to step S209. The parameter setting unit 16 determines whether or not the encoding parameter has been changed (step S209). The parameter setting unit 16 determines whether or not the encoding parameter has been changed, and changes the value of the encoding parameter.

When the encoding parameter is changed (YES in step S209), the parameter setting unit 16 of the photographing device 10 shifts the process to step S210. Specifically, the parameter setting unit 16 receives a parameter change request from the communication terminal 50 by the communication unit 11 or the input / output unit 13, or a parameter change request by the distributor by an input operation by the reception unit 12. Is accepted, the value of the encoding parameter stored in the encoding parameter management DB 1001 is changed. In the case of the example shown in FIG. 11, the parameter setting unit 16 changes the frame rate of the stream 2 from “10 fps” to “30 fps” among the encoding parameters whose frame rates differ for each stream. On the other hand, when the parameter setting unit 16 does not change the encoding parameter (NO in step S209), the parameter setting unit 16 repeats the process from step S204.

Next, the determination unit 14 changes the encoding method for each stream according to the changed encoding parameter, and changes the target encoder used for the encoding process (step S210). Specifically, in the case of the example shown in FIG. 11, when the frame rate of the stream 2 is changed, the determination unit 14 recalculates the encoder allocation, and the processing load is the highest at the high frame rate. The target encoder of stream 2 is changed from a software encoder to a hardware encoder, and the target encoder of stream 1 having a frame rate lower than that of stream 2 is changed from a hardware encoder to a software encoder.

Then, the photographing device 10 repeats the processing from step S204 using the changed target encoder, and continues the distribution of the streaming image data. On the other hand, when the encoding parameter is not changed, the parameter setting unit 16 continues to deliver the encoded streaming image data using the target encoder determined in step S203.

As a result, even if the encoding parameters are changed during the distribution of the image data, the distribution system 1a can recalculate the encoder allocation and automatically change to the encoding method, so that the distribution system 1a distributes with the optimum encoder allocation. Can be continued.

● Second embodiment ●
Subsequently, the configuration of the distribution system according to the second embodiment will be described with reference to FIGS. 12 to 15. The same configurations and the same functions as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted. The distribution system according to the second embodiment is a system capable of distributing the streaming image data generated by the photographing device 10 to the communication terminal 50 via the distribution control device 90.

● System Configuration FIG. 12 is a diagram showing an example of the system configuration of the distribution system according to the second embodiment. In the distribution system 2 shown in FIG. 12, a distribution control device 90 is added to the configuration according to the first embodiment. The distribution control device 90 can perform mutual communication with the photographing device 10 and the communication terminal 50 via the communication network 5.

The distribution control device 90 is a server computer and controls the distribution of streaming image data transmitted from the photographing device 10. The distribution control device 90 may be constructed by a single computer, or may be constructed by a plurality of computers arbitrarily assigned by dividing each part (function or means). Further, since the hardware configuration of the distribution control device 90 is the same as or partially modified from the communication terminal 50 shown in FIG. 4, the description thereof will be omitted.

● Schematic FIG. 13 is a diagram showing an outline example of processing of the distribution system according to the second embodiment. The process of generating the streaming image data for each stream by the photographing device 10 is the same as the process described in the first embodiment.

The photographing device 10 transmits the streamed image data for each stream generated to the distribution control device 90 via the communication network 5. The distribution control device 90 stores a plurality of received streaming data in the storage unit 9000. Then, the distribution control device 90 distributes (transmits) the streaming image data transmitted from the photographing device 10 and stored in the storage unit 9000 to the communication terminal 50 in response to the request from the communication terminal 50.

As shown in FIG. 13, the distribution control device 90 can simultaneously distribute streaming image data to a plurality of communication terminals 50 (communication terminal 50A and communication terminal 50B). Further, the distribution control device 90 can distribute the streaming image data corresponding to the stream according to the request from the communication terminal 50, and can also distribute the streaming image data of a different stream for each communication terminal 50. As a result, the distribution system 2 uses the distribution control device 90 to make the streaming image data transmitted from the photographing device 10 suitable for each communication terminal 50 according to the state of the network band of each communication terminal 50 and the like. It can be streamed.

● Functional configuration Next, the functional configuration of the distribution system according to the second embodiment will be described. FIG. 14 is a diagram showing an example of the functional configuration of the distribution system according to the second embodiment. Since the functional configurations of the photographing device 10 and the communication terminals 50 (communication terminals 50A and 50B) are the same as the configurations shown in FIG. 5, the description thereof will be omitted.

The distribution control device 90 has a communication unit 91 and a storage / reading unit 99. Each of these parts is a function realized or a means for functioning by operating any of the components shown in FIG. 4 by an instruction from the CPU 501 according to a program developed on the RAM 503. Further, the distribution control device 90 has a storage unit 9000 constructed by the ROM 502 shown in FIG. 4 or the external storage.

The communication unit 91 is realized by the command from the CPU 501 shown in FIG. 4 and the network I / F 508, and transmits / receives various data or information between the photographing device 10 and the communication terminal 50 via the communication network 5. It is a function to perform. The storage / reading unit 99 is a function executed by an instruction from the CPU 501 shown in FIG. 4 to store various data in the storage unit 9000 or read various data from the storage unit 9000. The storage unit 9000 stores the streaming image data transmitted from the photographing device 10.

● Processing or operation of the second embodiment Next, the processing or operation of the distribution system according to the second embodiment will be described with reference to FIG. FIG. 15 is a sequence diagram showing an example of image distribution processing in the distribution system according to the second embodiment.

First, the communication unit 91 of the distribution control device 90 transmits an image acquisition request indicating that the image acquisition device 10 is requested to acquire the streaming image (step S51). As a result, the communication unit 11 of the photographing device 10 receives the image acquisition request transmitted from the distribution control device 90. The image acquisition request transmitted from the distribution control device 90 may include information on the number of streams for which image acquisition is requested and the encoding parameter for each stream. In this case, the distribution control device 90 may, for example, read and transmit the information stored in the storage unit 9000 in advance, or may transmit the information received by the input from the user to the distribution control device 90. .. Further, the photographing device 10 may start a process for automatically acquiring an image triggered by a user operation on the operation unit 115 or a power-on of the photographing device 10.

Next, the photographing processing unit 15 of the photographing apparatus 10 performs a photographing process (step S52). Then, the photographing device 10 executes an encoding process on the photographed image data acquired by the photographing processing unit 15 (step S53). Since the encoding process in step S53 is the same as the content shown in FIG. 8, the description thereof will be omitted.

Next, the communication unit 11 of the photographing device 10 transmits (distributes) the plurality of streaming data encoded in step S14 to the distribution control device 90 via the communication network 5 (step S54). As a result, the communication unit 91 of the distribution control device 90 receives the plurality of streaming image data transmitted from the photographing device 10. Then, the storage / reading unit 99 of the distribution control device 90 stores the streaming image data received by the communication unit 91 in the storage unit 9000 (step S55). In this way, the distribution control device 90 temporarily stores the streaming image data transmitted from the photographing device 10.

Next, the communication unit 51A of the communication terminal 50A transmits an image distribution request indicating that the distribution control device 90 is requested to distribute the streaming image in response to the request from the user received by the reception unit 52. (Step S56). As a result, the communication unit 91 of the distribution control device 90 receives the image distribution request transmitted from the communication terminal 50A. Next, the communication unit 91 of the distribution control device 90 transmits (distributes) a plurality of streaming image data stored in the storage unit 9000 to the communication terminal 50A (step S57).

The communication terminal 50A selects the data to be received from the plurality of streaming image data distributed from the distribution control device 90 (step S58). Specifically, the reception unit 52A of the communication terminal 50A accepts the selection of the streaming image data to be received by the user's predetermined input operation to the input device 504. Further, the determination unit 54A of the communication terminal 50A selects the streaming image data to be received according to the information of the network band by the communication unit 51A. Then, the display control unit 53A of the communication terminal 50A displays the image related to the streaming image data selected and received in step S58 on the display 505 (step S59).

On the other hand, the communication unit 51B of the communication terminal 50B transmits an image distribution request indicating that the distribution control device 90 is requested to distribute the streaming image in response to the request from the user received by the reception unit 52B. (Step S60). As a result, the communication unit 91 of the distribution control device 90 receives the image distribution request transmitted from the communication terminal 50B. Next, the communication unit 91 of the distribution control device 90 transmits (distributes) a plurality of streaming image data stored in the storage unit 9000 to the communication terminal 50B (step S61).

Similar to the process in step S58, the communication terminal 50B selects the data to be received from the plurality of streaming image data distributed from the distribution control device 90 (step S62). Then, the display control unit 53B of the communication terminal 50B displays the image related to the streaming image data selected and received in step S62 on the display 505 (step S63). The order of the processes of steps S56 to S59 and the processes of steps S60 to S63 may be changed or may be performed in parallel.

● Effect of the Second Embodiment Therefore, the distribution system according to the second embodiment uses the distribution control device 90 to transfer the streaming image data transmitted from the photographing device 10 to the network band of each communication terminal 50. Depending on the state and the like, it can be distributed in a stream suitable for each communication terminal 50.

● Summary ●
As described above, the photographing device according to the embodiment of the present invention is the photographing device 10 that distributes the image data of a plurality of streams, and the photographed image data photographed by the imaging unit 101 (an example of the photographing means). The shooting processing unit 15 (an example of the acquisition means) for acquiring the data, and the determination unit 14 (an example of the determination means) for determining the encoding method of the acquired captured image data according to the encoding parameters for each stream were determined. An encoding processing unit 18 (an example of a generation means) that generates streaming image data for each stream using an encoding method, and a communication unit 11 (an example of a distribution means) that distributes a plurality of generated streaming image data. Be prepared. As a result, the photographing device 10 can simultaneously deliver the image data of a plurality of streams by performing the encoding process using an appropriate encoding method according to the image quality.

Further, the photographing apparatus according to the embodiment of the present invention includes at least one hardware encoder 106 and a software encoder. Then, in the photographing apparatus 10, the encoding processing unit 18 (an example of the generation means) uses the hardware encoder 106 to stream image data for the stream corresponding to the encoding parameter of high resolution or high frame rate having a large processing load. To generate. As a result, the photographing device 10 can simultaneously deliver the image data of a plurality of streams even when the number of hardware encoders is limited.

Further, in the photographing apparatus according to the embodiment of the present invention, the encoding processing unit 18 (an example of the generation means) generates a plurality of streaming image data in parallel by using the hardware encoder 106 and the software encoder. As a result, the photographing device 10 can simultaneously distribute the image data of a plurality of streams by using the photographing device 10 in which the number of hardware encoders is limited.

Further, the photographing apparatus according to the embodiment of the present invention includes a parameter setting unit 16 (an example of setting means) for setting encoding parameters for each stream. Then, in the photographing apparatus 10, the determination unit 14 (an example of the determination means) changes the determined encoding method when the set encoding parameter is changed. As a result, even if the encoding parameters are changed during the distribution of the streaming image data, the photographing device 10 can recalculate the encoder allocation and automatically change to the encoding method, so that the optimum encoder allocation is used. Streaming distribution can be continued.

● Supplement ●
Each function of the embodiment described above can be realized by one or more processing circuits. Here, the "processing circuit" in the present specification is a processor programmed to execute each function by software such as a processor implemented by an electronic circuit, or a processor designed to execute each function described above. It shall include devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules.

Further, the various tables of the embodiment described above may be generated by the learning effect of machine learning, and by classifying the data of each related item by machine learning, the table is not used. You may. Here, machine learning is a technology for making a computer acquire learning ability like a human being, and the computer autonomously generates an algorithm necessary for judgment such as data identification from learning data taken in advance. However, it refers to a technology that applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, enhanced learning, and deep learning, and may be a learning method that combines these learning methods. Machine learning It doesn't matter how you learn for.

Although the photographing apparatus, the distribution system, the distribution method and the recording medium according to the embodiment of the present invention have been described so far, the present invention is not limited to the above-described embodiment, and the addition of other embodiments. Changes or deletions can be made within the range that can be conceived by those skilled in the art, and any aspect is included in the scope of the present invention as long as the actions and effects of the present invention are exhibited.

1 Distribution system 5 Communication network 10 Imaging device 11 Communication unit (example of distribution means)
14 Judgment unit (an example of decision-making means)
15 Shooting processing unit (example of acquisition means)
16 Parameter setting unit (example of setting means)
18 Encoding processing unit (example of generation means)
50 Communication terminal 90 Distribution control device 101 Imaging unit (example of imaging means)

Japanese Unexamined Patent Publication No. 2011-77564

Claims (10)

A shooting device that distributes image data of multiple streams.
An acquisition means for acquiring captured image data taken by the photographing means, and
Determining means for determining the encoding method of the acquired captured image data according to the encoding parameters for each stream, and
A generation means for generating streaming image data for each stream using the determined encoding method, and
A distribution method that distributes multiple generated streaming image data,
A shooting device equipped with. The imaging device according to claim 1.
Equipped with at least one hardware encoder and software encoder
The generation means is a photographing device that generates the streaming image data by using the hardware encoder for a stream corresponding to the encoding parameter having a large processing load. The encoding parameter is the resolution of the streaming image data.
The photographing apparatus according to claim 2, wherein the generation means generates the streaming image data for the stream having a high resolution by using the hardware encoder. The encoding parameter is the frame rate of the streaming image data.
The imaging device according to claim 2 or 3, wherein the generation means generates the streaming image data for the stream having a high frame rate by using the hardware encoder. The photographing apparatus according to any one of claims 2 to 4, wherein the generation means uses the hardware encoder and the software encoder to generate a plurality of streaming image data in parallel. The photographing apparatus according to any one of claims 1 to 5.
A setting means for setting the encoding parameter for each stream is provided.
The determination means is an imaging device that determines an encoding method of the acquired captured image data according to a set encoding parameter. The imaging device according to claim 6, wherein the determination means changes the determined encoding method when the set encoding parameter is changed. A distribution system equipped with a photographing device that distributes image data of a plurality of streams.
An acquisition means for acquiring captured image data taken by the photographing means, and
Determining means for determining the encoding method of the acquired captured image data according to the encoding parameters for each stream, and
A generation means for generating streaming image data for each stream using the determined encoding method, and
A distribution method that distributes multiple generated streaming image data,
Delivery system with. It is a distribution method executed by a shooting device that distributes image data of multiple streams.
The acquisition step of acquiring the captured image data captured by the photographing means, and
A determination step of determining the encoding method of the acquired captured image data according to the encoding parameters for each stream, and
A generation step of generating streaming image data for each stream using the determined encoding method, and
A distribution step that distributes multiple generated streaming image data,
Delivery method to execute. For shooting devices that distribute image data of multiple streams
The acquisition step of acquiring the captured image data captured by the photographing means, and
A determination step of determining the encoding method of the acquired captured image data according to the encoding parameters for each stream, and
A generation step of generating streaming image data for each stream using the determined encoding method, and
A distribution step that distributes multiple generated streaming image data,
A computer-readable recording medium that contains a program for executing the program.

Images