EP3419294B1 - Imaging apparatus, processing method for imaging apparatus, and storage medium - Google Patents

Imaging apparatus, processing method for imaging apparatus, and storage medium Download PDF

Info

Publication number
EP3419294B1
EP3419294B1 EP18169475.3A EP18169475A EP3419294B1 EP 3419294 B1 EP3419294 B1 EP 3419294B1 EP 18169475 A EP18169475 A EP 18169475A EP 3419294 B1 EP3419294 B1 EP 3419294B1
Authority
EP
European Patent Office
Prior art keywords
moving image
distribution
segment
unit
image stream
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP18169475.3A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3419294A1 (en
Inventor
Tomohiro Sakai
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Canon Inc filed Critical Canon Inc
Publication of EP3419294A1 publication Critical patent/EP3419294A1/en
Application granted granted Critical
Publication of EP3419294B1 publication Critical patent/EP3419294B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/103—Selection of coding mode or of prediction mode
    • H04N19/107—Selection of coding mode or of prediction mode between spatial and temporal predictive coding, e.g. picture refresh
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/70—Media network packetisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146—Data rate or code amount at the encoder output
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/157—Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
    • H04N19/159—Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/164—Feedback from the receiver or from the transmission channel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/23439—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements for generating different versions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/235—Processing of additional data, e.g. scrambling of additional data or processing content descriptors
    • H04N21/2353—Processing of additional data, e.g. scrambling of additional data or processing content descriptors specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/61—Network physical structure; Signal processing
    • H04N21/6106—Network physical structure; Signal processing specially adapted to the downstream path of the transmission network
    • H04N21/6125—Network physical structure; Signal processing specially adapted to the downstream path of the transmission network involving transmission via Internet
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/845—Structuring of content, e.g. decomposing content into time segments
    • H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof

Definitions

  • the present invention relates to an imaging apparatus, a processing method for the imaging apparatus, and a program.
  • the adaptive streaming technique has become popular.
  • the first technique is the Dynamic Adaptive Streaming over HTTP (DASH) standardized by the Moving Picture Experts Group (MPEG).
  • the second technique is the HTTP Live Streaming (HLS).
  • DASH Dynamic Adaptive Streaming over HTTP
  • HLS HTTP Live Streaming
  • HTTP Hypertext Transfer Protocol
  • a distribution server divides a moving image stream into a plurality of moving image files called segments and distributes the segments to a client.
  • a client acquires a media presentation description (MPD) file including information about moving image streams distributable by a distribution server. Then, the client selects a moving image stream having an appropriate resolution and/or a bit rate from among the moving image streams according to the display performance and/or communication condition of the client. Then, the client downloads and reproduces a moving image in a format of MPEG2-TS or MP4 in segment units according to a description of the MPD.
  • MPD media presentation description
  • a moving image codec of a moving image in a format of MPEG2-TS and MP4 With a moving image codec of a moving image in a format of MPEG2-TS and MP4, a moving image frame encoded through intraframe coding (i.e., I-frame) and a moving image frame encoded through interframe coding (i.e., P-frame and B-frame) are generated.
  • the client starts reproducing a moving image from the I-frame.
  • a segment to be subsequently distributed to a client is announced before a distribution request of a moving image is received from the client, and a request of cancelling the distribution or a request of changing the distribution order is received from the client prior to the distribution.
  • an I-frame is generated when a distribution request of the moving image is transmitted from the client.
  • JP 2016 531466 A and JP 2002 344932 A no attention is given to an imaging method of a moving image frame for live-style DASH distribution (DASH live distribution) using a system such as a network camera having an imaging unit and a distribution server. Therefore, it may increase a delay time for creating a segment including an I-frame necessary for the client to start reproducing the moving image.
  • DASH live distribution live-style DASH distribution
  • JP 2016 531466 A does not provide a description with respect to a case where a segment including an I-frame does not exist in a distribution server when the DASH live distribution to the client is started. In such a case, a delay time will be longer by a period in which the client waits for the distribution server to newly create a segment including an I-frame to acquire the segment.
  • a delay time will be longer by a period in which the client transmits the moving image distribution request to acquire a segment including an I-frame.
  • US 2014/380376 A1 discloses a technique for streaming video, especially a CPT stream to a client device via an HTTP live streaming (HLS) protocol having a plurality of HLS media files.
  • the thus disclosed method comprises generating an HLS playlist specifying a plurality HLS media files before generating any of the HLS media files, wherein each of the plurality of HLS media files has a unique name, transmitting the HLS playlist to the client device, after generating the HLS playlist, transcoding the CPT stream into at least a subset of the plurality of HLS media files, and providing at least one of the subset of plurality of HLS media files to the client device, wherein the at least one of the plurality of HLS media files is a client-requested media file selected from the HLS playlist.
  • HLS HTTP live streaming
  • an imaging apparatus as specified in claims 1 to 8.
  • a processing method of an imaging apparatus as specified in claim 9.
  • a program as specified in claim 10.
  • Fig. 1 is a diagram illustrating a configuration example of an imaging system according to a first exemplary embodiment of the present disclosure.
  • the imaging system includes an imaging apparatus 101, a client 102, and a network 103.
  • the imaging apparatus 101 may be a network camera, and the network 103 may be an internet protocol (IP) network such as a local area network (LAN).
  • IP internet protocol
  • the imaging apparatus 101 and the client 102 are mutually connected and communicable via the network 103.
  • the imaging apparatus 101 distributes a captured image to the client 102 via the network 103.
  • the client 102 is an external device of the imaging apparatus 101.
  • the client 102 transmits various request commands to the imaging apparatus 101.
  • the imaging apparatus 101 transmits responses to these commands to the client 102.
  • the network 103 includes a plurality of routers, switches, and cables which satisfy a communication standard such as Ethernet®.
  • a communication standard, a scale, and a configuration thereof are not restricted as long as the network 103 allows communication between the imaging apparatus 101 and the client 102 to be executable.
  • Networks from the internet to a wireless LAN are applicable.
  • a number of imaging apparatuses may be two or more.
  • Fig. 2 is a block diagram illustrating a hardware configuration example of the imaging apparatus 101 according to the present exemplary embodiment.
  • a central processing unit (CPU) 201 In the imaging apparatus 101, a central processing unit (CPU) 201, a primary storage device 202, a secondary storage device 203, an image-capturing interface 206, and a network interface 207 are mutually connected via an internal bus 204.
  • the imaging apparatus 101 includes an image sensor 205.
  • the primary storage device 202 is a writable high-speed storage device such as a random access memory (RAM), on which an operating system (OS), various programs, and various data are loaded.
  • the primary storage device 202 is also used as a work area of the OS or various programs.
  • the secondary storage device 203 is a non-volatile storage device such as a flash memory, a hard disk drive (HDD), or a secure digital (SD) card.
  • the secondary storage device 203 is used as a permanent storage area of the OS, various programs, and various data, or used as a short-term storage area of various data.
  • the image sensor 205 is a charge coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor for generating moving image data through photoelectric conversion.
  • the image-capturing interface 206 is connected to the image sensor 205 and converts the moving image data acquired from the image sensor 205 into a predetermined format. Then, the image-capturing interface 206 compresses and transfers the converted moving image data to the primary storage device 202 as a moving image frame.
  • the network interface 207 is connected to the client 102 via the network 103 and responsible for communication with the client 102 that requests an image from the imaging apparatus 101.
  • Fig. 3 is a block diagram illustrating a functional configuration example of the imaging apparatus 101 according to the present exemplary embodiment.
  • the imaging apparatus 101 includes a temporary storage unit 301, a distribution unit 302, an imaging processing unit 303, a management unit 304, and a communication unit 305.
  • the temporary storage unit 301 corresponds to the primary storage device 202 in Fig. 2 .
  • the CPU 201 in Fig. 2 executes a program to cause the distribution unit 302, the imaging processing unit 303, the management unit 304, and the communication unit 305 to function as respective processing units.
  • the imaging processing unit 303 acquires a moving image frame generated by the image sensor 205 via the image-capturing interface 206. Then, the imaging processing unit 303 encodes the moving image frame and saves the moving image frame in the temporary storage unit 301 together with moving image meta-information such as a sequence parameter set (SPS) or a picture parameter set (PPS) necessary for reproducing the moving image.
  • moving image meta-information such as a sequence parameter set (SPS) or a picture parameter set (PPS) necessary for reproducing the moving image.
  • SPS sequence parameter set
  • PPS picture parameter set
  • H.264/Advanced Visual Communication (AVC) or H.265/High Efficiency Video Coding (HEVC) can be used as a coding method.
  • the imaging processing unit 303 may select a coding method from a plurality of coding methods to execute encoding.
  • the temporary storage unit 301 temporarily stores the generated moving image meta-information and the moving image frame.
  • the management unit 304 generates initialization segment information and segment information respectively from the moving image meta-information and the moving image frame that are saved by the imaging processing unit 303 in the temporary storage unit 301, and notifies the distribution unit 302 of the initialization segment information and the segment information.
  • the initialization segment information is information used for generating an initialization segment from the moving image meta-information saved in the temporary storage unit 301.
  • the initialization segment information may be a path of the moving image meta-information saved as a file.
  • the segment information is information for generating a segment from the moving image frame saved in the temporary storage unit 301. For example, text in which paths of data files of moving image frames included in the segment is described in an order from a smallest moving image number may be used.
  • the moving image frames included in each segment are determined based on a number of moving image frames (i.e., a number of segment frames) included in the segment, and the number of segment frames is specified when a start request of a moving image stream is output from the distribution unit 302. A specification method thereof will be described specifically when segment time information 414 in Fig. 4 is described.
  • the management unit 304 retains the moving image frame in the temporary storage unit 301 until the distribution unit 302 releases the segment information.
  • the distribution unit 302 distributes a playlist in which information about all of distributable moving image streams is described, and an initialization segment and a segment generated by the management unit 304 to the client 102 via the communication unit 305 in response to a request from the client 102.
  • a playlist an MPD file defined by the International Organization for Standardization/International Electro-Technical Communication (ISO/IEC) 23009-1:2014 is used.
  • the distribution unit 302 generates a header portion of each of the initialization segment and the segment, and combines the header portion with moving image meta-information and a data portion of a moving image frame described in the initialization segment information and the segment information generated by the management unit 304 to generate an initialization segment and a segment. Then, the distribution unit 302 distributes the initialization segment and the segment to the client 102 via the communication unit 305.
  • the management unit 304 may generate moving image frames without receiving a request and sequentially distribute a segment through a Hypertext Transfer Protocol version 2 (HTTP/2) or a Websocket protocol at the time when the segment is completed.
  • HTTP/2 Hypertext Transfer Protocol version 2
  • Fig. 4 is a diagram illustrating a structure example of the MPD file 400 distributed to the client 102 from the imaging apparatus 101.
  • the MPD file 400 is described in the Extensible Markup Language (XML) format, and schema information 401 includes a namespace specified by an xmlns attribute and a schema document specified by an xsi attribute to define a description method of the MPD file 400.
  • XML Extensible Markup Language
  • Profile information 402 specifies a profile "urn:mpeg:dash:profile:isoff-live:2011" defined by the ISO/IEC 23009-1:2014. This profile is designed for DASH live distribution. With this profile, the MPD file 400 can be updated during distribution of the moving image according to a change request of stream setting output to the imaging apparatus 101 from the client 102, an operation status of the imaging apparatus 101 such as a band used rate of the primary storage device 202, or a communication band of the network 103.
  • the imaging processing unit 303 of the imaging apparatus 101 outputs moving image frames of three distributable moving image streams 1, 2, and 3.
  • information about a moving image distributed through the moving image streams 1, 2, and 3 are defined by data structures Representation 405, 406, and 407.
  • the MPD file 400 is a playlist in which information relating to a plurality of moving image streams can be described.
  • the Representation 405 includes coding information 409, resolution information 410, frame rate information 411, and distribution bit rate information 412.
  • the coding information 409 describes H.264/AVC and a main profile level as 3.1 as a coding method of the moving image.
  • Resolution information 410 describes a width and a height (480 and 270, respectively) of the moving image.
  • the frame rate information 411 describes a number of moving image frames (30) displayed per second.
  • the distribution bit rate information 412 describes a value of a distribution bit rate of the moving image (680000 bps).
  • the Representation 405 further includes initialization segment uniform resource identifier (URI) information 416 and segment URI information 415 of the moving image stream 1.
  • the client 102 uses MPDBaseURI information 403 including an IP address of the imaging apparatus 101 and RepresentationBaseURI information 413 including an identifier of the moving image stream 1 to access the following URI. Then, the client 102 acquires an initialization segment and a segment of the moving image stream 1.
  • An actual URI of each of the segments of the moving image stream 1 is "http://192.169.100.1:80/video/stream1/media1_1.m4s", "http://192.169.100.1:80/video/stream1/media1_2.m4s", or "http://192.169.100.1:80/video/stream1/media1_3.m4s".
  • $RepresentationID$ may be used as the Representation ID 408 of the Representation 405.
  • $BandWidth$ may be used for a value of the distribution bit rate 412 of the Representation 405.
  • a combination of these expressions can also be used.
  • the Representation 406 includes coding information 418, resolution information 419, frame rate information 420, distribution bit rate information 421, and segment time information 423.
  • the Representation 406 further includes a Representation ID 417, initialization segment URI information 425, and segment URI information 424.
  • the Representation 407 includes coding information 427, resolution information 428, frame rate information 429, distribution bit rate information 430, and segment time information 432.
  • the Representation 407 further includes a Representation ID 426, initialization segment URI information 434, and segment URI information 433.
  • the Representation 406 and 407 express various values with respect to the moving image streams 2 and 3.
  • the number of moving image streams to be defined does not have to be three, and only one moving image stream may be defined according to a request of the client 102.
  • Fig. 5 is a block diagram illustrating a relationship between the moving image stream 1 and respective segments of the moving image stream 1.
  • the imaging processing unit 303 receives a start request of the moving image stream 1, the imaging processing unit 303 consecutively generates moving image meta-information 501 including the SPS and the PPS and moving image frames 511 to 516, and, at the same time, outputs a generation notification to the management unit 304.
  • Each of the moving image frames 511 and 514 represents an I-frame, which includes moving image information necessary for constituting a complete moving image frame, and the other moving image frames represent P-frames, which include moving image information about a difference with an immediately preceding image.
  • the I-frame is a moving image frame encoded through intraframe coding
  • the P-frame is a moving image frame encoded through interframe coding.
  • the imaging processing unit 303 encodes a captured moving image frame through the intraframe coding or the interframe coding.
  • the imaging processing unit 303 continuously generates moving image frames after generating the moving image frame 516 and outputs a generation notification to the management unit 304 at the same time each of the moving image frame segments is generated.
  • the management unit 304 first generates an initialization segment 500 including moving image meta-information 501, subsequently generates a segment 1-1 520 including moving image frames 511, 512, and 513, and further generates a segment 1-2 530 including moving image frames 514, 515, and 516.
  • the management unit 304 serves as a generation unit, and continuously generates segments with respect to the moving image frames generated by the imaging processing unit 303 after generating the segment 1-2 530.
  • the management unit 304 also divides each of the moving image frames generated by the imaging processing unit 303 into segments according to the specified number of segment frames.
  • a P-frame is the moving image information about a difference
  • a B-frame may be the moving image information about a difference.
  • the first moving image frame included in the segment does not have to be the I-frame.
  • Fig. 6 is a flowchart illustrating a processing example of the management unit 304.
  • the management unit 304 is activated by the imaging processing unit 303 outputting a start request immediately after the imaging processing unit 303 is activated.
  • the management unit 304 waits for occurrence of an event, and switches a processing operation according to the determination of the occurring event. If the management unit 304 determines that the occurring event is a moving image meta-information generation event output from the imaging processing unit 303 ("MOVING IMAGE META-INFORMATION GENERATION EVENT" in step S600), the processing proceeds to step S601.
  • step S600 If the management unit 304 determines that the occurring event is a moving image frame generation event output from the imaging processing unit 303 ("MOVING IMAGE FRAME GENERATION EVENT" in step S600), the processing proceeds to step S605. If the management unit 304 determines that the occurring event is a segment release instruction event output from the distribution unit 302 ("SEGMENT RELEASE INSTRUCTION EVENT" in step S600), the processing proceeds to step S616. If the management unit 304 determines that the occurring event is a moving image stream ending request event output from the distribution unit 302 ("MOVING IMAGE STREAM ENDING REQUEST EVENT" in step S600), the processing proceeds to step S617.
  • step S601 the management unit 304 saves the moving image meta-information.
  • step S602 the management unit 304 generates an initialization segment including the moving image meta-information and outputs a generation notification to the distribution unit 302.
  • step S603 the management unit 304 saves segment structure information such as a number of segment frames specified when a start request of the moving image stream is received.
  • step S604 the management unit 304 sets the own state to an I-frame waiting state. Then, the processing returns to step S600.
  • step S605 the management unit 304 determines the own state. Then, if the management unit 304 determines that the own state is the I-frame waiting state (YES in step 5605), the processing proceeds to step S606. If the management unit 304 determines that the own state is a next frame waiting state (NO in step S605), the processing proceeds to step S612.
  • step S606 the management unit 304 determines whether the generated moving image frame is the I-frame. If the management unit 304 determines that the moving image frame is the I-frame (YES in step S606), the processing proceeds to step S607. If the management unit 304 determines that the moving image frame is not the I-frame (NO in step S606), the processing returns to step S600.
  • step S607 the management unit 304 saves the moving image frame information.
  • step S608 the management unit 304 determines whether the moving image frames of a number equivalent to the number of segment frames included in the segment structure information saved in step S603 have been generated. Then, if the management unit 304 determines that the moving image frames of a number equivalent to the number of segment frames have been generated (YES in step S608), the processing proceeds to step S609. If the management unit 304 determines that the moving image frames of a number equivalent to the number of segment frames have not been generated (NO in step S608), the processing returns to step S600.
  • step S609 the management unit 304 saves a segment number and the moving image frame information included in the segment.
  • step S610 the management unit 304 notifies the distribution unit 302 that the segment has been generated.
  • step S611 the management unit 304 sets the own state to the next frame waiting state. Then, the processing returns to step S600.
  • steps S609 and S610 described above it is important that the management unit 304 should save only the segment information and the moving image frame information and output the notification. If the management unit 304 generates the segment from a moving image data portion of the moving image frame, the temporary storage unit 301 may be insufficient because it is necessary to save the segment for each client 102 in addition to executing copy processing of data.
  • step S612 the management unit 304 saves the moving image frame information.
  • step S613 the management unit 304 determines whether the moving image frames of a number equivalent to the number of segment frames included in the segment structure information saved in step S603 have been generated. Then, if the management unit 304 determines that the moving image frames of a number equivalent to the number of segment frames have been generated (YES in step S613), the processing proceeds to step S614. If the management unit 304 determines that the moving image frames of a number equivalent to the number of segment frames have not been generated (NO in step S613), the processing returns to step S600.
  • step S614 the management unit 304 saves the segment number and the moving image frame information included in the segment.
  • step S615 the management unit 304 notifies the distribution unit 302 that the segment has been generated. Then, the processing returns to step S600.
  • the management unit 304 may determine whether the moving image frames are generated consecutively, and may execute processing of returning the state to the I-frame waiting state again if the moving image streams are not generated consecutively.
  • step S616 the management unit 304 deletes the saved segment information and the moving image frames included in that segment. Then, the processing returns to step S600.
  • step S617 the management unit 304 deletes all of the saved segment information and the moving image frames included in that segment.
  • step S618 the management unit 304 determines whether all of the moving image streams have been ended. If the management unit 304 determines that all of the moving image streams have been ended (YES in step S618), the processing is ended. If the management unit 304 determines that not all of the moving image streams have been ended (NO in step S618), the processing returns to step S600.
  • the management unit 304 when the distribution unit 302 outputs a start request of the moving image stream, the management unit 304 is activated at the same time the imaging processing unit 303 is activated. However, the management unit 304 may be activated constantly, so that the segment information of all of the clients 102 is managed by the single management unit 304.
  • Fig. 7 is a sequence diagram illustrating processing when the client 102 receives a moving image distributed from the imaging apparatus 101.
  • the imaging apparatus 101 includes the imaging processing unit 303 for executing imaging processing, the management unit 304 for generating and managing a segment based on the moving image meta-information and the moving image frame generated by the imaging processing unit 303, and the distribution unit 302 for executing distribution processing with respect to the client 102.
  • step S700 the client 102 transmits to the imaging apparatus 101 a distribution request of the MPD file 400 that collectively includes pieces of moving image information.
  • step S701 when the distribution unit 302 of the imaging apparatus 101 receives the distribution request of the MPD file 400, the distribution unit 302 generates the MPD file 400 that collectively includes distributable moving image streams and distributes the MPD file 400 to the client 102.
  • step S709 the distribution unit 302 outputs a stream start request of a moving image stream 1 described in the MPD file 400, to the imaging processing unit 303. Then, in step S710, the distribution unit 302 outputs a stream start request of a moving image stream 2 described in the MPD file 400, to the imaging processing unit 303. Then, in step S711, the distribution unit 302 outputs a stream start request of a moving image stream 3 described in the MPD file 400, to the imaging processing unit 303.
  • the imaging processing unit 303 When the imaging processing unit 303 receives the start request of the moving image stream 1 in step S709, in step S721, the imaging processing unit 303 generates moving image meta-information including the SPS and the PPS with respect to the moving image stream 1, and outputs a generation notification to the management unit 304.
  • the imaging processing unit 303 receives the start request of the moving image stream 2 in step S710, in step S722, the imaging processing unit 303 generates moving image meta-information including the SPS and the PPS with respect to the moving image stream 2, and outputs a generation notification to the management unit 304.
  • the imaging processing unit 303 When the imaging processing unit 303 receives the start request of the moving image stream 3 in step S711, in step S723, the imaging processing unit 303 generates moving image meta-information including the SPS and the PPS with respect to the moving image stream 3, and outputs a generation notification to the management unit 304. Then, in step S724, the imaging processing unit 303 consecutively generates the moving image frames and outputs a generation notification to the management unit 304.
  • the management unit 304 When the management unit 304 receives the generation notification of the moving image meta-information in step S721, in step S712, the management unit 304 generates an initialization segment 1, and outputs a generation notification to the distribution unit 302 if generation thereof has been completed.
  • the management unit 304 receives the generation notification of the moving image meta-information in step S722, in step S713, the management unit 304 generates an initialization segment 2, and outputs a generation notification to the distribution unit 302 if generation thereof has been completed.
  • the management unit 304 receives the generation notification of the moving image meta-information in step S723, in step S714, the management unit 304 generates an initialization segment 3, and outputs a generation notification to the distribution unit 302 if generation thereof has been completed.
  • the management unit 304 generates a segment based on the segment structure information specified when the start request of the moving image stream is output thereto. More specifically, the management unit 304 generates segments 1-1, 1-2, ..., and so on, with respect to the moving image stream 1, generates segments 2-1, 2-2, ..., and so on, with respect to the moving image stream 2, and generates segments 3-1, 3-2, ..., and so on, with respect to the moving image stream 3. Then, the management unit 304 outputs a generation notification to the distribution unit 302 when generation of each of the segments has been completed.
  • the client 102 analyzes the MPD file 400 acquired in step S701, and selects a moving image stream to be received based on the CPU capability, the display resolution, and the communication band of its own apparatus.
  • the client 102 transmits a distribution request of an initialization segment including information necessary for executing reproduction to the imaging apparatus 101.
  • the client 102 selects the moving image stream 1 and transmits a distribution request of the initialization segment 1 to the imaging apparatus 101.
  • the client 102 may transmit a distribution request of the initialization segment of another moving image stream described in the MPD file 400, or may acquire the initialization segment from the imaging apparatus 101 through a server-push method.
  • step S703 the distribution unit 302 distributes the specified initialization segment to the client 102.
  • the distribution unit 302 distributes the initialization segment 1 to the client 102.
  • step S704 the client 102 transmits a distribution request of a segment 1-1 to the imaging apparatus 101.
  • the distribution unit 302 distributes the segment 1-1 to the client 102 when the segment 1-1 is generated in step S715.
  • step S708 the client 102 reproduces the moving image after receiving the segment 1-1. If necessary, the client 102 may buffer the segment 1-1 for a certain period before starting reproduction of the moving image.
  • step S706 the client 102 transmits a distribution request of a next segment 1-2 to the imaging apparatus 101.
  • the distribution unit 302 distributes the segment 1-2 to the client 102 when the segment 1-2 is generated in step S718.
  • the client 102 After reproducing the moving image frames included in the segment 1-1, the client 102 sequentially reproduces the moving image frames included in the segment 1-2 so as not to interrupt the reproduction of the moving image. Then, the client 102 similarly and sequentially acquires segments from the imaging apparatus 101 and reproduces the moving image.
  • Fig. 8 is a flowchart illustrating a processing example of the distribution unit 302 according to the present exemplary embodiment.
  • the distribution unit 302 waits for occurrence of an event and switches a processing operation based on the determination of the occurring event. If the distribution unit 302 determines that the occurring event is an MPD distribution request event transmitted from the client 102 ("MPD DISTRIBUTION REQUEST EVENT" in step S800), the processing proceeds to step S801. If the distribution unit 302 determines that the occurring event is a segment generation notification event output from the management unit 304 ("SEGMENT GENERATION NOTIFICATION EVENT" in step S800), the processing proceeds to step S806.
  • step S800 If the distribution unit 302 determines that the occurring event is a distribution request of an initialization segment or a segment transmitted from the client 102 ("INITIALIZATION SEGMENT/SEGMENT DISTRIBUTION REQUEST EVENT" in step S800), the processing proceeds to step S807. If the distribution unit 302 determines that the occurring event is a distribution ending request event of a moving image stream ("DISTRIBUTION ENDING REQUEST EVENT" in step S800), the processing proceeds to step S811.
  • step S801 the distribution unit 302 generates the MPD file 400 collectively including information about one or a plurality of currently-distributable moving image streams.
  • step S802 the distribution unit 302 distributes the MPD file 400 to the client 102.
  • step S803 the distribution unit 302 determines whether all of the moving image streams described in the MPD file 400 have been started. Then, if the distribution unit 302 determines that all of the moving image streams have already been started (YES in step S803), the processing proceeds to step S804. If the distribution unit 302 determines that not all of the moving image streams have been started (NO in step S803), the processing proceeds to step S805.
  • step S804 the distribution unit 302 outputs generation requests of I-frames of all of the moving image streams to the imaging processing unit 303, and the processing returns to step S800.
  • step S805 the distribution unit 302 concurrently outputs start requests of all of the moving image streams described in the MPD file 400 to the imaging processing unit 303, and the processing returns to step S800.
  • the distribution unit 302 concurrently outputs the start requests of all of the moving image streams in step S805, as illustrated in Fig. 5
  • the I-frame is generated as a first frame to be generated after start of the moving image stream.
  • the start request of the moving image stream and the generation request of the I-frame may be executed sequentially instead of being executed concurrently with respect to all of the moving image streams.
  • the distribution unit 302 concurrently or sequentially outputs generation requests of the moving image frames of all of the moving image streams included in the MPD file 400 (i.e., playlist) encoded through the intraframe coding to the imaging processing unit 303.
  • step S803 all of the moving image streams have already been started (YES in step S803) in a case where the client 102 has already completed the processing in the first steps S801 to S803 and S805.
  • all of the moving image streams have already been started if a client other than the client 102 has connected to the imaging apparatus 101 and the processing in steps S801 to S803 and S805 is completed.
  • step S806 the distribution unit 302 saves the segment information including the moving image frames. Then, the processing returns to step S800.
  • step S807 the distribution unit 302 determines whether the requested initialization segment or the requested segment has already been generated. Then, if the distribution unit 302 determines that the initialization segment or the segment has already been generated (YES in step S807), the processing proceeds to step S808. If the distribution unit 302 determines that the initialization segment or the segment has not been generated (NO in step S807), the processing proceeds to step S810.
  • step S808 the distribution unit 302 generates a header portion of the segment and distributes the segment together with the header portion attached to a data portion of the moving image frame to the client 102 via the communication unit 305.
  • step S809 the distribution unit 302 instructs the management unit 304 to release the distributed segment. Then, the processing returns to step S800.
  • step S808 it is important that the distribution unit 302 generates only a header portion and distributes a segment while attaching the header portion to a data portion of a moving image frame.
  • step S810 the distribution unit 302 distributes an error response to the client 102. Then, the processing returns to step S800. In the above, although the distribution unit 302 distributes the error response if a requested segment does not exist, the distribution unit 302 may wait until the segment is generated.
  • step S811 the distribution unit 302 outputs an ending request of the moving image stream to the management unit 304 to stop generation of segments of the moving image stream.
  • step S812 the distribution unit 302 determines whether all of the moving image streams have been ended. If the distribution unit 302 determines that all of the moving image streams have been ended (YES in step S812), the processing is ended. If the management unit 304 determines that not all of the moving image streams have been ended (NO in step S812), the processing returns to step S800.
  • event determination in step S800 may be executed continuously until the ending request is received from another processing unit of the imaging apparatus 101.
  • the imaging processing unit 303 encodes the moving image streams included in the MPD file 400 in step S724 before a distribution request of a specific moving image stream is received in step S702. More specifically, the imaging processing unit 303 serves as a coding unit and encodes a moving image frame corresponding to a moving image stream described in the MPD file 400 through intraframe coding.
  • the distribution unit 302 receives a distribution request of a specific moving image stream (i.e., an initialization segment or a segment)
  • the distribution unit 302 distributes a moving image stream corresponding to the specific moving image stream including a moving image frame encoded through intraframe coding.
  • the imaging apparatus 101 starts generation of all of the moving image streams when the distribution request of the MPD file 400 is received, so that the initialization segment or the segment can be transmitted instantly even if the client 102 selects any moving image stream. With this processing, the imaging apparatus 101 can distribute a moving image to the client 102 with a short delay time.
  • Fig. 9 is a flowchart illustrating a part of a processing example of the distribution unit 302 according to a second exemplary embodiment of the present disclosure.
  • the distribution unit 302 executes the processing in steps S900 to S905 in place of the processing in steps S803 to S805 executed by the distribution unit 302 in the first exemplary embodiment. Descriptions of portions common to the first and the present exemplary embodiments are omitted, and portions different in the first and the present exemplary embodiments will be described.
  • Fig. 9 is a flowchart illustrating processing executed by the distribution unit 302, in which a start request of a moving image stream is executed in an order from a lowest distribution bit rate.
  • the distribution unit 302 refers to distribution bit rates 412, 421, and 430 of the moving image streams 1, 2, and 3 described in the MPD file 400, and sorts the ids (identification numbers) of the moving image streams 1 to 3 in an order from the lowest distribution bit rate, using an array OBW[j].
  • any sorting algorithm such as bubble sorting or binary tree sorting may be used as the sorting method.
  • Bit rates of the moving image streams 1, 2, and 3 and a sorting example of the array OBW[j] are illustrated as follows. (Distribution Bit Rate 680000 [bps] of Moving Image Stream 1) ⁇ (Distribution Bit Rate 1360000 [bps] of Moving Image Stream 2) ⁇ (Distribution Bit Rate 5680000 [bps] of Moving Image Stream 3)
  • step S901 the distribution unit 302 executes loop processing of a total number of moving image streams described in the MPD file 400 with respect to a variable j starting from a value 1.
  • the distribution unit 302 ends the loop processing of the total number of moving image streams.
  • step S902 the distribution unit 302 determines whether a moving image stream of the array OBW[j] of the moving image streams described in the MPD file 400 has already been started. If the distribution unit 302 determines that the moving image stream of the array OBW[j] has already been started (YES in step S902), the processing proceeds to step S903. If the distribution unit 302 determines that the moving image stream of the array OBW[j] has not been started (NO in step S902), the processing proceeds to step S905.
  • step S903 the distribution unit 302 outputs a generation request of the I-frame of the moving image stream of the array OBW[j] to the imaging processing unit 303. Then, the processing proceeds to step S904.
  • step S905 the distribution unit 302 outputs a start request of the moving image stream of the array OBW[j] to the imaging processing unit 303. Then, the processing proceeds to step S904. At that time, the distribution unit 302 outputs the start request of the moving image stream in step S905, and, as illustrated in Fig. 5 , the I-frame is generated as a first frame to be generated after start of the moving image stream.
  • step S904 if the variable j is not equal to the total number of moving image streams, the distribution unit 302 increments the variable j, and the processing returns to step S901. If the variable j is equal to the total number of moving image streams, the processing returns to step S800 in Fig. 8 .
  • the distribution unit 302 outputs generation requests of moving image frames of all of the moving image streams described in the MPD file 400 encoded through intraframe coding to the imaging processing unit 303 in an order from the lowest distribution bit rate.
  • the imaging apparatus 101 outputs the start requests of the moving image streams in an order from the lowest distribution bit rate when the distribution request of the MPD file 400 is received. With this configuration, it is expected that a starting preparation of the moving image stream from data having a small processing data size per unit time is completed while the imaging apparatus 101 distributes the moving image to the client 102 with a short delay time. Therefore, a memory used amount per unit time of the imaging apparatus 101 can be reduced.
  • Fig. 10 is a flowchart illustrating a part of a processing example of the distribution unit 302 according to a third exemplary embodiment of the present disclosure.
  • the distribution unit 302 executes the processing in steps S1000 to S1006 in place of the processing in steps S803 to S805 executed by the distribution unit 302 in the first exemplary embodiment. Descriptions of portions common to the first and the present exemplary embodiments are omitted, and portions different in the first and the present exemplary embodiments will be described.
  • Fig. 10 is a flowchart illustrating processing executed by the distribution unit 302, in which a start request of a moving image stream is output in an order from a highest distribution frequency.
  • the distribution unit 302 reads out distribution history information (described below) of each of the moving image streams 1, 2, and 3 saved in the temporary storage unit 301, calculates a distribution frequency evaluation value val(i) of each moving image stream "i", and stores the distribution frequency evaluation value val(i) in the temporary storage unit 301.
  • a calculation formula of a total distribution time of the moving image stream i from a time T m to a time T M is expressed as a formula 1 as an example of the distribution frequency evaluation value val(i) of the moving image stream i.
  • strm s i , T m ⁇ T o s , T c s ⁇ T M
  • a symbol “s” represents a structure including distribution meta-information of a series of moving image stream distribution.
  • a symbol “strm(s)” represents a moving image stream id of a series of moving image stream distribution.
  • a symbol “To(s)” represents a distribution start time of a series of moving image stream distribution.
  • a symbol “Tc(s)” represents a distribution ending time of a series of moving image stream distribution.
  • the distribution frequency evaluation value val(i) of the moving image stream i is defined by the formula 1, another definition using a number of times of access from the client 102 to the moving image stream i may be also used.
  • step S1001 the distribution unit 302 refers to the distribution frequency evaluation value val(i) of each of the moving image streams 1, 2, and 3 described in the MPD file 400 from the temporary storage unit 301. Then, the distribution unit 302 sorts the ids of the moving image streams using an array OTM[j] in an order from the highest distribution frequency evaluation value val(i). In addition, any sorting algorithm such as bubble sorting or binary tree sorting may be used as the sorting method.
  • step S1002 the distribution unit 302 executes loop processing of a total number of moving image streams in the MPD file 400 with respect to a variable j starting from a value 1.
  • the distribution unit 302 ends the loop processing of the total number of moving image streams.
  • step S1003 the distribution unit 302 determines whether a moving image stream of the array OTM[j] described in the MPD file 400 has already been started. If the distribution unit 302 determines that the moving image stream of the array OTM[j] has already been started (YES in step S1003), the processing proceeds to step S1004. If the distribution unit 302 determines that the moving image stream of the array OTM[j] has not been started (NO in step S1003), the processing proceeds to step S1006.
  • step S1004 the distribution unit 302 outputs a generation request of the I-frame of the moving image stream of the array OTM[j] to the imaging processing unit 303. Then, the processing proceeds to step S1005.
  • step S1006 the distribution unit 302 outputs a start request of the moving image stream of the array OTM[j] to the imaging processing unit 303. Then, the processing proceeds to step S1005. At that time, the distribution unit 302 outputs the start request of the moving image stream in step S1006, and, as illustrated in Fig. 5 , the I-frame is generated as a first frame to be generated after start of the moving image stream.
  • step S1005 if the variable j is not equal to the total number of moving image streams, the distribution unit 302 increments the variable j, and the processing returns to step S1002. If the variable j is equal to the total number of moving image streams, the processing returns to step S800.
  • Fig. 11 is a flowchart illustrating another part of the processing example of the distribution unit 302 according to the third exemplary embodiment of the present disclosure.
  • the distribution unit 302 executes the processing in steps S1100 and S1101 in place of the processing in step S811 executed by the distribution unit 302 in the first exemplary embodiment.
  • Fig. 11 is a flowchart illustrating processing by the distribution unit 302 for ending a moving image stream and saving a stream distribution history information.
  • the distribution unit 302 outputs to the management unit 304 an ending request of the moving image stream for which a distribution ending request is received from the client 102, and stops generation of segments of the moving image stream.
  • the distribution unit 302 saves the distribution history information of the moving image stream ended in step S1100 in the temporary storage unit 301. For example, as the distribution history information of the moving image stream, a distribution starting time and a distribution ending time of the moving image stream may be saved for a series of moving image stream distribution.
  • the distribution unit 302 outputs generation requests of moving image frames of all of the moving image streams described in the MPD file 400 encoded through the intraframe coding to the imaging processing unit 303 in an order from the highest distribution frequency.
  • the imaging apparatus 101 when the distribution request of the MPD file 400 is received, the imaging apparatus 101 outputs a start request of the moving image stream in an order from the highest distribution frequency, so that a moving image can be distributed to the client 102 with a short delay time. Further, there is a case where the imaging apparatus 101 cannot capture images of a part of the moving image stream at a specified frame rate (i.e., dropping frame) because of a high load applied to the CPU 201 or an increase of a memory used amount. Even in such a case, the imaging apparatus 101 can distribute the moving image stream with high distribution frequency without dropping frames, so that the client 102 can smoothly execute reproduction of the moving image.
  • a specified frame rate i.e., dropping frame
  • Fig. 12 is a flowchart illustrating a processing example of the distribution unit 302 according to a fourth exemplary embodiment of the present disclosure.
  • the imaging apparatus 101 generates I-frames of all of the moving image streams in a case where the imaging apparatus 101 does not receive the distribution request event of the initialization segment from the client 102 even though a certain time period has passed after receiving the distribution request of the MPD file 400.
  • the distribution unit 302 waits for occurrence of an event and switches a processing operation based on the determination of the occurring event. If the distribution unit 302 determines that the occurring event is an MPD distribution request event transmitted from the client 102 ("MPD DISTRIBUTION REQUEST EVENT" in step S1200), the processing proceeds to step S1201.
  • MPD DISTRIBUTION REQUEST EVENT MPD DISTRIBUTION REQUEST EVENT
  • step S1207 If the distribution unit 302 determines that the occurring event is a segment generation notification event output from the management unit 304 ("SEGMENT GENERATION NOTIFICATION EVENT" in step S1200), the processing proceeds to step S1207. If the distribution unit 302 determines that the occurring event is an initialization segment/segment distribution event transmitted from the client 102 ("INITIALIZATION SEGMENT/SEGMENT DISTRIBUTION REQUEST EVENT" in step S1200), the processing proceeds to step S1208. If the distribution unit 302 determines that the occurring event is a distribution ending request event of a moving image stream ("DISTRIBUTION ENDING REQUEST EVENT" in step S1200), the processing proceeds to step S1214.
  • step S1201 the distribution unit 302 generates the MPD file 400 collectively including information about one or a plurality of currently-distributable moving image streams.
  • step S1202 the distribution unit 302 distributes the MPD file 400 to the client 102.
  • step S1203 the distribution unit 302 determines whether all of the moving image streams described in the MPD file 400 have already been started. If the distribution unit 302 determines that all of the moving image streams have already been started (YES in step S1203), the processing proceeds to step S1204. If the distribution unit 302 determines that not all of the moving image streams have been started (NO in step S1203), the processing proceeds to step S1205.
  • step S1204 the distribution unit 302 outputs generation requests of I-frames of all of the moving image streams to the imaging processing unit 303. Then, the processing proceeds to step S1206.
  • step S1205 the distribution unit 302 concurrently outputs the start requests of all of the moving image streams described in the MPD file 400 to the imaging processing unit 303. Then, the processing proceeds to step S1206.
  • the distribution unit 302 concurrently outputs the start requests of all of the moving image streams in step S1205, as illustrated in Fig. 5
  • the I-frame is generated as a first frame to be generated after start of the moving image stream.
  • the start request of the moving image stream and the generation request of the I-frame may be executed sequentially instead of being executed concurrently with respect to all of the moving image streams.
  • step S1203 all of the moving image streams have already been started (YES in step S1203) in a case where the client 102 has already completed the processing in the first steps S1201 to S1203 and S1205.
  • all of the moving image streams have already been started if a client other than the client 102 has connected to the imaging apparatus 101 and processing in the steps S1201 to S1203 and S1205 has been completed.
  • step S1206 the distribution unit 302 sets a timer count value T to 0, and starts counting up a timer. Then, the processing returns to step S1200.
  • step S1200 if the distribution unit 302 determines that the timer count value T is the threshold value or more ("T IS THRESHOLD VALUE OR MORE" in step S1200), the processing proceeds to step S1203.
  • step S1207 the distribution unit 302 saves the segment information including moving image frames. Then, the processing returns to step S1200.
  • step S1208 the distribution unit 302 determines whether the requested initialization segment or the requested segment has already been generated. If the distribution unit 302 determines that the initialization segment or the segment has already been generated (YES in step S1208), the processing proceeds to step S1209. If the distribution unit 302 determines that the initialization segment or the segment has not been generated (NO in step S1208), the processing proceeds to step S1213.
  • step S1209 the distribution unit 302 determines whether the occurring event is a distribution request event of the initialization segment. If the distribution unit 302 determines that the occurring event is a distribution request event of the initialization segment (YES in step S1209), the processing proceeds to step S1210. If the distribution unit 302 determines that the occurring event is not a distribution request event of the initialization segment (NO in step S1209), the processing proceeds to step S1211.
  • step S1210 the distribution unit 302 ends counting of the timer.
  • step S1211 the distribution unit 302 generates a header portion of the segment and distributes the segment with the header portion attached to a data portion of the moving image frame, to the client 102 via the communication unit 305.
  • step S1212 the distribution unit 302 instructs the management unit 304 to release the distributed segment. Then, the processing returns to step S1200.
  • step S1213 the distribution unit 302 distributes an error response to the client 102. Then, the processing returns to step S1200.
  • step S1214 the distribution unit 302 outputs an ending request of the moving image stream to the management unit 304 to stop generation of segments of the moving image stream.
  • step S1215 the distribution unit 302 determines whether all of the moving image streams have been ended. If the distribution unit 302 determines that all of the moving image streams have been ended (YES in step S1215), the processing is ended. If the distribution unit 302 determines that not all of the moving image streams have been ended (NO in step S1215), the processing returns to step S1200. In addition, although the processing is ended when the distribution unit 302 determines that all of the moving image streams have been ended in step S1215, event determination in step S1200 may be executed continuously until the ending request is received from another processing unit of the imaging apparatus 101.
  • start request of the moving image stream and a generation request of the I-frame are concurrently output with respect to all of the moving image streams
  • start request and the generation request may be sequentially output to each of the moving image streams.
  • the distribution unit 302 sets the timer count value T to 0 in step S1206 after outputting generation requests of moving image frames of all of the moving image streams described in the MPD file 400 encoded through the intraframe coding, to the imaging processing unit 303. Then, in a case where a waiting time for receiving the distribution request of the specific moving image stream (i.e., an initialization segment or a segment) has become a threshold value or more, the processing proceeds to step S1203. In step S1204 or S1205, the distribution unit 302 outputs generation requests of moving image frames of all of the moving image streams described in the MPD file 400 encoded through the intraframe coding to the imaging processing unit 303 again.
  • the imaging apparatus 101 generates the I-frames of all of the moving image streams at each threshold time interval until the imaging apparatus 101 receives a distribution request of the initialization segment after receiving a distribution request of the MPD file 400.
  • the imaging apparatus 101 can distribute a moving image to the client 102 with a short delay time.
  • Fig. 13 is a sequence diagram illustrating an example of a mutual interaction between the imaging apparatus 101 and the client 102 for executing distribution of a moving image according to a fifth exemplary embodiment of the present disclosure.
  • the client 102 receives a moving image distributed from the imaging apparatus 101.
  • the present exemplary embodiment is different from the first exemplary embodiment in the sequence of the distribution unit 302 in steps S1300 and S1301, the sequence of the management unit 304 in step S718 and the subsequent steps for generating segments, and the sequence of the imaging processing unit 303 in step S1302 and the subsequent steps for generating moving image frames. Descriptions of portions common to the first and the present exemplary embodiments are omitted, and portions different in the first and the present exemplary embodiments will be described.
  • step S704 the client 102 transmits a distribution request of the segment 1-1 of the moving image stream 1 to the imaging apparatus 101.
  • the distribution unit 301 receives the distribution request of the segment 1-1 of the moving image stream 1, and specifies the moving image streams 2 and 3 from the MPD file 400 as the moving image streams other than the moving image stream 1.
  • step S1300 the distribution unit 302 outputs an ending request of the specified moving image stream 2 to the imaging processing unit 303 and the management unit 304.
  • step S1301 the distribution unit 302 outputs an ending request of the specified moving image stream 3 to the imaging processing unit 303 and the management unit 304.
  • the imaging processing unit 303 receives the ending requests of the moving image streams 2 and 3. Then, the imaging processing unit 303 ends generation of moving image frames of the moving image streams 2 and 3 from among generation of moving image frames executed in step S724 and the subsequent steps, and continuously executes generation of moving image frames of the moving image stream 1 in step S1302 and the subsequent steps.
  • the management unit 304 receives the ending requests of the moving image streams 2 and 3 in steps S1300 and S1301. Then, the management unit 304 ends generation of segments of the moving image streams 2 and 3. From the sequence in step S718 onward, the management unit 304 continuously generates segments of the moving image stream 1.
  • Fig. 14 is a flowchart illustrating processing executed when the distribution unit 302 receives a distribution request of an initialization segment.
  • the distribution unit 302 When a distribution request of an initialization segment is received from the client 102, the distribution unit 302 outputs an ending request of a moving image stream other than a moving image stream corresponding to that initialization segment.
  • the distribution unit 302 additionally executes the processing in steps 1400 to S1405 at the timing between the processing of steps S807 and S808 executed by the distribution unit 302 in the first exemplary embodiment. Descriptions of portions common to the first and the present exemplary embodiments are omitted, and portions different in the first and the present exemplary embodiments will be described.
  • step S1400 the distribution unit 302 determines whether the occurring event is a distribution request event of the initialization segment. If the distribution unit 302 determines that the occurring event is a distribution request event of the initialization segment (YES in step S1400), the processing proceeds to step S1401. If the distribution unit 302 determines that the occurring event is not a distribution request event of the initialization segment (NO in step S1400), the processing proceeds to step S808 in Fig. 8 .
  • step S1401 the distribution unit 302 saves distribution stream information including an id of the moving image stream in the temporary storage unit 301.
  • step S1402 the distribution unit 302 executes loop processing of a total number of moving image streams in the MPD file 400 with respect to a variable i starting from a value 1.
  • the distribution unit 302 ends the loop processing of the total number of moving image streams.
  • step S1403 the distribution unit 302 reads out the distribution stream information saved in the temporary storage unit 301 and determines whether the distribution stream information includes the id of the moving image stream i.
  • step S1403 determines that the distribution stream information includes the id of the moving image stream i (YES in step S1403). If the distribution unit 302 determines that the distribution stream information does not include the id of the moving image stream i (NO in step S1403), the processing proceeds to step S1405. In step S1405, the distribution unit 302 outputs the ending request of the moving image stream i to the imaging processing unit 303, and the processing proceeds to step S1404. In step S1404, if the variable i is not equal to the total number of moving image streams, the distribution unit 302 increments the variable i, and the processing returns to step S1402. If the variable i is equal to the total number of moving image streams, the processing proceeds to step S808.
  • step S1400 the distribution unit 302 determines whether the occurring event is the distribution request event of the initialization segment.
  • the client 102 or a client other than the client 102 transmits a distribution request of an initialization segment of a moving image stream other than the moving image stream 1 at a timing between steps S1301 and S1302.
  • the distribution unit 302 may determine whether the occurring event is a distribution request event of the first segment.
  • the distribution unit 302 receives a distribution request of a specific moving image stream (an initialization segment or a segment) and outputs generation ending requests of moving image streams other than the specific moving image stream in the MPD file 400 to the imaging processing unit 303.
  • the imaging apparatus 101 starts all of the moving image streams when a distribution request of the MPD file 400 is received, and ends the moving image streams other than the moving image stream to be distributed when the client 102 transmits a distribution request of the initialization segment.
  • the imaging apparatus 101 can distribute a moving image to the client 102 with a short delay time while reducing a memory used amount per unit time.
  • Fig. 15 is a flowchart illustrating a processing example when the distribution unit 302 according to a sixth exemplary embodiment of the present disclosure receives an MPD distribution request event.
  • the distribution unit 302 executes the processing in steps S1500 to S1505 in place of the processing in steps S803 to S805 executed by the distribution unit 302 in the first exemplary embodiment. Descriptions of portions common to the first and the present exemplary embodiments are omitted, and portions different in the first and the present exemplary embodiments will be described.
  • Fig. 15 is a flowchart illustrating processing for outputting a start request of a moving image stream executed by the distribution unit 302 when the MPD file 400 defines only one moving image stream.
  • the distribution unit 302 determines whether the number of moving image streams defined in the MPD file 400 is one. If the distribution unit 302 determines that the number of moving image streams defined in the MPD file 400 is one (YES in step S1500), the processing proceeds to step S1501. If the distribution unit 302 determines that the number of moving image streams defined in the MPD file 400 is not one (NO in step S1500), the processing proceeds to step S1505.
  • step S1501 the distribution unit 302 sets a multiple-stream flag to OFF.
  • step S1502 the distribution unit 302 determines whether the moving image stream described in the MPD file 400 has been started already. If the distribution unit 302 determines that the moving image stream has already been started (YES in step S1502), the processing proceeds to step S1503. If the distribution unit 302 determines that the moving image stream has not been started (NO in step S1502), the processing proceeds to step S1504.
  • step S1503 the distribution unit 302 outputs a generation request of an I-frame of the moving image stream to the imaging processing unit 303, and the processing returns to step S800 in Fig. 8 .
  • step S1504 the distribution unit 302 outputs a start request of the moving image stream to the imaging processing unit 303, and the processing returns to step S800 in Fig. 8 .
  • the distribution unit 302 outputs the start request of the moving image stream in step S1504, and, as illustrated in Fig. 5 , the I-frame is generated as a first frame to be generated after start of the moving image stream.
  • the moving image stream has already been started if the client 102 has already completed the processing in the first steps S1500 to S1502 and S1504.
  • the moving image stream has already been started if a client other than the client 102 has connected to the imaging apparatus 101 and the processing in steps S1500 to S1502 and S1504 has been completed.
  • the distribution unit 302 sets the multiple-stream flag to ON, and the processing returns to step S800 in Fig. 8 .
  • Fig. 16 is a flowchart illustrating an example of processing to be executed when the distribution unit 302 receives a distribution request event of an initialization segment.
  • the distribution unit 302 additionally executes the processing in steps S1600 to S1604 immediately before executing the processing in step S807 executed by the distribution unit 302 in the first exemplary embodiment.
  • the distribution unit 302 outputs a start request of a moving image stream when a distribution request event of an initialization segment is received from the client 102.
  • step S1600 the distribution unit 302 determines whether the occurring event is a distribution request event of an initialization segment. If the distribution unit 302 determines that the occurring event is a distribution request event of an initialization segment (YES in step S1600), the processing proceeds to step S1601. If the distribution unit 302 determines that the occurring event is not a distribution request event of an initialization segment (NO in step S1600), the processing proceeds to step S807 in Fig. 8 .
  • step S1601 the distribution unit 302 determines whether the multiple-stream flag is ON. If the distribution unit 302 determines that the multiple-stream flag is ON (YES in step S1601), the processing proceeds to step S1602. If the distribution unit 302 determines that the multiple-stream flag is not ON (NO in step S1601), the processing returns to step S800 in Fig. 8 .
  • step S1602 the distribution unit 302 determines whether the moving image stream of the initialization segment determined in step S1600 has already been started. If the distribution unit 302 determines that the moving image stream has already been started (YES in step S1602), the processing proceeds to step S1603. If the distribution unit 302 determines that the moving image stream has not been started (NO in step S1602), the processing proceeds to step S1604.
  • step S1603 the distribution unit 302 outputs to the imaging processing unit 303 a generation request of the I-frame of the moving image stream of the initialization segment determined in step S1600, and the processing proceeds to step S807 in Fig. 8 .
  • step S1604 the distribution unit 302 outputs to the imaging processing unit 303 a start request of the moving image stream of the initialization segment determined in step S1600, and the processing proceeds to step S807 in Fig. 8 .
  • the distribution unit 302 outputs the start request of the moving image stream in step S1604, and, as illustrated in Fig. 5 , the I-frame is generated as a first frame to be generated after start of the moving image stream.
  • the distribution unit 302 outputs a generation request of the I-frame of the one moving image stream before receiving a distribution request of the specific moving image stream (an initialization segment or a segment). More specifically, the distribution unit 302 outputs generation requests of moving image frames of one moving image stream described in the MPD file 400 encoded through the intraframe coding, to the imaging processing unit 303.
  • the distribution unit 302 outputs a generation request of the I-frame of the specific moving image stream after receiving a distribution request of the specific moving image stream (an initialization segment or a segment). More specifically, the distribution unit 302 outputs a generation request of a moving image frame of the specific moving image stream encoded through the intraframe coding to the imaging processing unit 303.
  • the imaging apparatus 101 starts the moving image stream when the moving image stream to be distributed to the client 102 is specified, a memory used amount of the imaging apparatus 101 can be minimized.
  • DASH distribution is taken as an example
  • present disclosure is also applicable to another adaptive streaming technique such as HLS distribution.
  • the present disclosure can be realized in such a manner that a program for realizing one or more functions in the above-described exemplary embodiments is supplied to a system or an apparatus via a network or a storage medium, and one or more processors in the system or the apparatus reads and executes the program. Further, the present disclosure can be also realized with a circuit (e.g., application specific integrated circuit (ASIC)) that realizes one or more functions.
  • ASIC application specific integrated circuit
  • a delay time occurring between receipt of a distribution request of a moving image stream and distribution of the moving image stream can be reduced.
  • Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s).
  • computer executable instructions e.g., one or more programs
  • a storage medium which may also be referred to more fully as
  • the computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions.
  • the computer executable instructions may be provided to the computer, for example, from a network or the storage medium.
  • the storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)TM), a flash memory device, a memory card, and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Library & Information Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
EP18169475.3A 2017-06-20 2018-04-26 Imaging apparatus, processing method for imaging apparatus, and storage medium Active EP3419294B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2017120595A JP6797755B2 (ja) 2017-06-20 2017-06-20 撮像装置、撮像装置の処理方法およびプログラム

Publications (2)

Publication Number Publication Date
EP3419294A1 EP3419294A1 (en) 2018-12-26
EP3419294B1 true EP3419294B1 (en) 2020-11-18

Family

ID=62067472

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18169475.3A Active EP3419294B1 (en) 2017-06-20 2018-04-26 Imaging apparatus, processing method for imaging apparatus, and storage medium

Country Status (4)

Country Link
US (1) US10951887B2 (enExample)
EP (1) EP3419294B1 (enExample)
JP (1) JP6797755B2 (enExample)
CN (1) CN109104552B (enExample)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7327953B2 (ja) * 2019-03-11 2023-08-16 キヤノン株式会社 動画配信装置、動画配信方法及びプログラム
EP3761654B1 (en) * 2019-07-04 2025-11-05 Dolby International AB Media streaming

Family Cites Families (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10257466A (ja) * 1997-03-12 1998-09-25 Matsushita Electric Ind Co Ltd 有料放送送信システムおよび方法
JP2000312360A (ja) * 1999-04-27 2000-11-07 Matsushita Electric Ind Co Ltd 情報提供装置
JP4721028B2 (ja) 2001-05-22 2011-07-13 株式会社メガチップス 動画像配信システム、動画像配信方法及びそれらのプログラム
US9314691B2 (en) 2002-12-10 2016-04-19 Sony Computer Entertainment America Llc System and method for compressing video frames or portions thereof based on feedback information from a client device
US7936819B2 (en) * 2005-07-08 2011-05-03 Tag Networks, Inc. Video encoder with latency control
JP2008079061A (ja) * 2006-09-22 2008-04-03 Matsushita Electric Ind Co Ltd 画像ストリーミングシステム
EP2201723B1 (en) 2007-09-11 2011-01-12 Nxp B.V. Method of transcoding content, transcoder unit, and UPnP network system
GB201105502D0 (en) * 2010-04-01 2011-05-18 Apple Inc Real time or near real time streaming
JP2014509109A (ja) * 2011-01-11 2014-04-10 アップル インコーポレイテッド リアルタイム又はリアルタイムに近いストリーミング
WO2012135804A2 (en) * 2011-04-01 2012-10-04 Mixaroo, Inc. System and method for real-time processing, storage, indexing, and delivery of segmented video
US8813116B2 (en) * 2011-04-27 2014-08-19 Morega Systems Inc. Adaptive video server with virtual file system and methods for use therewith
SG11201408361TA (en) * 2012-06-22 2015-01-29 Panasonic Ip Corp America Image coding method, image decoding method, image coding apparatus, image decoding apparatus, and image coding and decoding apparatus
JP6139872B2 (ja) * 2012-12-10 2017-05-31 キヤノン株式会社 情報処理装置及びその制御方法、プログラム、記憶媒体、並びに、映像処理システム
JP6116240B2 (ja) * 2012-12-28 2017-04-19 キヤノン株式会社 送信装置、送信方法、及びプログラム
US9191725B2 (en) * 2013-03-15 2015-11-17 Arris Technology, Inc. Method and apparatus for streaming video
US8869218B2 (en) * 2013-03-15 2014-10-21 Wowza Media Systems, LLC On the fly transcoding of video on demand content for adaptive streaming
US9674251B2 (en) * 2013-06-17 2017-06-06 Qualcomm Incorporated Mediating content delivery via one or more services
US9756369B2 (en) * 2013-06-19 2017-09-05 Telefonaktiebolaget Lm Ericsson (Publ) Method and apparatus for streaming media data segments of different lengths wherein the segment of different length comprising data not belonging to the actual segment and beginning with key frames or containing key frames only
KR102264477B1 (ko) 2013-07-12 2021-06-15 캐논 가부시끼가이샤 푸시 메시지 제어를 이용하는 적응적 데이터 스트리밍 방법
US9462306B2 (en) 2013-07-16 2016-10-04 The Hong Kong University Of Science And Technology Stream-switching in a content distribution system
US20160073106A1 (en) 2014-09-08 2016-03-10 Apple Inc. Techniques for adaptive video streaming
CN104735470B (zh) 2015-02-11 2018-06-19 海信集团有限公司 一种流媒体数据传输方法及装置
US10051294B2 (en) * 2015-03-31 2018-08-14 Avago Technologies General Ip (Singapore) Pte. Ltd. Compressed video buffering
US10114835B2 (en) * 2015-04-29 2018-10-30 Box, Inc. Virtual file system for cloud-based shared content
EP3107261B1 (en) * 2015-06-18 2020-01-29 Sky Italia S.R.L. System, method and devices for low latency transmission
US10582201B2 (en) * 2016-05-19 2020-03-03 Qualcomm Incorporated Most-interested region in an image
US10148989B2 (en) * 2016-06-15 2018-12-04 Divx, Llc Systems and methods for encoding video content
US10063612B2 (en) * 2016-09-30 2018-08-28 Amazon Technologies, Inc. Request-based encoding for streaming content portions

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
CN109104552A (zh) 2018-12-28
US20180367792A1 (en) 2018-12-20
CN109104552B (zh) 2021-10-08
JP2019009492A (ja) 2019-01-17
EP3419294A1 (en) 2018-12-26
JP6797755B2 (ja) 2020-12-09
US10951887B2 (en) 2021-03-16

Similar Documents

Publication Publication Date Title
CN108605160B (zh) 信息处理设备和信息处理方法
KR102324326B1 (ko) 상이한 인코딩 파라미터를 이용해 인코딩되는 복수의 인코딩 스트리밍
US9967598B2 (en) Adaptive streaming method and apparatus
CN101287106B (zh) 影像分发装置、影像接收装置以及关键帧分发方法
US20080143875A1 (en) Method and system for synchronous video capture and output
KR102394959B1 (ko) 멀티미디어 데이터를 관리하는 방법 및 장치
US11102527B2 (en) Video distribution apparatus, video reception apparatus, video distribution method, and recording medium
EP3419294B1 (en) Imaging apparatus, processing method for imaging apparatus, and storage medium
US11496535B2 (en) Video data transmission apparatus, video data transmitting method, and storage medium
JP6193569B2 (ja) 受信装置、受信方法、及びプログラム、撮像装置、撮像方法、及びプログラム、送信装置、送信方法、及びプログラム
JP2020113922A (ja) 動画配信装置、動画配信方法及びプログラム
CN113542747B (zh) 服务器装置、通信系统以及存储介质
US10778740B2 (en) Video image distribution apparatus, control method, and recording medium
KR20210004877A (ko) 미디어 데이터 프로세싱 방법 및 장치
JP6400163B2 (ja) 受信装置、受信方法、送信装置、送信方法、及びプログラム
US11303940B2 (en) Transmission apparatus, transmission method, and non-transitory computer-readable storage medium
US20240086451A1 (en) Information processing apparatus, reception apparatus, information processing method, and storage medium
US11523156B2 (en) Method and system for distributing an audiovisual content
KR20170120972A (ko) 네트워크 상에서 실시간 멀티미디어 서비스의 품질을 제어하는 방법 및 이를 위한 장치
JP2018113568A (ja) 送信装置、送信方法、およびプログラム
CN113315997A (zh) 发送装置、服务器装置、发送方法以及程序

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17P Request for examination filed

Effective date: 20190626

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

17Q First examination report despatched

Effective date: 20190801

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20200610

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018009744

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1337036

Country of ref document: AT

Kind code of ref document: T

Effective date: 20201215

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1337036

Country of ref document: AT

Kind code of ref document: T

Effective date: 20201118

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20201118

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210219

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210218

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210318

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210218

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210318

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018009744

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20210819

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210426

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20210430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210430

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210426

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210318

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20180426

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20250319

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20250319

Year of fee payment: 8

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250319

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201118