WO2023053687A1 - 画像処理方法、画像処理システム、画像処理装置、及び、サーバ - Google Patents

画像処理方法、画像処理システム、画像処理装置、及び、サーバ Download PDF

Info

Publication number
WO2023053687A1
WO2023053687A1 PCT/JP2022/028252 JP2022028252W WO2023053687A1 WO 2023053687 A1 WO2023053687 A1 WO 2023053687A1 JP 2022028252 W JP2022028252 W JP 2022028252W WO 2023053687 A1 WO2023053687 A1 WO 2023053687A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
background
subject
image processing
compressed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/028252
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
健介 中井
一真 元田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Intellectual Property Management Co Ltd
Original Assignee
Panasonic Intellectual Property Management Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Panasonic Intellectual Property Management Co Ltd filed Critical Panasonic Intellectual Property Management Co Ltd
Priority to EP22875564.1A priority Critical patent/EP4412201A4/en
Priority to JP2023550408A priority patent/JPWO2023053687A1/ja
Priority to CN202280065153.7A priority patent/CN118020292A/zh
Publication of WO2023053687A1 publication Critical patent/WO2023053687A1/ja
Priority to US18/612,493 priority patent/US12495149B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/115—Selection of the code volume for a coding unit prior to coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/119—Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/167—Position within a video image, e.g. region of interest [ROI]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/20—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding
    • H04N19/23—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using video object coding with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Definitions

  • the present disclosure relates to image processing methods, image processing systems, image processing apparatuses, and servers.
  • Patent Document 1 discloses a technique for extracting a subject from an image containing the subject using a background image.
  • the present disclosure provides an image processing method and the like capable of suppressing deterioration of an image of a subject due to compression.
  • An image processing method is an image processing method for extracting a subject from an image including the subject using a background image, and is obtained by removing a background area not including the subject in the image from the image. compressing the background-removed image; transmitting the compressed background-removed image over a network; receiving the compressed background-removed image over the network; The compressed background-removed image is decoded, the decoded background-removed image and the background image are synthesized to generate a restored image, and the restored image and the background image are compared to extract the subject. .
  • an image processing system is an image processing system that extracts a subject from an image including the subject using a background image
  • the image processing system includes an image processing device and a server that are connected to each other via a network.
  • the image processing apparatus comprising: a first generating unit that generates a background-removed image obtained by removing a background area that does not include the subject in the image; and a compression unit that compresses the background-removed image.
  • the server includes a receiving unit configured to receive the compressed background-removed image via the network; a decoding unit that decodes the compressed background-removed image; a second generation unit that combines the decoded background-removed image and the background image to generate a restored image; and the restored image and the background image. and an extracting unit that compares and extracts the subject.
  • an image processing method is an image processing system that includes an image processing device that transmits a compressed image including a subject, and a server that extracts the subject from the compressed image using a background image.
  • An image processing method by an image processing device comprising: generating a background-removed image obtained by removing a background area not including a subject in an image including the subject from the image; compressing the background-removed image; The background-removed image thus obtained is transmitted to the server via a network.
  • the image processing apparatus is an image processing system that includes an image processing apparatus that transmits a compressed image including a subject, and a server that extracts the subject from the compressed image using a background image.
  • An image processing apparatus comprising: a generation unit for generating a background-removed image obtained by removing a background area not including the subject from the image including the subject; and a compression unit for compressing the background-removed image. and a transmitting unit configured to transmit the compressed background-removed image over a network.
  • an image processing method is an image processing system that includes an image processing device that transmits a compressed image including a subject, and a server that extracts the subject from the compressed image using a background image.
  • An image processing method by a server comprising: receiving the compressed background-removed image via a network; decoding the received compressed background-removed image; a restored image is generated by synthesis, the restored image and the background image are compared to extract the subject, and the compressed background-removed image is obtained by removing a background region of the image that does not include the subject from the image. This is an image obtained by compressing the background-removed image obtained by removing.
  • the server in the present disclosure is the server in an image processing system including an image processing device that transmits a compressed image including a subject, and a server that extracts the subject from the compressed image using a background image.
  • a receiving unit for receiving the compressed background-removed image via a network; a decoding unit for decoding the received compressed background-removed image; and the decoded background-removed image and the background image.
  • a generating unit that generates a restored image by combining; and an extracting unit that compares the restored image and the background image to extract the subject, wherein the compressed background-removed image is the subject in the image.
  • This is an image obtained by compressing a background-removed image obtained by removing a background region that does not contain from the image.
  • the image processing method and the like according to the present disclosure can suppress deterioration of the image of the subject due to compression.
  • FIG. 1 is a schematic diagram of an image processing system according to an embodiment.
  • FIG. 2 is a diagram illustrating an example of the functional configuration of the image processing apparatus according to the embodiment; 3 is a diagram illustrating an example of a functional configuration of a server according to the embodiment;
  • FIG. 4 is a diagram for explaining processing in the image processing apparatus according to the embodiment.
  • 5 is a diagram illustrating an example of a functional configuration of a display device according to an embodiment;
  • FIG. FIG. 6 is a diagram for explaining processing in the server according to the embodiment;
  • FIG. 7 is a sequence diagram illustrating an example of the operation of the image processing system according to the embodiment;
  • the present inventors have found an image processing method and the like that can suppress deterioration of the image of the subject due to compression.
  • An image processing method is an image processing method for extracting a subject from an image including the subject using a background image, and is obtained by removing a background area not including the subject in the image from the image. compressing the background-removed image; transmitting the compressed background-removed image over a network; receiving the compressed background-removed image over the network; The compressed background-removed image is decoded, the decoded background-removed image and the background image are synthesized to generate a restored image, and the restored image and the background image are compared to extract the subject. .
  • a background-removed image is generated by removing the background area other than the subject from the image, and this background-removed image is compressed. Therefore, the ratio of the amount of data occupied by the subject area in the compressed background-removed image can be increased. Therefore, the amount of data of the compressed background-removed image to be transmitted via the network can be reduced without substantially reducing the image quality of the subject area. Therefore, it is possible to suppress deterioration in image quality of the image of the subject extracted at the transmission destination.
  • the background-removed image may be an image after the background area has been changed to an image in which the compression rate at the time of compression is higher than that of the area including the subject in the image.
  • a subject area including the subject in the image is extracted, and the subject area is cut out from the image to generate a subject image.
  • a certain background area is extracted, a background mask image is generated by changing the background area into an image in which a compression ratio is higher than that of the subject area when the background area is compressed, and the subject image is placed in the subject area of the background mask image.
  • the background-removed image may be generated by superimposing.
  • the image is a frame included in a moving image
  • the image processing method includes generation of the background-removed image, compression, transmission, reception, decoding, generation of the restored image, and extraction of the subject. may be executed for each frame included in the moving image.
  • the background image may be an uncompressed image or an image having a lower compression ratio than the compressed background-removed image.
  • the image processing method is executed by an image processing system comprising an image processing apparatus and a server that are communicably connected to each other over a network, and the generation, compression, and transmission of the background-removed image are performed by the image processing method.
  • the receiving, the decoding, the decompressed image generation and the object extraction may be performed by the server.
  • the boundary between the background area and the subject area may be set so as to overlap with the block boundary of blocks that are processing units in the compression.
  • the background-removed image can be compressed efficiently.
  • a plurality of pixels forming the background area of the background-removed image may each have substantially the same pixel value.
  • the background-removed image can be compressed so that the compression ratio when the background area is compressed is higher than that of the area including the subject in the image.
  • an image processing system is an image processing system that extracts a subject from an image including the subject using a background image
  • the image processing system includes an image processing device and a server that are connected to each other via a network.
  • the image processing apparatus comprising: a first generating unit that generates a background-removed image obtained by removing a background area that does not include the subject in the image; and a compression unit that compresses the background-removed image.
  • the server includes a receiving unit configured to receive the compressed background-removed image via the network; a decoding unit that decodes the compressed background-removed image; a second generation unit that combines the decoded background-removed image and the background image to generate a restored image; and the restored image and the background image. and an extracting unit that compares and extracts the subject.
  • a background-removed image is generated by removing the background area other than the subject from the image, and this background-removed image is compressed. Therefore, the ratio of the amount of data occupied by the subject area in the compressed background-removed image can be increased. Therefore, the amount of data of the compressed background-removed image to be transmitted via the network can be reduced without substantially reducing the image quality of the subject area. Therefore, it is possible to suppress deterioration in image quality of the image of the subject extracted at the transmission destination.
  • an image processing method is an image processing system that includes an image processing device that transmits a compressed image including a subject, and a server that extracts the subject from the compressed image using a background image.
  • An image processing method by an image processing device comprising: generating a background-removed image obtained by removing a background area not including a subject in an image including the subject from the image; compressing the background-removed image; The background-removed image thus obtained is transmitted to the server via a network.
  • a background-removed image is generated by removing the background area other than the subject from the image, and this background-removed image is compressed. Therefore, the ratio of the amount of data occupied by the subject area in the compressed background-removed image can be increased. Therefore, the amount of data of the compressed background-removed image to be transmitted via the network can be reduced without substantially reducing the image quality of the subject area. Therefore, it is possible to suppress deterioration in image quality of the image of the subject extracted at the transmission destination.
  • the image processing apparatus is an image processing system that includes an image processing apparatus that transmits a compressed image including a subject, and a server that extracts the subject from the compressed image using a background image.
  • An image processing apparatus comprising: a generation unit for generating a background-removed image obtained by removing a background area not including the subject from the image including the subject; and a compression unit for compressing the background-removed image. and a transmitting unit configured to transmit the compressed background-removed image over a network.
  • a background-removed image is generated by removing the background area other than the subject from the image, and this background-removed image is compressed. Therefore, the ratio of the amount of data occupied by the subject area in the compressed background-removed image can be increased. Therefore, the amount of data of the compressed background-removed image to be transmitted via the network can be reduced without substantially reducing the image quality of the subject area. Therefore, it is possible to suppress deterioration in image quality of the image of the subject extracted at the transmission destination.
  • an image processing method is an image processing system that includes an image processing device that transmits a compressed image including a subject, and a server that extracts the subject from the compressed image using a background image.
  • An image processing method by a server comprising: receiving the compressed background-removed image via a network; decoding the received compressed background-removed image; a restored image is generated by synthesis, the restored image and the background image are compared to extract the subject, and the compressed background-removed image is obtained by removing a background region of the image that does not include the subject from the image. This is an image obtained by compressing the background-removed image obtained by removing.
  • the server in the present disclosure is the server in an image processing system including an image processing device that transmits a compressed image including a subject, and a server that extracts the subject from the compressed image using a background image.
  • a receiving unit for receiving the compressed background-removed image via a network; a decoding unit for decoding the received compressed background-removed image; and the decoded background-removed image and the background image.
  • a generating unit that generates a restored image by combining; and an extracting unit that compares the restored image and the background image to extract the subject, wherein the compressed background-removed image is the subject in the image.
  • This is an image obtained by compressing a background-removed image obtained by removing a background region that does not contain from the image.
  • FIG. 1 is a schematic diagram of an image processing system according to an embodiment.
  • the image processing system 1 includes an image processing device 100 , a server 200 and a display device 300 .
  • the image processing device 100, the server 200, and the display device 300 are connected via a network 400 so as to be communicable.
  • the image processing device 100 and the display device 300 are, for example, PCs (Personal Computers).
  • the image processing device 100 may be a camera that captures moving images and still images.
  • the image processing device 100 may have the functions of the display device 300 , and the display device 300 may have the functions of the image processing device 100 .
  • Network 400 may be, for example, a general-purpose network such as the Internet, or may be a dedicated line.
  • FIG. 2 is a diagram showing an example of the functional configuration of the image processing device according to the embodiment.
  • FIG. 3 is a diagram for explaining processing in the image processing apparatus.
  • the image processing device 100 includes an acquisition unit 110 , a generation unit 120 , a compression unit 130 and a communication unit 140 .
  • Functions of the image processing apparatus 100 can be realized by a processor executing a predetermined program using a memory. Each component will be described below.
  • the acquisition unit 110 acquires a camera image 503, which is an image including a subject.
  • Acquisition unit 110 may acquire an image including a subject captured by a camera included in image processing apparatus 100 as camera video 503, or an image captured by a camera connected to an external terminal (communication interface) of image processing apparatus 100.
  • An image including the captured subject may be obtained as the camera image 503 .
  • the acquisition unit 110 acquires a background image 501 in which the subject is not shown.
  • the shooting target for which the camera image 503 is shot is the same space as the shooting target for which the background image 501 is shot.
  • the position, orientation, angle of view, focal length, and white balance of the camera when the camera image 503 is captured are the same as the position, orientation, angle of view, focal length, and white balance of the camera when the background image 501 is captured. and the same as white balance.
  • the camera parameters (extrinsic parameters and intrinsic parameters) of the camera are the same when the camera video 503 is captured and when the background image 501 is captured.
  • Acquisition unit 110 may be implemented by a camera included in image processing apparatus 100 or may be implemented by a communication interface included in image processing apparatus 100 .
  • the acquisition unit 110 may acquire a moving image including a subject as the camera image 503.
  • the processing for the image including the subject is performed for each frame included in the moving image.
  • the generation unit 120 is realized by cooperation of, for example, a CPU (Central Processing Unit) or a GPU (Graphical Processing Unit) and a memory (Memory) installed in a PC or a camera, and from the camera image 503 acquired by the acquisition unit, A background-removed image 506 is generated by removing a background area that does not include a subject in the image. Specifically, the generation unit 120 extracts a subject area 504 a including the subject in the camera image 503 . Although the subject area 504a is illustrated as a rectangular area, it may be an area having other shapes such as a circle and an ellipse. In the process of extracting the subject area 504a, for example, a process of recognizing the subject using a machine learning model is executed.
  • a process of recognizing the subject using a machine learning model is executed.
  • the machine learning model creates a data set that combines a group of images that include images of people (or objects) as subjects and correct data that indicates the area where the subject is captured in each image in the group of images. It is a model obtained by machine learning using Deep learning may be used in the process of extracting the subject region 504a. In this case, an object detection (subject detection) network may be used, or a segmentation network may be used.
  • subject area 504a may be an area slightly larger than the actually recognized subject area.
  • the generating unit 120 generates a subject image by cutting out a subject area 504a from the camera image 503.
  • the generation unit 120 extracts a background region 504b, which is a region other than the subject region 504a, from the camera image 503, and generates a background mask image 505 in which the pixel values of the plurality of pixels forming the background region 504b are substantially the same. do.
  • the boundary between the background region 504b and the subject region 504a may be set so as to overlap the block boundary of blocks that are processing units in compression by the compression unit 130 .
  • the subject detection result 504 (an example of an image including the subject) of the camera image 503 includes the subject area 504a including the subject and the background area 504b not including the subject.
  • the generation unit 120 generates a background-removed image 506 by superimposing the subject image (subject area 504 a ) on the subject area 505 a of the background mask image 505 . Therefore, the plurality of pixels forming the background region 506b of the background-removed image 506 have approximately the same pixel value.
  • the background-removed image 506 generated by the generating unit 120 is an image after the background area is changed to an image in which the compression rate at the time of compression is higher than that of the area including the subject in the camera image 503 . For example, pixels may each have a pixel value of 0 out of 0-255.
  • substantially the same pixel value may be a pixel value within a range of 0-10 among 0-255 pixel values.
  • the plurality of pixels may have pixel values within a predetermined range. In this case, even if the pixel values within the predetermined range are present in the subject region 506a, since the subject region 504a is determined in advance, the pixels having the pixel values within the predetermined range are It is not misidentified as the background area 506b.
  • a subject area 506 a of the background-removed image 506 is the same as the subject area 504 a extracted from the camera image 503 .
  • the generator 120 is an example of a first generator.
  • the background-removed image may be a subject image composed only of the subject area.
  • the subject image may include information indicating the position of the subject area in the image (or information indicating the position of the background area in the image).
  • the compression unit 130 is implemented by cooperation of, for example, a CPU or GPU installed in a PC or a camera and a memory, and compresses (encodes) the background-removed image 506 to a first data amount.
  • the first data amount may be a predetermined fixed data amount, or may be a variable data amount determined according to the communicable band of the network 400 . More specifically, the compression unit 130 compresses the background-removed image 506 such that the compression rate of the background region 506b of the background-removed image 506 is higher than the compression rate of the subject region 506a of the background-removed image 506.
  • the generation unit 120 generates the background mask image 505 such that the compression rate of the background area 506b is higher than the compression rate of the subject area 506a when the background-removed image 506 is compressed.
  • the compression unit 130 may generate the transmission background image 502 by compressing the background image 501 to a second data amount larger than the first data amount. That is, the compression unit 130 may compress the background image 501 at a compression rate lower than that of the background-removed image 506 .
  • the compression unit 130 may generate the transmission background image 502 without compressing the background image 501 . In this case, the transmission background image 502 is an uncompressed image and is the same as the background image 501 .
  • the communication unit 140 transmits and receives data to and from the server 200 or the display device 300 via the network 400 . Specifically, communication unit 140 transmits the compressed background-removed image to server 200 via network 400 . Also, the communication unit 140 transmits the transmission background image 502 to the server 200 via the network 400 .
  • Communication unit 140 is realized by a communication IF for communicating with a gateway connected to network 400 .
  • the communication unit 140 may perform communication by wired communication, or may perform communication by wireless communication. Note that the communication unit 140 is an example of a transmission unit.
  • FIG. 4 is a diagram showing an example of the functional configuration of the server according to the embodiment.
  • FIG. 5 is a diagram for explaining processing in the server.
  • the server 200 includes a communication unit 210, a decoding unit 220, a generation unit 230, an extraction unit 240, a synthesis unit 250, and a storage unit 260.
  • the functions of the server 200 can be realized by the processor executing a predetermined program using memory. Each component will be described below.
  • the communication unit 210 transmits and receives data to and from the image processing device 100 or the display device 300 via the network 400 .
  • communication unit 210 receives received video 512 that is a compressed background-removed image via network 400 .
  • the communication unit 210 receives a background image (received background image 511 ) from the image processing apparatus 100 .
  • the received background image 511 is the same as the transmitted background image 502 .
  • the communication unit 210 may transmit the synthesized image 516 generated by the synthesis unit 250 to the display device 300 via the network 400 .
  • Communication unit 210 is realized by a communication IF for communicating with a gateway connected to network 400 .
  • the communication unit 210 may perform communication by wired communication, or may perform communication by wireless communication. Note that the communication unit 210 is an example of a receiving unit.
  • the decoding unit 220 decodes the received video 512 received. A background-removed image is thus obtained. Note that the decoding unit 220 may decode only the subject region 512a of the received video 512, or may decode the entire video including the background region 512b. The decoding unit 220 can identify the subject area 512a by referring to the pixel values assigned to the background area 512b of the background-removed image.
  • the generation unit 230 generates a restored video 513 (restored image) by synthesizing the background-removed image obtained by decoding and the received background image 511 . Specifically, the generation unit 230 extracts the subject area 512 a from the background-removed image and superimposes the subject area 512 a on the area corresponding to the subject area 512 a of the received background image 511 to generate the restored video 513 .
  • the generator 230 is an example of a second generator.
  • the extraction unit 240 compares the restored video 513 and the received background image 511 to extract the subject. As a result, a subject extraction image 514 is obtained.
  • the extraction unit 240 compares the restored video 513 and the received background image 511 and extracts an area where the pixel value difference is larger than a predetermined difference, thereby generating the subject extraction video 514 from the restored video 513.
  • the extracting unit 240 compares the restored video 513 and the received background image 511, and extracts the subject extraction video 514 by removing from the restored video 513 areas where the pixel value difference is smaller than a predetermined difference. You may
  • the extraction unit 240 may extract the subject by comparing the subject area 512a of the received video 512 and the area corresponding to the subject area 512a of the received background image 511.
  • the synthesizing unit 250 generates a synthetic video 516 (composite image) by superimposing the extracted subject extraction video 514 on an arbitrary background image (compositing background image 515) different from the received background image 511.
  • the synthesis unit 250 may superimpose the subject extraction video 514 on a specific position of the background image 515 for synthesis, or may superimpose the extraction video 514 on a position specified by the user in the background image 515 for synthesis. good.
  • the storage unit 260 stores the received background image 511 received by the communication unit 210 .
  • the storage unit 260 may store the background image 515 for synthesis.
  • the storage unit 260 is implemented by, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like.
  • FIG. 6 is a diagram showing an example of the functional configuration of the display device according to the embodiment.
  • the display device 300 includes a communication section 310 and a display section 320 .
  • the functions of the display device 300 can be realized by the processor executing a predetermined program using memory. Each component will be described below.
  • the communication unit 310 transmits and receives data to and from the image processing apparatus 100 or the server 200 via the network 400 . Specifically, communication unit 310 receives composite video 516 via network 400 .
  • Communication unit 310 is implemented by a communication IF for communicating with a gateway connected to network 400 .
  • the communication unit 310 may perform communication by wired communication, or may perform communication by wireless communication.
  • the display unit 320 displays the composite image 516 received by the communication unit 310.
  • the display unit 320 can be implemented by, for example, a display.
  • the processing by the image processing system 1 is performed sequentially (in real time) every time each frame included in the camera video 503 as a moving image is shot, and the synthesized video 516 may be delivered live.
  • the composite image 516 may be stored in the storage unit 260 of the server 200 .
  • the display device 300 may transmit a reproduction request for the composite video 516 stored in the storage unit 260 of the server 200 in response to an instruction from the user, and reproduce (display) the composite video 516 distributed from the server 200. .
  • FIG. 7 is a sequence diagram illustrating an example of the operation of the image processing system according to the embodiment.
  • the image processing device 100 acquires a background image (S101).
  • the image processing apparatus 100 transmits the transmission background image 502 to the server 200 via the network 400 (S102).
  • the server 200 receives the transmitted background image 502 (received background image 511) (S103).
  • the server 200 stores the received background image 511 in the storage unit 260 (S104).
  • the image processing apparatus 100 acquires the camera image 503, which is an image including the subject (S105).
  • the image processing apparatus 100 extracts a subject area 504a including the subject in the camera image 503 (S106).
  • the image processing apparatus 100 generates a subject image by cutting out the subject area 504a from the camera image 503 (S107).
  • the image processing apparatus 100 extracts a background region 504b, which is a region other than the subject region 504a, from the camera image 503, and extracts a background mask in which the pixel values of the plurality of pixels forming the background region 504b are substantially the same pixel value.
  • An image 505 is generated (S108).
  • the image processing apparatus 100 generates a background-removed image 506 by superimposing the subject image (subject area 504a) on the subject area 505a of the background mask image 505 (S109).
  • the image processing apparatus 100 compresses the background-removed image 506 to a first data amount (S110).
  • the image processing apparatus 100 transmits the compressed background-removed image to the server 200 via the network 400 (S111).
  • the server 200 receives the received video 512, which is a compressed background-removed image, via the network 400 (S112).
  • the server 200 decodes the received video 512 and generates a background-removed image (S113).
  • the server 200 combines the background-removed image obtained by decoding and the received background image 511 to generate a restored video 513 (restored image) (S114).
  • the server 200 extracts the subject by comparing the restored video 513 and the received background image 511, and generates a subject extraction video 514 (S115).
  • the server 200 generates a composite video 516 (composite image) by superimposing the extracted subject extraction video 514 on an arbitrary background image (compositing background image 515) different from the received background image 511 (S116). ).
  • the server 200 transmits the generated composite image 516 to the display device 300 via the network 400 (S117).
  • the display device 300 receives the composite image 516 via the network 400 (S118).
  • the display device 300 displays the received composite image 516 (S119).
  • the image processing system 1 uses the background image 501 to extract the subject from the camera image 503, which is an image including the subject.
  • the image processing system 1 includes an image processing apparatus 100 and a server 200 that are connected to each other via a network 400 .
  • the image processing apparatus 100 includes a generation section 120 , a compression section 130 and a communication section 140 .
  • the generation unit 120 generates a background-removed image 506 obtained by removing a background area that does not include the subject in the image.
  • a compression unit 130 compresses the background-removed image 506 .
  • Communication unit 140 transmits the compressed background-removed image via network 400 .
  • the server 200 includes a communication unit 210 , a decryption unit 220 , a generation unit 230 and an extraction unit 240 .
  • the communication unit 210 receives the compressed background-removed image via the network 400 .
  • the decoding unit 220 decodes the received compressed background-removed image.
  • the generation unit 230 generates a restored video 513 by synthesizing the decoded background-removed image and the received background image 511 .
  • the extraction unit 240 compares the restored video 513 and the received background image 511 to extract the subject.
  • a background-removed image 506 is generated by removing the background area other than the subject from the camera image 503, and this background-removed image 506 is compressed. Therefore, the ratio of the data amount (code amount) occupied by the subject area in the compressed background-removed image can be increased. Therefore, the amount of data of the compressed background-removed image 506 to be transmitted via the network 400 can be reduced without substantially reducing the image quality of the subject area. Therefore, it is possible to suppress deterioration in image quality of the image of the subject extracted at the transmission destination.
  • a plurality of pixels forming the background region 506b of the background-removed image 506 have substantially the same pixel value. Therefore, the amount of data (code amount) for the background region of the background-removed image can be reduced.
  • the subject area 504a including the subject in the camera image 503 is extracted, and the subject area 504a is cut out from the camera image 503 to generate the subject image.
  • a background region 504b which is a region other than the subject region, is extracted from the camera image 503, and a background region 504b having substantially the same pixel value is extracted from a plurality of pixels forming the background region 504b.
  • a mask image 505 is generated.
  • the background-removed image 506 is generated by superimposing the subject image (subject area 504 a ) on the subject area 505 a of the background mask image 505 . Therefore, the background-removed image 506 can be easily generated.
  • the camera image 503 is a moving image.
  • the moving image includes generation and compression of the background-removed image 506, transmission of the compressed background-removed image 506, reception and decoding of the compressed background-removed image 506, generation of the restored video 513, and extraction of the subject. is executed for each frame Therefore, it is possible to suppress deterioration in the image quality of the subject image extracted by the transmission destination device (that is, the server 200) for each frame of the moving image.
  • the transmission background image 502 is an image with a lower compression rate than the uncompressed image or the compressed background-removed image 506 . Therefore, the quality of the background image can be made high.
  • the boundary between the background region 504b and the subject region 504a is set so as to overlap with the block boundary of blocks that are processing units in compression by the compression unit . Therefore, the background-removed image 506 can be compressed efficiently.
  • the image processing system 1 performs processing for extracting a subject using a background image captured by a fixed camera and camera video
  • the background image includes a plurality of background images captured in advance for each posture by panning the camera within a limited angular range, and the camera video is captured while panning the camera within the limited angular range.
  • each background image is associated with the orientation of the camera and stored in the storage unit 260 of the server 200, and the server 200 receives the camera video from the image processing device 100 for each frame together with the orientation information indicating the orientation of the camera. receive.
  • the server 200 may specify the background image associated with the posture information for each frame, and extract the subject using the background image and the received video generated by the frame.
  • the server 200 associates and stores the camera parameters related to the orientation of the camera with the background image obtained by changing the camera parameters and the background image obtained by the camera parameters.
  • the background image corresponding to the parameter is specified and the subject is extracted using the frame and the specified background image, the present invention is not limited to this. The same can be applied to changing camera parameters other than camera posture.
  • Each processing unit included in the image processing system, image processing apparatus, server, display apparatus, etc. according to the above embodiments is typically implemented as an LSI, which is an integrated circuit. These may be made into one chip individually, or may be made into one chip so as to include part or all of them.
  • circuit integration is not limited to LSIs, and may be realized with dedicated circuits or general-purpose processors.
  • An FPGA Field Programmable Gate Array
  • a reconfigurable processor that can reconfigure the connections and settings of the circuit cells inside the LSI may be used.
  • each component may be configured with dedicated hardware or realized by executing a software program suitable for each component.
  • Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor.
  • the present disclosure may be implemented as a video transmission method, a transmission method, a display method, and the like executed by an image processing system, an image processing device, a server, a display device, and the like.
  • the division of functional blocks in the block diagram is an example, and a plurality of functional blocks can be realized as one functional block, one functional block can be divided into a plurality of functional blocks, and some functions can be moved to other functional blocks.
  • single hardware or software may process the functions of a plurality of functional blocks having similar functions in parallel or in a time-sharing manner.
  • each step in the flowchart is executed is for illustrative purposes in order to specifically describe the present disclosure, and orders other than the above may be used. Also, some of the above steps may be executed concurrently (in parallel) with other steps.
  • the present disclosure is applicable to an image processing system and the like that can suppress deterioration of an image of a subject due to compression.
  • image processing system 100 image processing device 110 acquisition unit 120 generation unit 130 compression units 140, 210, 310 communication unit 200 server 220 decoding unit 230 generation unit 240 extraction unit 250 synthesis unit 260 storage unit 300 display device 320 display unit 400 network 501 Background image 502 Transmitted background image 503 Camera image 504 Subject detection results 504a, 505a, 506a, 512a Subject areas 504b, 506b, 512b Background area 505 Background mask image 506 Background removed image 511 Received background image 512 Received image 513 Restored image 514 Subject extraction Video 515 Synthetic background image 516 Synthetic video

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Compression Of Band Width Or Redundancy In Fax (AREA)
PCT/JP2022/028252 2021-09-30 2022-07-20 画像処理方法、画像処理システム、画像処理装置、及び、サーバ Ceased WO2023053687A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP22875564.1A EP4412201A4 (en) 2021-09-30 2022-07-20 IMAGE PROCESSING METHOD, IMAGE PROCESSING SYSTEM, IMAGE PROCESSING DEVICE, AND SERVER
JP2023550408A JPWO2023053687A1 (https=) 2021-09-30 2022-07-20
CN202280065153.7A CN118020292A (zh) 2021-09-30 2022-07-20 图像处理方法、图像处理系统、图像处理装置以及服务器
US18/612,493 US12495149B2 (en) 2021-09-30 2024-03-21 Image processing method, image processing system, image processing device, and server

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021160750 2021-09-30
JP2021-160750 2021-09-30

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/612,493 Continuation US12495149B2 (en) 2021-09-30 2024-03-21 Image processing method, image processing system, image processing device, and server

Publications (1)

Publication Number Publication Date
WO2023053687A1 true WO2023053687A1 (ja) 2023-04-06

Family

ID=85782240

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/028252 Ceased WO2023053687A1 (ja) 2021-09-30 2022-07-20 画像処理方法、画像処理システム、画像処理装置、及び、サーバ

Country Status (5)

Country Link
US (1) US12495149B2 (https=)
EP (1) EP4412201A4 (https=)
JP (1) JPWO2023053687A1 (https=)
CN (1) CN118020292A (https=)
WO (1) WO2023053687A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024224970A1 (ja) * 2023-04-28 2024-10-31 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ 復号装置、符号化装置、ビットストリーム生成装置、復号方法及び符号化方法

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113014799B (zh) * 2021-01-28 2023-01-31 维沃移动通信有限公司 图像显示方法、装置和电子设备
CN118524258B (zh) * 2024-07-25 2024-10-18 浙江嗨皮网络科技有限公司 离线视频背景处理方法、系统及可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005260912A (ja) * 2004-02-10 2005-09-22 Matsushita Electric Ind Co Ltd 映像通信装置および映像通信方法
WO2016013147A1 (ja) * 2014-07-22 2016-01-28 パナソニックＩｐマネジメント株式会社 符号化方法、復号方法、符号化装置および復号装置
JP2019176306A (ja) * 2018-03-28 2019-10-10 キヤノン株式会社 監視システム、監視システムの制御方法及びプログラム
JP2020014051A (ja) 2018-07-13 2020-01-23 ローランド株式会社 情報処理方法、及び情報処理装置

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07288789A (ja) 1994-04-15 1995-10-31 Hitachi Ltd 知的符号化装置及び画像通信装置
US5710590A (en) 1994-04-15 1998-01-20 Hitachi, Ltd. Image signal encoding and communicating apparatus using means for extracting particular portions of an object image
JP2000050259A (ja) 1998-08-03 2000-02-18 Hitachi Telecom Technol Ltd 動画像符号化装置および動画像復号装置
JP2005203865A (ja) 2004-01-13 2005-07-28 Konica Minolta Photo Imaging Inc 画像処理システム
JP6894687B2 (ja) * 2016-10-11 2021-06-30 キヤノン株式会社 画像処理システム、画像処理装置、制御方法、及び、プログラム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005260912A (ja) * 2004-02-10 2005-09-22 Matsushita Electric Ind Co Ltd 映像通信装置および映像通信方法
WO2016013147A1 (ja) * 2014-07-22 2016-01-28 パナソニックＩｐマネジメント株式会社 符号化方法、復号方法、符号化装置および復号装置
JP2019176306A (ja) * 2018-03-28 2019-10-10 キヤノン株式会社 監視システム、監視システムの制御方法及びプログラム
JP2020014051A (ja) 2018-07-13 2020-01-23 ローランド株式会社 情報処理方法、及び情報処理装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4412201A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024224970A1 (ja) * 2023-04-28 2024-10-31 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ 復号装置、符号化装置、ビットストリーム生成装置、復号方法及び符号化方法

Also Published As

Publication number Publication date
EP4412201A1 (en) 2024-08-07
EP4412201A4 (en) 2025-04-09
CN118020292A (zh) 2024-05-10
US20240236338A1 (en) 2024-07-11
US12495149B2 (en) 2025-12-09
JPWO2023053687A1 (https=) 2023-04-06

Similar Documents

Publication Publication Date Title
US20220210512A1 (en) Content based stream splitting of video data
WO2023053687A1 (ja) 画像処理方法、画像処理システム、画像処理装置、及び、サーバ
US10504246B2 (en) Distinct encoding and decoding of stable information and transient/stochastic information
US20160345009A1 (en) Accelerating image analysis and machine learning through in-flash image preparation and pre-processing
US20210092479A1 (en) Video processing apparatus
CN115248924A (zh) 一种二维码处理方法、装置、电子设备和存储介质
WO2019184822A1 (zh) 多媒体文件处理方法和装置、存储介质及电子装置
CN116800965A (zh) 一种图像处理方法、编码装置、解码装置及存储介质
US10783670B2 (en) Method for compression of 360 degree content and electronic device thereof
JPWO2023053687A5 (https=)
US20240048738A1 (en) Methods, apparatuses, computer programs and computer-readable media for processing configuration data
CN110740352B (zh) 显卡透传环境下基于spice协议的差异图像显示方法
EP3352133B1 (en) An efficient patch-based method for video denoising
EP3985989A1 (en) Detection of modification of an item of content
WO2022127565A1 (zh) 一种视频处理方法、装置以及设备
US20240087170A1 (en) Method for multiview picture data encoding, method for multiview picture data decoding, and multiview picture data decoding device
CN117014562B (zh) 多路视频流混合编解码方法、设备及存储介质
CN112559111B (zh) 一种共享桌面时的截屏方法和装置
US20170109596A1 (en) Cross-Asset Media Analysis and Processing
CN112822511A (zh) 视频处理方法、系统、智能终端及计算机可读存储介质
JP2024519925A5 (https=)
US12483745B2 (en) Rendering media streams
CN113411544A (zh) 视频分片文件的发送方法及装置、存储介质及电子装置
US20220256191A1 (en) Panoramic video generation method, video collection method, and related apparatus
CN121056644A (zh) 图像处理方法及图像处理装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22875564

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023550408

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 202280065153.7

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022875564

Country of ref document: EP

Effective date: 20240430

WWW Wipo information: withdrawn in national office

Ref document number: 2022875564

Country of ref document: EP