JP7426021B2 - Systems, methods, and computer-readable media for video processing - Google Patents

Description

The present invention relates to image or video processing in live video streaming or video teleconferencing.

Various techniques are known that allow users to participate in online communications with each other. Its applications include live streaming, live conference calls, and more. With the spread of these applications, there is an increasing demand from users for improved interaction experiences during communication.

A method according to one embodiment of the present invention is a video processing method. The method includes the steps of displaying a first user's live video in a first area of a user terminal, and displaying a second user's live video in a second area of the user terminal, A portion of the video is stretched to the second area of the user terminal.

A system according to an embodiment of the present invention is a system for video processing, and includes one or more processors, and the one or more processors execute machine-readable instructions to convert live video of a first user to a user terminal. A step of displaying the video of the second user in the second region of the user terminal is executed. A portion of the live video of the first user is extended to the second area of the user terminal.

A computer-readable medium according to an embodiment of the present invention is a non-transitory computer-readable medium, and includes a program for video processing, and the program transmits live video of a first user to a user terminal. A step of displaying the second user's video in the second region of the user terminal is executed. A portion of the live video of the first user is extended to the second area of the user terminal.

FIG. 2 is a schematic diagram showing an example of a group call. 1 is a schematic diagram illustrating an example of a group call in accordance with some embodiments of the present invention; FIG. 1 is a schematic diagram illustrating an example of a group call in accordance with some embodiments of the present invention; FIG. 1 is a schematic diagram illustrating an example of a group call in accordance with some embodiments of the present invention; FIG. 1 is a schematic diagram illustrating an example of a group call in accordance with some embodiments of the present invention; FIG. 1 is a schematic diagram illustrating an example of a group call in accordance with some embodiments of the present invention; FIG. 1 is a schematic diagram illustrating a configuration of a communication system according to some embodiments of the present invention. FIG. 1 is an exemplary functional block diagram of a communication system according to some embodiments of the present invention. FIG. 1 is an exemplary sequence diagram illustrating the operation of a communication system according to some embodiments of the present invention. FIG.

Some live streaming services, applications (APPs) or platforms allow multiple users (streamers, viewers, broadcasters, anchors, etc.) to participate in group call mode or conference call mode, of which: Images of multiple users are displayed simultaneously on the screen of the user terminal to indicate their group call or participation in the group call. The user terminal can be a smartphone, tablet, personal computer, or laptop, and is used by one of the users to participate in the group call.

FIG. 1 shows a schematic diagram illustrating an example of a group call. S1 is a screen of the user terminal that displays the group call. RA is one area within the screen S1, and displays user A's live video. RB is one area within the screen S1, and displays user B's live video. The live video of the user A may be captured and provided by a video capture device such as a camera placed near the user A. The live video of the user B may be captured and provided by a video capturing device such as a camera placed near the user B.

Conventionally, the video of the user A can only be displayed within the area RA, and cannot be displayed within the area RB. Similarly, the video of the user B can only be displayed within the area RB, and cannot be displayed within the area RA. This may cause inconvenience during communication or may cause problems with the application. For example, in an exemplary scenario in which User B presents a newly developed product to User A in the group call, User A is unable to pinpoint a part or portion of the product for detailed discussion. Therefore, it is desirable to have more interaction during group calls and conference calls.

FIG. 2 is a schematic diagram illustrating an example of a group call in accordance with some embodiments of the present invention. As shown in FIG. 2, user A's portion A1 has been extended or reproduced/duplicated to area RB where user B is displayed. In this embodiment, the portion A1 is the user A's hand in the area RA, and the portion A11 is a stretched, reproduced, or duplicated version of the portion A1 displayed in the area RB. The portion A11 points to or is oriented towards the object B1 within the region RB. In some implementations, the video of user B displayed within region RB is live video. In some embodiments, the video of user B displayed within region RB is a played video.

In some embodiments, the portion A11 follows the movement or trajectory of the portion A1. In some embodiments, the portion A11 moves simultaneously with the portion A1. The user A can control or move the part A11 to point to a position within the area RB that the user A wants to discuss by simply moving the hand that is the part A1. In some implementations, the portion A11 may be represented or displayed as a graphical or animated object.

As shown in FIG. 2, there is a boundary A3 within the area RA. The boundary A3 defines an area A31 and an area A32 within the area RA. In this embodiment, the area A31 surrounds the area A32. The area A31 can be called or defined as an interactive area. The portion A1 that is extended or reproduced within the area RB is within the interactive area A31. The portion A1 is extended toward the user B within the area RA. In some implementations, only the portion within the interactive area A31 may be extended or displayed within the area RB. In some implementations, if user A wants to interact with user B by extending a part of user A into region RB, user A can simply move the part to the interactive area A31 and Displayed in RB. In this embodiment, the area RA and the area RB are separated from each other. In some embodiments, the area RA and the area RB may at least partially overlap on the screen S1.

As shown in FIG. 2, there is a boundary B3 within the region RB. The boundary B3 defines a region B31 and a region B32 within the region RB. In this embodiment, the area B31 surrounds the area B32. The area B31 can be called or defined as an interactive area. In some implementations, a portion within the interactive area B31 may be extended or displayed within the area RA. In some implementations, if user B wants to interact with user A by extending a part of user B to area RA, user B can simply move the part to the interactive area B31 and Displayed in RA. In some embodiments, the boundary A3 and/or the boundary B3 may not be displayed on the area RA and/or the area RB.

In FIG. 2, user A and user B, or area RA and area RB, are arranged horizontally on the screen S1 of the user terminal, and the part A1 of the live video of the user A is displayed by the user in the area RA. It is extended towards B.

FIG. 3 shows a schematic diagram illustrating another example of group calling according to some embodiments of the present invention. At least four users, ie, user A, user B, user C, and user D, are participating in the group call. In FIG. 3, user A and user B are lined up in the vertical direction on the screen S1 of the user terminal. As shown in FIG. 3, user A's portion A2 has been extended or reproduced/duplicated to area RB where user B is displayed. In this embodiment, the portion A2 includes the user A's hand and the object held in the hand, and the portion A21 is a stretched, reproduced, or duplicated version of the portion A2 displayed within the region RB. be. The portion A21 approaches or is directed toward the user B within the region RB. When the portion A21 comes into contact with the user B, a special effect SP1 is displayed in the area RB. The special effect SP1 may include a graphic object or an animation object. In some embodiments, the special effect SP1 may include sound effects.

In some embodiments, the portion A21 follows the movement or trajectory of the portion A2. In some embodiments, the portion A21 moves simultaneously with the portion A2. The user A controls the part A21 to point to or touch a position in the area RB with which the user A wants to interact by simply moving the user A's hand (which may be holding an object). You can do or move it. In some implementations, the portion A21 may be represented or displayed as a graphical or animated object.

As shown in FIG. 3, there is a boundary A3 within the area RA. The boundary A3 defines an area A31 and an area A32 within the area RA. In this embodiment, the area A31 surrounds the area A32. The area A31 can be called or defined as an interactive area. The portion A2 that is extended or reproduced within the area RB is within the interactive area A31. The portion A2 is extended toward the user B within the area RA. In some implementations, only the portion within the interactive area A31 may be extended or displayed within the area RB. In some implementations, if user A wants to interact with user B by extending a part of user A into region RB, user A can simply move the part to the interactive area A31 and Displayed in RB.

FIG. 4 shows a schematic diagram illustrating another example of group calling according to some embodiments of the present invention. At least four users, ie, user A, user B, user C, and user D, are participating in the group call. In FIG. 4, users A and D are lined up diagonally on the screen S1 of the user terminal. As shown in FIG. 4, user A's portion A1 has been extended or reproduced/duplicated to region RD where user D is displayed. In this embodiment, the portion A1 is user A's hand, and the portion A11 is a stretched, reproduced, or duplicated version of the portion A1 displayed in region RD. The portion A11 points to or is directed toward the user D within the region RD.

In some embodiments, the portion A11 follows the movement or trajectory of the portion A1. In some embodiments, the portion A11 moves simultaneously with the portion A1. The user A can control or move the part A11 so as to point to a position in the area RD with which the user A wants to interact by simply moving the hand that is the part A1. In some implementations, the portion A11 may be represented or displayed as a graphical or animated object.

As shown in FIG. 4, there is a boundary A3 within the area RA. The boundary A3 defines an area A31 and an area A32 within the area RA. The area A31 surrounds the area A32. The area A31 can be called or defined as an interactive area. In this embodiment, the interactive area A31 includes a sub-area A311. The portion A1 that is stretched or reproduced within the region RD is within the sub-region A311. The sub-area A311 is located between users A and D. The sub-area A311 is located in the area RA at a position facing the area RD when viewed from the user A.

As shown in the examples of FIGS. 2, 3, and 4, the direction in which the portion of user A is stretched in area RA determines the area in which the stretched, duplicated, and reproduced version of user A is displayed. may be done. Therefore, user A can decide which area (and corresponding user) to interact with simply by moving or stretching that part of user A in the corresponding direction. For example, user A can extend the portion laterally and interact with users whose display areas are lined up or placed laterally with respect to user A on the screen S1. In another example, user A can extend a portion vertically and interact with a user whose display area is aligned or arranged vertically with respect to user A on the screen S1. In yet another example, user A can extend the portion diagonally and interact with users whose display areas are lined up or arranged diagonally with respect to user A on the screen S1. .

In some implementations, a user can adjust the shape of the interactive area to more conveniently interact with another user. FIG. 5 shows a schematic diagram illustrating another example of group calling according to some embodiments of the present invention. At least four users, ie, user A, user B, user C, and user D, are participating in the group call. As shown in FIG. 5, the boundary A3 defines the interactive area A31, which is the area used by user A to interact with other users, as described in the previous exemplary embodiment. The interactive area A31 includes a sub-area A311. The boundary A3 includes at least a boundary line BR1 and a boundary line BR2. In some implementations, if user A wants to interact more conveniently with another user, user A adjusts the position of said border line BR1 and/or the position of said border line BR2, and adjusts the shape of said interactive area A31. The shape of the sub-region A311 can be adjusted. The boundary line BR1 corresponds to the direction of the user C or the region RC with respect to the region RA, and is between the region RA and the region RC. The boundary line BR2 corresponds to the direction of the user B or the area RB with respect to the area RA, and is between the area RA and the area RB.

For example, user A drags or moves the boundary line BR1 closer to user A so that sub-area A312 of interactive area A31 between users A and C becomes wider and closer to user A. can do. This makes it easier for user A to interact with user C using the user A part. User A only needs to extend the corresponding part of user A by a relatively short distance, cross the boundary line BR1, and reach the corresponding sub-area A312 of the interactive area A31, and then user C will be able to reach the corresponding sub-area A312 in the displayed area RC. The part is stretched, copied or reproduced.

In another example, user A moves the boundary line BR2 closer to user A so that the sub-area A313 of the interactive area A31 between users A and B becomes wider and closer to user A. Can be dragged or moved. This makes it easier for user A to interact with user B using the user A part. User A only needs to extend the corresponding part of user A by a relatively short distance, cross the boundary line BR2 and reach the corresponding sub-area A312 of the interactive area A31, and then user B will be able to reach the corresponding sub-area A312 of the interactive area A31. The part is stretched, copied or reproduced.

In yet another example, the user A may select the border line BR1 and/or the border line so that the sub-area A311 of the interactive area A31 between the user A and the user D is wider and closer to the user A. BR2 can be dragged or moved closer to user A. This makes it easier for user A to interact with user D using the user A part. User A only needs to extend the corresponding part of user A in the diagonal direction by a relatively short distance and reach the corresponding sub-area A311 of the corresponding interactive area A31, and then the corresponding part will be displayed in the area RD where user D is displayed. To be stretched, copied or reproduced.

FIG. 6 shows a schematic diagram illustrating another example of group calling according to some embodiments of the present invention. In some implementations, only the outside of the interactive area is extracted and displayed on the screen S1. More specifically, for user A, only the area surrounded by the boundary A3 is displayed on the screen S1. For users B, C, and D, only the area surrounded by the boundaries B3, C3, and D3 is displayed on the screen S1. Thereby, the sense of realism of the interaction can be improved. For example, when user A stretches a portion into another user's display area, the portion is not displayed in user A's display area.

FIG. 7 shows a schematic diagram showing the configuration of a communication system 1 based on some embodiments of the present invention. The communication system 1 can provide live streaming services with interaction through content. "Content" here refers to digital content that can be played back on a computer device. The communication system 1 allows users to participate in real-time interactions with other users online. The communication system 1 includes a plurality of user terminals 10, a backend server 30, and a streaming server 40. The user terminal 10, the backend server 30, and the streaming server 40 are connected via a network 90 (for example, the Internet may be used). Backend server 30 may be a server that synchronizes interactions between user terminals and/or streaming server 40 . In some implementations, backend server 30 may be an application (APP) provider's origin server. Streaming server 40 is a server for handling or providing streaming data or video data. In some implementations, backend server 30 and streaming server 40 may be independent servers. In some implementations, backend server 30 and streaming server 40 may be integrated into one server. In some implementations, user terminal 10 is a client device for live streaming. In some implementations, user terminal 10 may be referred to as a viewer, streamer, anchor, podcaster, audience, listener, etc. The user terminal 10, the backend server 30, and the streaming server 40 are each examples of information processing devices. In some implementations, streaming can be live streaming or video playback. In some implementations, streaming can be audio streaming and/or video streaming. In some implementations, streaming can include content such as online shopping, talk shows, talent shows, entertainment events, sporting events, music videos, movies, comedy, concerts, group calls, conference calls, and the like.

FIG. 8 shows an exemplary functional block diagram of a communication system according to some embodiments of the present invention. In FIG. 8, the network 90 is omitted.

The backend server 30 includes a message unit 32 . The message unit 32 is configured to receive data or information from a user terminal, process and/or store the data, and send the data to the user terminal. In some implementations, the message unit 32 may be a separate unit from the backend server 30.

The streaming server 40 includes a data receiving device 400 and a data transmitting device 402. The data receiving device 400 is configured to receive data or information such as streaming data or video data from various user terminals. The data transmission device 402 is configured to transmit data or information, such as streaming data or video data, to various user terminals.

The user terminal 10A may be a user terminal operated by user A. The user terminal 10A includes a camera 700, a renderer 702, a display 704, an encoder 706, a decoder 708, a result transmitting device 710, a matting unit 712, and an object recognition unit 714.

The camera 700 may be or include any type of video capture device. The camera 700 is configured to capture video data of user A, for example.

The renderer 702 receives video data from the camera 700 (user A's video data), receives video data from the decoder 708 (which may include video data from user B), and displays the video data on the display 704. It is configured to generate rendered video for display (such as video displaying a group call in which User A and User B are displayed).

The display 704 is configured to display rendered video from the renderer 702. In some implementations, the display 704 may be a screen on the user terminal 10A.

The encoder 706 is configured to encode video data from the camera 700 and send the encoded video data to the data receiving device 400 of the streaming server 40 . The encoded data may be transmitted as streaming data.

The decoder 708 receives video data or streaming data (which may include video data from user B) from the data transmission device 402 of the streaming server 40, decodes the video data, converts it into decoded video data, and converts it into decoded video data. The rendered video data is sent to the renderer 702 for rendering.

The matting unit 712 is configured to perform a matting process (image matting or video matting) on the video data (user A's video data) from the camera 700. The matting process may include a contour recognition process, an image comparison process, a moving object detection process, and/or a cutting process. The matting process may be performed using techniques including constant-color matting, difference matting, and natural image matting. Further, the algorithm related to the matting process may include Bayesian matting, Poisson matting, or Robust matting. In some implementations, the image comparison process periodically compares the initial or default background image and the current or live image to detect user A's portion in the interactive area.

For example, the matting unit 712 receives user A's video data from the camera 700. The video data may include an interactive area as described above by way of example in FIGS. 2, 3, 4, and 5. In some implementations, the matting unit 712 performs a matting process to detect or extract the contour of user A in the video data. In some implementations, the matting unit 712 performs a matting process that detects or extracts a portion of user A within the interactive area (such as user A's hand or user A's hand holding an object). Execute. In some implementations, the matting unit 712 performs a cropping process that removes areas or portions outside the interactive area from the user A's video data. In some implementations, the matting unit 712 detects, recognizes, or determines the location within the interactive area where the portion of user A is detected. In some embodiments, a contour recognition process or an image comparison process may be performed before the cutting process to improve the accuracy of detecting the part of the user A in the interactive area.

In some implementations, the interactive area and corresponding boundaries or boundaries may be defined by a processor (not shown) of the user terminal 10A or an application enabling the group call. In some implementations, the interactive area and the corresponding boundaries or boundaries may be determined by the user A in a UI (user interface) unit (not shown) of the user terminal 10A. In some implementations, the matting unit 712 detects the portion of user A within the interactive area (or the live video of user A) by detecting the portion of user A beyond the border within the area RA. detect or determine the relevant part of the The boundary line within the area RA may be the boundary line BR1 or the boundary line BR2 in FIG. 5, for example.

The object recognition unit 714 is configured to perform object recognition processing on the output data from the matting unit 712. The output data may include a detected part or an extracted part of user A (user A's hand, user A's hand holding an object, etc.). The object recognition unit 714 performs an object recognition process to determine whether the detected portion of the user A includes any predetermined patterns, objects, and/or gestures. In some implementations, the object recognition process can include techniques such as template matching, pattern matching, contour matching, gesture recognition, skin recognition, shape matching, color or shape matching, and feature-based matching. In some implementations, the object recognition unit 714 calculates matching correlations between the detected portions of the user A (or portions thereof) and a predetermined set of patterns, and Determine whether any pattern within is matched or recognized. In some implementations, the object recognition unit 714 detects, recognizes, or determines the location within the interactive area where the portion of user A is detected. In some embodiments, the object recognition process may be performed on images or videos from the matting unit 712 that have not been subjected to the cropping process, thereby improving the accuracy of the object recognition process. can be done. In some embodiments, the object recognition unit 714 recognizes and extracts the image or video of the part of the user A within the interactive area, and transmits the extracted image or video to the result transmitting device 710.

The result sending device 710 is configured to send the output results of the object recognition unit 714 (which may include the output of the matting unit 712) to the message unit 32 of the backend server 30. In some implementations, the results sending device 710 may send the output directly to the results receiving device 810 instead of sending it via the message unit 32.

The user terminal 10B may be a user terminal operated by user B. The user terminal 10B includes a camera 800, a renderer 802, a display 804, an encoder 806, a decoder 808, a result receiving device 810, and an image processing device 812.

The camera 800 may be or include any type of video capture device. The camera 800 is configured to capture video data of user B, for example. The camera 800 sends captured video data to the encoder 806, the renderer 802, and/or the image processing device 812.

The renderer 802 receives video data from the camera 800 (e.g., user B's video data), and receives video data from the decoder 808 (which may include video data from another user, such as user A). , configured to receive output data of the image processing device 812 and generate rendered video for display on the display 804 (such as video displaying a group call in which User A and User B are displayed). Ru.

The display 804 is configured to display rendered video from the renderer 802. In some implementations, the display 804 may be a screen on the user terminal 10B.

The encoder 806 is configured to encode data, including the video data from the camera 800 and/or the video data from the image processing device 812. The encoder 806 transmits the encoded video data to the data receiving device 400 of the streaming server 40 . The encoded data may be transmitted as streaming data.

The decoder 808 receives video data or streaming data (which may include video data from user A) from the data transmitting device 402 of the streaming server 40, decodes the video data, converts the data into decoded video data, and converts the data into decoded video data. The rendered video data is sent to the renderer 802 for rendering.

The result receiving device 810 is configured to receive output data from the message unit 32 of the backend server 30 and send the data to the image processing device 812 . The output data from the message unit 32 includes data or information from the matting unit 712 and the object recognition unit 714. In some implementations, the output data from the message unit 32 includes the results of the object recognition processing performed by the object recognition unit 714. For example, the output data from the message unit 32 may include information about matched or recognized patterns, objects or gestures. In some implementations, the output data from the message unit 32 includes information regarding the position within the interactive area (on the user terminal 10A), of which the part of user A is e.g. It is detected by the matting unit 712 or object recognition unit 714 of the terminal 10A. In some implementations, the output data from the message unit 32 includes a video or image of the portion of user A detected/recognized within the interactive area.

The image processing device 812 is configured to receive video data from the camera 800 and/or data or information from the results receiving device 810. In some implementations, the image processing device 812 performs image processing or video processing on the video data received from the camera 800 based on data or information received from the results receiving device 810. For example, receiving a result indicating that the object recognition process executed by the object recognition unit 714 has successfully recognized a predetermined pattern on the part of the user A (in the interactive area on the screen of the user terminal 10A). If the data received from device 810 indicates, the image processing device 812 may include, render, or overlay special effects corresponding to the predetermined pattern on the video data received from the camera 800. The superimposed video may then be sent to the renderer 802 and then displayed on the user terminal 804. In some implementations, the special effects data may be stored in storage on the user terminal 10B (not shown).

In some implementations, the message unit 32 determines the destination of its output data based on data from the matting unit 712 and/or data from the object recognition unit 714. In some implementations, the message unit 32 determines an area in which to stretch, duplicate, or reproduce the portion of user A based on the detected position of the portion of user A within the interactive area.

For example, as shown in FIG. 5, if the position of the interactive area A31 where the relevant part of user A is detected by the relevant matting unit 712 (or the relevant object recognition unit 714) is within the relevant sub-region A312, the relevant The message unit 32 can determine the user terminal of the user C as the destination to which the output data of the message unit 32 is to be sent. The relevant part of user A is then stretched or copied/reproduced/displayed within the region RC, which may be performed by the image processing device of the relevant user terminal of user C.

In another example, when the position of the interactive area A31 where the part of user A is detected by the matting unit 712 is within the sub-area A311, the message unit 32 outputs the output data of the message unit 32. The user terminal of user D can be determined as the destination for transmitting the message. Thereafter, the portion of user A is stretched or duplicated/reproduced/displayed within region RD, and this may be performed in cooperation with an image processing device and/or a renderer in user D's user terminal.

In yet another example, if the position of the interactive area A31 in which the part of user A is detected by the matting unit 712 is within the sub-area A313, the message unit 32 The user terminal of user B can be determined as the destination for transmitting the output data. The portion of user A is then stretched or duplicated/reproduced/displayed within the region RB, and this may be performed in cooperation with an image processing device and/or a renderer in the user terminal of user B.

In some embodiments, the output data of the message unit 32 may include an image or video of the part of user A detected in the interactive area of area RA. The image processing device 812 may then overlay, duplicate, or reproduce the portion of user A on the video of user B received from the camera 800. In this way, the portion of user A within the interactive area may be extended to area B without being represented as a graphical or animated object.

In some implementations, after receiving the image or video data of user A through the decoder 808, the image processing device 812 receives information from the message unit 32 (such as superimposing, duplicating, or reproducing the portion of user A in the interactive area on the video of user B received from the camera 800; can do. In this way, the portion of user A within the interactive area may be extended to area B without being represented as a graphical or animated object.

In some implementations, the matting unit 712 and/or the object recognition unit 714 may not be implemented within the user terminal 10A. For example, the matting unit 712 and the object recognition unit 714 may be implemented within the backend server 30 or the streaming server 40.

FIG. 9 shows an exemplary sequence diagram illustrating the operation of a communication system according to some embodiments of the present invention. In some implementations, FIG. 9 shows how a portion of a user (eg, user A) is extended into an area where another user (eg, user B) is displayed.

In step S200, the camera 700 of the user terminal 10A transmits the video data of user A to the matting unit 712 of the user terminal 10A.

In step S202, the matting unit 712 detects the part of the user A within the interactive area on the screen of the user terminal 10A. This detection may include a matting process and/or a cropping process. In some implementations, the matting unit 712 determines the location within the interactive area where the portion of user A is detected.

In step S204, the object recognition unit 714 of the user terminal 10A receives the output data from the matting unit 712, performs object recognition processing on the output of the matting unit 712, and converts the interactive area It is determined whether any predetermined pattern, gesture, or object is recognized in the part of user A detected within. In some implementations, the object recognition process may include a matching process, a gesture recognition process, and/or a skin recognition process.

In step S206, when the object recognition unit 714 recognizes a predetermined pattern, gesture, or object, the object recognition unit 714 collects related information such as the position and size of the predetermined pattern, gesture, or object, and collects related information such as the position and size of the predetermined pattern, gesture, or object. Decide on the destination to send to.

In step S208, the output of the object recognition unit 714 is sent to the message unit 32 of the backend server 30 via the result transmission device 710 of the user terminal 10A.

In step S210, the message unit 32 determines the destination to which the data from the user terminal 10A is sent based on the information regarding the position of the part of the user A in the interactive area, which is included in the data from the user terminal 10A. Determine. The information may be determined in step S206, for example.

In step S211, the message unit 32 transmits the data from the user terminal 10A to the result receiving device 810 of the user terminal 10B (in the exemplary case where the message unit 32 determines the destination as user B or region RB). scenario).

In step S212, the result receiving device 810 transmits the received data to the image processing device 812 of the user terminal 10B.

In step S214, the image processing device 812 converts the detected portion of user A (or a portion of the detected portion of user A within the interactive area of area RA) into the video of user B. Overlay or superimpose on data. In some implementations, the image or video data of the detected portion of user A is sent to the user terminal 10B via the streaming server 40. In some implementations, the image or video data of the detected portion of user A is sent to the user terminal 10B via the message unit 32. The image or video data of user B is transmitted from the camera 800 of the user terminal 10B to the image processing device 812.

In step S216, the image processing device 812 sends the processed image or video data to the renderer 802 of the user terminal 10B for rendering. For example, the processed image or video data may be rendered with video data from the decoder 808 of the user terminal 10B and/or video data from the camera 800.

In step S218, the rendered video data is sent to the display 804 of the user terminal 10B for display on the screen of the user terminal 10B.

In step S220, the image processing device 812 sends the processed image or video data to the encoder 806 of the user terminal 10B for an encoding process.

In step S222, the encoded video data is sent to the streaming server 40.

In step S224, the streaming server 40 sends the encoded video data (from the user terminal 10B) to the decoder 708 of the user terminal 10A for a decoding process.

In step S226, the decoded video data is sent to the renderer 702 of the user terminal 10A for a rendering process.

In step S228, the rendered video data is sent to the display 804 for display on the screen of the user terminal 10A.

The example processes or steps described above may be performed continuously or periodically. For example, the matting unit 712 continuously or periodically detects user A's part within the interactive area. The object recognition unit 714 continuously or periodically performs recognition processing on the part of the user A within the interactive area. The message unit 32 continuously or periodically determines the destination to which the data received from the user terminal 10A is to be sent. The image processing device 812 of the user terminal 10B continuously or periodically performs the superimposition or superimposition process based on the information received from the message unit 32, and performs the process of superimposing or superimposing the stretched or reproduced/duplicated image in the area RB. ensure that that part of user A moved in synchronization with the part of user A in the area RA. In some embodiments, the user terminal 10B includes a processing unit such as a CPU or a GPU, and determines whether the stretched or reproduced portion of the user A in the region RB touches the image or video of the user B. to judge. The result of this determination can be used by the image processing device 812 to determine whether or not to include special effects in the region RB.

The present invention makes conference calls and group calls more convenient, fun, and interactive. The present invention can prevent misunderstandings when a user wants to talk about an object that is in another user's display area. The present invention can increase user motivation to participate in a group call chat room (which may be in the form of live streaming). The present invention can attract more streamers and viewers to participate in live streaming group calls.

The processes and procedures described in this invention, in addition to those explicitly described, may be implemented by software, hardware, or any combination thereof. For example, the processes and procedures described herein implement logic corresponding to the processes and procedures in a medium such as an integrated circuit, volatile memory, non-volatile memory, non-transitory computer-readable medium, magnetic disk, etc. This can be achieved by Further, the processes and procedures described in this specification can be realized as a computer program corresponding to the processes and procedures, and can be executed by various computers.

The systems or methods described in the above embodiments may be integrated into programs stored on non-transitory computer-readable media, such as solid state storage, optical disk storage, magnetic disk storage, and the like. Alternatively, the program may be downloaded from a server via the Internet and executed by the processor.

Although the technical contents and features of the present invention have been described above, those with ordinary knowledge in the technical field to which the present invention pertains will appreciate that many variations and modifications can be made without departing from the teachings and disclosure of the present invention. It can be performed. Therefore, the scope of the invention is not limited to the embodiments already disclosed, but includes other variations and modifications that do not depart from the invention and are included in the scope of the claims.

S1 Screen RA Area RB Area RC Area RD Area A1 Part A11 Part A2 Part A21 Part A3 Boundary A31 Interactive area A311 Sub area A312 Sub area A313 Sub area A32 Area B1 Object B3 Boundary B31 Interactive area B32 Area BR1 Boundary line BR2 Boundary line SP1 Special effects 1 System 10 User terminal 30 Back-end server 32 Message unit 40 Streaming server 400 Data receiving device 402 Data transmitting device 90 Network 700 Camera 702 Renderer 704 Display 706 Encoder 708 Decoder 710 Result transmitting device 712 Matting unit 714 Object recognition unit 800 Camera 802 Renderer 804 Display 806 Encoder 808 Decoder 810 Result receiving device 812 Image processing device

Claims (19)

An image processing method,
Displaying live video of the first user in a first area on the user terminal;
Displaying a second user's video in a second area on the user terminal;
a portion of the live video of the first user is extended to the second area on the user terminal ;
further comprising determining the portion of the live video of the first user by detecting a portion of the first user that exceeds a boundary within the first area on the user terminal; An image processing method characterized by: further defining an interactive area within the first area;
detecting a portion of the first user within the interactive area;
displaying the portion of the first user within the second area;
The video processing method according to claim 1, characterized in that it includes: 3. The video processing method of claim 2, wherein the step of detecting the part of the first user within the interactive area includes a matting process. 3. The video processing method according to claim 2, wherein the step of detecting the part of the first user within the interactive area includes an object recognition process. 5. The object recognition process further comprises the step of displaying a special effect on the user terminal when a predetermined pattern is recognized in the part of the first user. Video processing method. 5. The video processing method according to claim 4, wherein the object recognition process includes a gesture recognition process or a skin recognition process. 3. The video processing method according to claim 2, wherein the step of detecting the part of the first user within the interactive area includes image comparison processing or moving object detection processing. The first user and the second user are arranged horizontally on the user terminal, and the part of the live video of the first user is directed towards the second user within the first area. The video processing method according to claim 1, wherein the video processing method is stretched. The first user and the second user are vertically aligned on the user terminal, and the part of the live video of the first user is directed toward the second user within the first area. The video processing method according to claim 1, wherein the video processing method is stretched. The first user and the second user are diagonally arranged on the user terminal, and the part of the live video of the first user is directed toward the second user within the first area. 2. The video processing method according to claim 1, wherein the video processing method is performed by stretching the video. An image processing method,
Displaying live video of the first user in a first area on the user terminal;
Displaying a second user's video in a second area on the user terminal;
a portion of the live video of the first user is extended to the second area on the user terminal;
further defining an interactive area within the first area;
detecting a portion of the first user within the interactive area;
displaying the portion of the first user within the second area;
including;
further detecting a position of the portion of the first user within the interactive area;
determining a position of the second area based on the position of the portion of the first user within the interactive area;
A video processing method characterized by comprising: The video processing method according to claim 1 , wherein the position of the boundary line is determined by the first user. The video processing method according to claim 1 , wherein the boundary line corresponds to a direction of the second area with respect to the first area and is between the first area and the second area. . The video processing method according to claim 1, wherein the video of the second user is a live video. The video processing method according to claim 1, wherein the part of the live video of the first user extended to the second area is represented as a graphical object. moreover. 5. The method of claim 1, further comprising the step of displaying a special effect in the second area when the part of the live video of the first user is stretched to touch the second area of the second user. 1. The video processing method according to 1. A video processing system comprising one or more processors, the one or more processors executing machine-readable instructions,
Displaying live video of the first user in a first area on the user terminal;
Displaying a second user's video in a second area on the user terminal;
a portion of the live video of the first user is extended to the second area on the user terminal ;
Further, determining the portion of the live video of the first user by detecting a portion of the first user that exceeds a boundary within the first area on the user terminal. An image processing system characterized by the following . The one or more processors execute the machine-readable instructions, and further:
defining an interactive area within the first area;
detecting a portion of the first user within the interactive area;
displaying the portion of the first user within the second area;
The video processing system according to claim 17 , characterized in that the video processing system executes the following. A non-transitory computer-readable medium containing a program for image processing, the program being stored in one or more computers,
Displaying live video of the first user in a first area on the user terminal;
Displaying a second user's video in a second area on the user terminal;
a portion of the live video of the first user is extended to the second area on the user terminal;
Further, determining the portion of the live video of the first user by detecting a portion of the first user that exceeds a boundary line within the first area on the user terminal. , a non-transitory computer-readable medium.