KR102191050B1 - Personalized shopping mall system using virtual camera - Google Patents

Abstract

The present invention relates to a personalized shopping mall system using a virtual camera, by generating a virtual camera for each user with a different ROI from an image of an offline store photographed by a physical camera, and including it in a shopping mall page and streaming It was designed to give people the feeling of shopping in an actual offline store.

Description

Personalized shopping mall system using virtual camera

The present invention relates to an online shopping mall, and more particularly, to a personalized shopping mall system using a virtual camera.

Technology that crops and outputs only the part where movement occurs or the part of the image of interest such as a person's face or vehicle license plate among the images output from the camera in Korean Patent Application Publication No. 10-2011-0072846 (2011. 06. 29) Are proposed.

In Korean Patent Laid-Open No. 10-2013-0039652 (2013. 04. 22), when searching for two or more subnets storing content, and when the search result is more than one subnet, requested content according to the network status among two or more searched subnets In the process of delivering content from a content server to users by selecting a subnet that provides a network and a node included in the subnet and delivering the content to the user, the optimal server node can be selected in the network to increase traffic efficiency. It is proposing a load balancing technology that can provide streaming services.

The present inventor creates a virtual camera for each user with a different ROI from an image of an offline store photographed by one physical camera using the image cropping technology described above, and streams it by including it in a shopping mall page. I did research on technology.

In addition, the inventors of the present invention use the above-described image cropping technology to sequentially connect images of offline stores each photographed by a plurality of physical cameras. We created a camera, included it in a shopping mall page, and researched the streaming technology.

In addition, the inventors of the present invention are the 3D objects from the offline store screen including the 3D objects detected using the stereo time difference of the 2D images of the offline store respectively photographed by a plurality of physical cameras that are arranged adjacent to each other to capture the image of the store in real time. A research was conducted on a technology for streaming by creating a plurality of different virtual cameras for each user using a location area as a region of interest (ROI) and including them in a shopping mall page.

In addition, the present inventor creates a virtual camera for each user with a different ROI from an image of an offline store photographed by a physical camera, and includes it in the shopping mall page, and includes it in the virtual camera area of the shopping mall pages for each user. A study was conducted on a technology for distributing and transmitting the streaming load of images of the region of interest played by virtual cameras for each user.

Republic of Korea Patent Publication No. 10-2011-0072846 (2011. 06. 29), Republic of Korea Patent Publication No. 10-2013-0039652 (2013. 04. 22)

The present invention is invented in the context of the above, and generates a virtual camera for each user with a different ROI from an image of an offline store photographed by a single physical camera, and includes it in a shopping mall page to stream a virtual camera. Its purpose is to provide a personalized shopping mall system using.

In addition, the present invention creates a virtual camera for each user with a different ROI from an offline store screen formed by sequentially connecting images of an offline store each photographed by a plurality of physical cameras, and includes it in a shopping mall page for streaming. Its purpose is to provide a personalized shopping mall system using a virtual camera.

In addition, the present invention provides 3D objects from an offline store screen including 3D objects detected using a stereo time difference of 2D images of an offline store respectively photographed by a plurality of physical cameras that are arranged adjacent to each other to capture images of each store in real time. It is an object of the present invention to provide a personalized shopping mall system using a virtual camera that generates a plurality of different virtual cameras for each user having an area of interest as an ROI and includes them in a shopping mall page for streaming.

In addition, the present invention creates a virtual camera for each user with a different ROI from an image of an offline store photographed by a physical camera and includes it in a shopping mall page, but includes it in the virtual camera area of the shopping mall pages for each user. An object of the present invention is to provide a load balancing system for a personalized shopping mall page using a virtual camera that distributes and transmits the streaming load of images of a region of interest played by virtual cameras for each user.

According to an aspect of the present invention for achieving the above object, a personalized shopping mall system using a virtual camera includes a plurality of physical cameras for real-time recording of an offline store image; An integrated offline store screen is formed by sequentially connecting an image receiving unit for receiving images of an offline store respectively photographed by the plurality of physical cameras and images of the offline store received by the image receiving unit, and configured offline store An image processing apparatus including a virtual camera manager for generating a plurality of virtual cameras for each user having different ROIs from the screen; A region of interest selection unit that selects a region of interest among images of an offline store photographed by the physical camera, an region of interest reproducing unit that plays an image of the region of interest selected by the region of interest selection unit, and a user terminal Accordingly, there is provided a virtual camera including an image processing unit that performs pan/tilt or zoom-in/zoom-out of the image of the region of interest reproduced by the region of interest reproducing unit; It characterized in that it includes.

According to an additional aspect of the present invention, the virtual camera management unit includes a connection management unit that processes user terminal access and disconnection; A virtual camera generation unit for generating a virtual camera to be provided to a user terminal that has been accessed by the connection management unit; A shopping mall page generation unit that configures a shopping mall page in which the virtual camera generated by the virtual camera generation unit is included in the virtual camera area; A shopping mall page providing unit for providing a shopping mall page in which the virtual camera generated by the shopping mall page generating unit is included in a virtual camera area to a user terminal; A control right granting unit for granting a control right for a virtual camera included in a shopping mall page provided by the shopping mall page providing unit to a corresponding user terminal; It characterized in that it includes.

According to an additional aspect of the present invention, the control right for the virtual camera is a virtual camera screen pan (Pan) / tilt (Tilt) and zoom in (Zoom In) / zoom out (Zoom Out) rights.

According to an additional aspect of the present invention, the shopping mall page generated by the shopping mall page generation unit includes a product of interest area in which an image of a product of interest uploaded by the manager terminal is registered.

According to an additional aspect of the present invention, the shopping mall page generated by the shopping mall page generation unit includes a video chat area for processing video chat between a manager terminal and a user terminal.

According to an additional aspect of the present invention, the shopping mall page generated by the shopping mall page generator includes a store information area providing offline store information.

According to an additional aspect of the present invention, in response to a Pan/Tilt or Zoom In/Zoom Out operation signal of the virtual camera, the region of interest reproduced by the region of interest reproducing unit A simulated pan-tilt-zoom processing unit for outputting a virtual camera control command instructing to pan/tilt or zoom in/zoom out an image within a range of a virtual camera area to the image processing unit; It characterized in that it further includes.

According to an additional aspect of the present invention, the virtual camera instructs to physically pan/tilt or zoom in/zoom out when the image of the ROI reproduced by the ROI reproducing unit is out of the virtual camera area range. An optical pan-tilt-zoom processor for outputting a physical camera driving command to the physical camera; It characterized in that it further includes.

According to an additional aspect of the present invention, the virtual camera comprises a setting information storage unit for storing virtual camera setting information; A virtual camera setting unit that receives virtual camera setting information from a user and stores the setting information storage unit; It characterized in that it further includes.

According to an additional aspect of the present invention, the virtual camera setting unit may set at least one of an initial region of interest location, region of interest size, resolution, quality, compression format, and virtual camera region pattern as virtual camera setting information.

According to an additional aspect of the present invention, the image processing unit includes a memory for storing a store image photographed by a physical camera; A region of interest extracting unit for extracting the image of the region of interest selected by the region of interest selecting unit from the store image stored in the memory to the size of the region of interest set in camera setting information; An image scaler for converting the region of interest image extracted by the region of interest extracting unit into a resolution set in camera setting information; A codec for compressing the ROI image converted by the image scaler into a compression format set in camera setting information; It characterized in that it includes.

According to an additional aspect of the present invention, the image processing unit may include a frame rate converter configured to convert a frame rate of an ROI image according to a quality set in camera setting information; It characterized in that it further includes.

According to an additional aspect of the present invention, the image processing unit extracts a background object and a foreground object from the image of the region of interest, and the image of the region of interest including only the background object from which the foreground object is removed is It is characterized in that processing to be reproduced through the region reproducing unit.

According to an additional aspect of the present invention, the image processing unit is characterized in that the image processing unit recognizes a person from an image of a region of interest, and masks a face of the recognized person.

According to an additional aspect of the present invention, ROIs for each user selected by the region of interest selector may overlap each other.

According to an additional aspect of the present invention, the image processing apparatus converts ROI images reproduced by a plurality of user-specific virtual cameras into packet image signals originating from different ports with the same network address, and multiplexing them. A multiplexer configured to convert the video stream into a single packet video stream; An image transmission unit for transmitting a single packet image stream multiplexed by the multiplexing unit to user terminals through a single physical layer; It characterized in that it further includes.

According to an additional aspect of the present invention, the virtual camera management unit may include a control right recovery unit for recovering a control right for a virtual camera granted to a user terminal that has been disconnected by the connection management unit; It characterized in that it further includes.

According to an additional aspect of the present invention, the plurality of physical cameras are installed in a row.

According to an additional aspect of the present invention, the plurality of physical cameras are radially installed.

According to an additional aspect of the present invention, the virtual camera management unit connects so that some of the images of the offline store are overlapped.

The present invention creates a virtual camera for each user with a different ROI from an image of an offline store captured by a single physical camera and includes it in a shopping mall page and streams it, giving the user the feeling of shopping in an actual offline store. There is an effect that can give.

In addition, the present invention creates a virtual camera for each user with a different ROI from an offline store screen formed by sequentially connecting images of an offline store each photographed by a plurality of physical cameras, and includes it in a shopping mall page for streaming. By doing so, it has the effect of giving users the same feeling as shopping in an actual offline store.

In addition, the present invention provides 3D objects from an offline store screen including 3D objects detected using a stereo time difference of 2D images of an offline store respectively photographed by a plurality of physical cameras that are arranged adjacent to each other to capture images of each store in real time. By creating a plurality of different virtual cameras for each user with an area of interest as an ROI, including them in a shopping mall page and streaming them, it is possible to give the user a feeling of shopping in an actual offline store.

In addition, the present invention creates a virtual camera for each user with a different ROI from an image of an offline store photographed by a physical camera and includes it in a shopping mall page, but includes it in the virtual camera area of the shopping mall pages for each user. By distributing and transmitting the streaming load of the images of the region of interest played by the virtual cameras for each user, it is possible to provide a shopping mall page that can give users the feeling of shopping in an actual offline store at a high speed.

1 is a block diagram showing the configuration of a first embodiment of a personalized shopping mall system using a virtual camera according to the present invention.
2 is a block diagram showing the configuration of a second embodiment of a personalized shopping mall system using a virtual camera according to the present invention.
FIG. 3 is a diagram illustrating an example of an installation form and a store screen of physical cameras in the personalized shopping mall system using the virtual camera shown in FIG. 2.
FIG. 4 is a diagram illustrating another example of an installation form of physical cameras and a store screen of the personalized shopping mall system using the virtual camera shown in FIG. 2.
5 is a block diagram showing the configuration of a third embodiment of a personalized shopping mall system using a virtual camera according to the present invention.
6 is a diagram illustrating an example of an installation form of physical cameras in the personalized shopping mall system using the virtual camera shown in FIG. 5.
7 is a block diagram showing the configuration of a fourth embodiment of a personalized shopping mall system using a virtual camera according to the present invention.
8 is a block diagram showing the configuration of an embodiment of a virtual camera management unit of an image processing apparatus of a personalized shopping mall system using a virtual camera according to the present invention.
9 is a diagram illustrating an example of a shopping mall page generated by an image processing apparatus of a personalized shopping mall system using a virtual camera according to the present invention.
10 is a block diagram showing a configuration of an embodiment of a virtual camera of a personalized shopping mall system using a virtual camera according to the present invention.
FIG. 11 is a flowchart illustrating an example of an operation of providing a shopping mall page in the personalized shopping mall system using the virtual camera shown in FIG. 1.
12 is a flowchart illustrating an example of an operation of providing a shopping mall page in the personalized shopping mall system using the virtual camera shown in FIG. 2.
13 is a flowchart illustrating an example of an operation of providing a shopping mall page in the personalized shopping mall system using the virtual camera shown in FIG. 5.
14 is a flowchart illustrating an example of an operation of providing a shopping mall page in the personalized shopping mall system using the virtual camera shown in FIG. 7.
15 is a flowchart illustrating an example of a procedure for extracting an ROI by the virtual camera shown in FIG. 10.

Hereinafter, the present invention will be described in detail so that those skilled in the art can easily understand and reproduce the present invention through preferred embodiments described with reference to the accompanying drawings.

In describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the embodiments of the present invention, the detailed description will be omitted.

Terms used throughout the specification of the present invention are terms defined in consideration of functions in the embodiments of the present invention, and can be sufficiently modified according to the intention and custom of the user or operator. It should be made based on the contents of

1 is a schematic diagram of a personalized shopping mall system using a virtual camera according to the present invention. As shown in FIG. 1, a personalized shopping mall system using a virtual camera according to this embodiment includes one physical camera 100 and an image processing device 200.

2 is a block diagram showing the configuration of a second embodiment of a personalized shopping mall system using a virtual camera according to the present invention. As shown in FIG. 2, a personalized shopping mall system using a virtual camera according to this embodiment includes a plurality of physical cameras 100 and an image processing apparatus 200.

FIG. 3 is a diagram illustrating an example of an installation form of physical cameras and a store screen of the personalized shopping mall system using the virtual camera shown in FIG. 2. As shown in FIG. 3, the plurality of physical cameras 100 may be installed in a line.

FIG. 4 is a diagram illustrating another example of an installation form of physical cameras and a store screen of the personalized shopping mall system using the virtual camera shown in FIG. 2. It may be installed radially as shown in FIG. 4.

5 is a block diagram showing the configuration of a third embodiment of a personalized shopping mall system using a virtual camera according to the present invention. As shown in FIG. 5, a personalized shopping mall system using a virtual camera according to this embodiment includes a plurality of adjacent physical cameras 100 and an image processing device 200.

6 is a diagram illustrating an example of an installation form of physical cameras in a personalized shopping mall system using the virtual camera shown in FIG. 5. As shown in FIG. 6, a plurality of physical cameras 100 disposed adjacent to each other may be installed in all directions.

7 is a block diagram showing the configuration of a fourth embodiment of a personalized shopping mall system using a virtual camera according to the present invention. As shown in FIG. 7, the personalized shopping mall system using a virtual camera according to this embodiment includes a physical camera 100, an image processing device 200, a local server 300, and a dedicated streaming server 400. ), including.

The camera 100 is installed in an offline store and captures an image of the offline store in real time. For example, the camera 100 may be a general high-resolution camera, a panoramic camera, a fisheye lens camera, or the like.

The image processing device 200 generates a plurality of user-specific virtual cameras having different ROIs from images of an offline store photographed by the physical camera 100, and the virtual cameras for each user are in the virtual camera area. A number of included shopping mall pages for each user are configured and provided to each user terminal.

In this case, the virtual camera for each user with a different ROI refers to a software module that crops and plays an image of a personalized ROI different for each user from an image of an offline store.

The local server 300 converts the region-of-interest images played by virtual cameras for each user included in the virtual camera area of shopping mall pages for each user provided to the user terminal by the image processing device 200 to each user terminal. Stream.

When the streaming load of the local server 300 exceeds a threshold, the streaming-only server 400 provides interest played by virtual cameras for each user included in the virtual camera area of shopping mall pages for each user instead of the local server. The streaming load of region images is distributed and transmitted.

8 is a block diagram showing the configuration of an embodiment of a virtual camera management unit of an image processing apparatus of a personalized shopping mall system using a virtual camera according to the present invention. The image processing apparatus 200 includes an image receiving unit 210 and a virtual camera management unit 220.

(Example 1)

The image receiving unit 210 receives an image of an offline store photographed by the physical camera 100.

The virtual camera management unit 220 generates a plurality of virtual cameras for each user having different ROIs from the images of the offline store received by the image receiving unit 210.

Meanwhile, the virtual camera management unit 220 includes a connection management unit 221, a virtual camera generation unit 222, a shopping mall page generation unit 223, a shopping mall page providing unit 224, and a control right grant unit 225. Includes.

The access management unit 221 processes user terminal access and disconnection. For example, the access management unit 221 may be implemented to process user terminal access by searching for a user ID and password input from a user terminal requesting access to the image processing apparatus 200 from a member registration DB (not shown). .

Meanwhile, the access management unit 221 may be implemented to process the disconnection of the user terminal according to a request for disconnection from the user terminal connected to the image processing apparatus 200. In this case, the user terminal may be a portable terminal such as a PC or a smart phone.

The virtual camera generation unit 222 generates a virtual camera to be provided to a user terminal that has been accessed by the connection management unit 221. For example, the virtual camera generator 222 may be implemented to generate a plurality of virtual cameras for each user having different ROIs from the received image of the offline store.

In this case, the virtual camera is a software module that crops and reproduces an image of a specific ROI of interest to a user from an image of an offline store, which will be described in detail later.

The shopping mall page generation unit 223 configures a shopping mall page in which the virtual camera generated by the virtual camera generation unit 222 is included in the virtual camera area. In this case, the shopping mall page provides a product of interest area in which an image of a product of interest uploaded by the manager terminal (not shown in the drawing) is registered, a video chat area that processes video chat between the manager terminal and the user terminal, and offline store information. It may further include a store information area.

The shopping mall page providing unit 224 provides a shopping mall page in which a virtual camera generated by the shopping mall page generating unit 223 is included in a virtual camera area to a user terminal.

When a shopping mall page is provided to a user terminal, a virtual camera included in the virtual camera area of the shopping mall page streams an interest area image of an actual offline store in real time.

The user feels like visiting an actual offline store and shopping while watching the video of an area of interest of an offline store streamed in real time by a virtual camera online.

The control right granting unit 225 grants a control right for a virtual camera included in the shopping mall page provided by the shopping mall page providing unit 224 to a corresponding user terminal.

In this case, the control right for the virtual camera may be the right to Pan/Tilt and Zoom In/Zoom Out on the virtual camera screen. On the other hand, the virtual camera screen Pan/Tilt and Zoom In/Zoom Out are not physical motion control, but software-based screen up/down/left/right movement and screen enlargement/reduction processing. to be.

For example, when a finger is touched in the up/down/left/right direction through the touch screen of the user terminal to which the control right is granted, a virtual camera screen pan/tilt event occurs, and the area of interest is moved and other offline stores The image of the region may be streamed and played by a virtual camera.

For example, when two fingers are spread or pinched through the touch screen of a user terminal to which control rights are granted, a virtual camera screen zoom in/zoom out event occurs, and the ROI image can be enlarged or reduced. have.

Each user independently performs control of the virtual camera included in the shopping mall page provided to his or her user terminal to receive the image of the area of interest to the user. It is not played.

By implementing in this way, a virtual camera for each user with a different ROI is created from the image of the offline store captured by a physical camera, and it is included in the shopping mall page and streamed, giving users the feeling of shopping in an actual offline store. Can give

(Example 2)

The image receiving unit 210 receives images of an offline store respectively photographed by the plurality of physical cameras 100.

The virtual camera management unit 220 sequentially connects the images of the offline store received by the image receiving unit 210 to configure an integrated offline store screen, and a plurality of regions of interest (ROI) different from the configured offline store screen Create virtual cameras for each user.

In this case, the virtual camera management unit 220 may be implemented to form an offline store screen by connecting some of the images of the offline store to overlap. In the case of connecting so that some of the videos of the offline store are overlapped, the missing area can be prevented.

Meanwhile, the virtual camera management unit 220 includes a connection management unit 221, a virtual camera generation unit 222, a shopping mall page generation unit 223, a shopping mall page providing unit 224, and a control right grant unit 225. Includes.

The access management unit 221 processes user terminal access and disconnection. For example, the access management unit 221 may be implemented to process user terminal access by searching for a user ID and password input from a user terminal requesting access to the image processing apparatus 200 from a member registration DB (not shown). .

Meanwhile, the access management unit 221 may be implemented to process the disconnection of the user terminal according to a request for disconnection from the user terminal connected to the image processing apparatus 200. In this case, the user terminal may be a portable terminal such as a PC or a smart phone.

The virtual camera generation unit 222 generates a virtual camera to be provided to a user terminal that has been accessed by the connection management unit 221. For example, the virtual camera generation unit 222 sequentially connects the received videos of the offline store to form an integrated offline store screen, and a plurality of virtual cameras for each user having different ROIs from the configured offline store screen Can be implemented to generate.

In this case, the virtual camera is a software module that crops and reproduces an image of a specific ROI of interest to a user from an image of an offline store, which will be described in detail later.

The shopping mall page generation unit 223 configures a shopping mall page in which the virtual camera generated by the virtual camera generation unit 222 is included in the virtual camera area. In this case, the shopping mall page provides a product of interest area in which an image of a product of interest uploaded by the manager terminal (not shown in the drawing) is registered, a video chat area that processes video chat between the manager terminal and the user terminal, and offline store information. It may further include a store information area.

The shopping mall page providing unit 224 provides a shopping mall page in which a virtual camera generated by the shopping mall page generating unit 223 is included in a virtual camera area to a user terminal.

When a shopping mall page is provided to a user terminal, a virtual camera included in the virtual camera area of the shopping mall page streams an interest area image of an actual offline store in real time.

The user feels like visiting an actual offline store and shopping while watching the video of an area of interest of an offline store streamed in real time by a virtual camera online.

The control right granting unit 225 grants a control right for a virtual camera included in the shopping mall page provided by the shopping mall page providing unit 224 to a corresponding user terminal.

In this case, the control right for the virtual camera may be the right to Pan/Tilt and Zoom In/Zoom Out on the virtual camera screen. On the other hand, the virtual camera screen Pan/Tilt and Zoom In/Zoom Out are not physical motion control, but software-based screen up/down/left/right movement and screen enlargement/reduction processing. to be.

For example, when a finger is touched in the up/down/left/right direction through the touch screen of the user terminal to which the control right is granted, a virtual camera screen pan/tilt event occurs, and the area of interest is moved and other offline stores The image of the region may be streamed and played by a virtual camera.

For example, when two fingers are spread or pinched through the touch screen of a user terminal to which control rights are granted, a virtual camera screen zoom in/zoom out event occurs, and the ROI image can be enlarged or reduced. have.

Each user independently performs control of the virtual camera included in the shopping mall page provided to his or her user terminal to receive the image of the area of interest to the user. It is not played.

By implementing in this way, a virtual camera for each user with a different ROI is created from the image of the offline store captured by a physical camera, and it is included in the shopping mall page and streamed, giving users the feeling of shopping in an actual offline store. Can give

(Example 3)

The image receiving unit 210 receives 2D images of an offline store respectively photographed by a plurality of physical cameras 100 disposed adjacent to each other.

The virtual camera management unit 220 detects 3D objects using the stereo time difference of 2D images of the offline store received by the image receiving unit 210, and configures and configures an offline store screen including the detected 3D objects. A plurality of different virtual cameras for each user are created using an area in which 3D objects are located from the screen as an ROI.

At this time, the offline store screen configured by the virtual camera management unit 220 is captured in real time by a plurality of physical cameras 100 installed in one of a plurality of physical cameras 100 installed in all directions and synthesized. It may be a screen of an offline store.

On the other hand, the stereo vision technique, which detects 3D objects using stereo visual differences, is a process in which the human visual system accepts different images due to the difference in the position of the left/right eyes, and the brain receives them as a three-dimensional shape to have a sense of distance. This is a 3D image processing technique using the principle of generating a three-dimensional effect in, and since it is a conventional technique already known and implemented before this filing, a detailed description thereof will be omitted.

Meanwhile, the virtual camera management unit 220 includes a connection management unit 221, a virtual camera generation unit 222, a shopping mall page generation unit 223, a shopping mall page providing unit 224, and a control right grant unit 225. Includes.

The access management unit 221 processes user terminal access and disconnection. For example, the access management unit 221 may be implemented to process user terminal access by searching for a user ID and password input from a user terminal requesting access to the image processing apparatus 200 from a member registration DB (not shown). .

Meanwhile, the access management unit 221 may be implemented to process the disconnection of the user terminal according to a request for disconnection from the user terminal connected to the image processing apparatus 200. In this case, the user terminal may be a portable terminal such as a PC or a smart phone.

The virtual camera generation unit 222 generates a virtual camera to be provided to a user terminal that has been accessed by the connection management unit 221. For example, the virtual camera generator 222 detects 3D objects using stereo time difference of 2D images of the received offline store, configures an offline store screen including the detected 3D objects, and places 3D objects from the configured offline store screen. It may be implemented to create a plurality of different virtual cameras for each user with the region as the region of interest (ROI).

In this case, the virtual camera is a software module that crops and plays an image of a personalized ROI in which different 3D objects of interest are located for each user from an offline store screen, and will be described in detail later.

The shopping mall page generation unit 223 configures a shopping mall page in which the virtual camera generated by the virtual camera generation unit 222 is included in the virtual camera area. In this case, the shopping mall page provides a product of interest area in which an image of a product of interest uploaded by the manager terminal (not shown in the drawing) is registered, a video chat area that processes video chat between the manager terminal and the user terminal, and offline store information. It may include a store information area.

The shopping mall page providing unit 224 provides a shopping mall page in which a virtual camera generated by the shopping mall page generating unit 223 is included in a virtual camera area to a user terminal.

When a shopping mall page is provided to a user terminal, a virtual camera included in the virtual camera area of the shopping mall page streams an interest area image of an actual offline store in real time. At this time, the 3D object detected by the stereo visual difference is rendered and displayed in the virtual camera area.

The user feels like visiting an actual offline store and shopping while watching a 3D object located in an area of interest image of an offline store streamed in real time by a virtual camera online.

The control right granting unit 225 grants a control right for a virtual camera included in the shopping mall page provided by the shopping mall page providing unit 224 to a corresponding user terminal.

In this case, the control right for the virtual camera may be a rotation, zoom in/zoom out, or pan/tilt authority of a 3D object displayed by being rendered. On the other hand, rotation, zoom in/zoom out, or pan/tilt of 3D objects rendered and displayed on the virtual camera screen is not a physical motion control, but a software screen rotation. , Screen enlargement/reduction and screen movement processing.

For example, when a finger is touched in the up/down/left/right direction through the touch screen of the user terminal to which the control right is granted, a 3D object rotation event occurs, and a 3D object rendered and displayed on the virtual camera screen is implemented to rotate in the corresponding direction. I can.

For example, when two fingers are spread or pinched through the touch screen of a user terminal to which control rights are granted, a 3D object zoom in/zoom out event occurs, and the 3D object rendered and displayed on the virtual camera screen is enlarged. It can be implemented to be reduced or reduced.

Each user independently performs control of the virtual camera included in the shopping mall page provided to his or her user terminal to receive an image of an area including a 3D object of interest. The video is played for each user terminal.

As a result of this implementation, 3D objects are generated from the screen of the offline store including 3D objects detected using the stereo time difference of the 2D images of the offline store respectively photographed by a plurality of physical cameras that are arranged adjacent to each other and photograph the image of the store in real time. By creating a plurality of different virtual cameras for each user with a region of interest as an ROI, including them in a shopping mall page and streaming them, the user can feel as if shopping in an actual offline store.

At this time, it is possible to control rotation, zoom in/zoom out, or pan/tilt for 3D objects rendered and displayed through a virtual camera for each user, When streaming video using a single camera, it is possible to overcome the limitation that only one person can control the video.

(Example 4)

The image receiving unit 210 receives an image of an offline store photographed by the physical camera 100.

The virtual camera management unit 220 generates a plurality of virtual cameras for each user having different ROIs from the images of the offline store received by the image receiving unit 210.

Meanwhile, the virtual camera management unit 220 includes a connection management unit 221, a virtual camera generation unit 222, a shopping mall page generation unit 223, a shopping mall page providing unit 224, and a control right grant unit 225. Includes.

The access management unit 221 processes user terminal access and disconnection. For example, the access management unit 221 may be implemented to process user terminal access by searching for a user ID and password input from a user terminal requesting access to the image processing apparatus 200 from a member registration DB (not shown). .

Meanwhile, the access management unit 221 may be implemented to process the disconnection of the user terminal according to a request for disconnection from the user terminal connected to the image processing apparatus 200. In this case, the user terminal may be a portable terminal such as a PC or a smart phone.

The virtual camera generation unit 222 generates a virtual camera to be provided to a user terminal that has been accessed by the connection management unit 221. For example, the virtual camera generator 222 may be implemented to generate a plurality of virtual cameras for each user having different ROIs from the received image of the offline store.

In this case, the virtual camera is a software module that crops and reproduces an image of a specific ROI of interest to a user from an image of an offline store, which will be described in detail later.

The shopping mall page generation unit 223 configures a shopping mall page in which the virtual camera generated by the virtual camera generation unit 222 is included in the virtual camera area. In this case, the shopping mall page provides a product of interest area in which an image of a product of interest uploaded by the manager terminal (not shown in the drawing) is registered, a video chat area that processes video chat between the manager terminal and the user terminal, and offline store information. It may further include a store information area.

The shopping mall page providing unit 224 provides a shopping mall page in which a virtual camera generated by the shopping mall page generating unit 223 is included in a virtual camera area to a user terminal.

When a shopping mall page is provided to a user terminal, a virtual camera included in the virtual camera area of the shopping mall page streams an interest area image of an actual offline store in real time.

The user feels like visiting an actual offline store and shopping while watching the video of an area of interest of an offline store streamed in real time by a virtual camera online.

The control right granting unit 225 grants a control right for a virtual camera included in the shopping mall page provided by the shopping mall page providing unit 224 to a corresponding user terminal.

In this case, the control right for the virtual camera may be the right to Pan/Tilt and Zoom In/Zoom Out on the virtual camera screen. On the other hand, the virtual camera screen Pan/Tilt and Zoom In/Zoom Out are not physical motion control, but software-based screen up/down/left/right movement and screen enlargement/reduction processing. to be.

For example, when a finger is touched in the up/down/left/right direction through the touch screen of the user terminal to which the control right is granted, a virtual camera screen pan/tilt event occurs, and the area of interest is moved and other offline stores The image of the region may be streamed and played by a virtual camera.

For example, when two fingers are spread or pinched through the touch screen of a user terminal to which control rights are granted, a virtual camera screen zoom in/zoom out event occurs, and the ROI image can be enlarged or reduced. have.

Each user independently performs control of the virtual camera included in the shopping mall page provided to his or her user terminal to receive the image of the area of interest to the user. It is not played.

The local server 300 streams the ROI images played by the virtual cameras for each user included in the virtual camera area of the shopping mall pages for each user provided to the user terminal by the image processing device 200 to each user terminal. do.

On the other hand, when the streaming load of the local server 300 exceeds the threshold, the streaming dedicated server 400 is played by virtual cameras for each user included in the virtual camera area of the shopping mall pages for each user instead of the local server. The streaming load of the ROI images is distributed and transmitted.

That is, when a shopping mall page is provided to a user terminal by the image processing device 200, an image of an area of interest of an actual offline store is transmitted to the local server 300 or a streaming dedicated server by a virtual camera included in the virtual camera area of the shopping mall page. 400).

In this case, depending on the streaming load of the local server 300, the local server 300 directly streams the ROI images played by the virtual cameras or indirectly to the virtual cameras through the streaming dedicated server 400. Streams the ROI images played back by.

Meanwhile, the load balancing system for a personalized shopping mall page using a virtual camera according to this embodiment may further include a shopping mall management server 500. The shopping mall management server 500 monitors the streaming load of the local server 300, determines whether the streaming load of the local server exceeds a threshold value, and when it is determined that the streaming load of the local server exceeds the threshold value, gives the streaming control right. It is collected from the local server 300 and transferred to the streaming dedicated server 400.

For example, determining whether the streaming load of the local server exceeds the threshold is the number of sessions per second that can be processed (SPS: Session Per Second), the number of packets per second that can be processed (PPS: Packet Per Second), and can be processed. It is possible by detecting the number of transactions per second (TPS), etc.

Meanwhile, the shopping mall management server 500 transfers the streaming control right from the streaming dedicated server 400 when the streaming load of the local server 300 falls below the threshold value in a state in which the streaming control right is transferred to the streaming dedicated server 400. It is recovered and returned to the local server 300.

By implementing in this way, the user feels like visiting an actual offline store and shopping while watching the video of an area of interest of an offline store streamed in real time by a virtual camera online.

In addition, it is possible to provide a shopping mall page at a high speed by distributing and transmitting the streaming load of images of a region of interest played by virtual cameras for each user included in the virtual camera area of the shopping mall pages for each user.

A shopping mall page generated according to the first to fourth embodiments is described with reference to FIG. 9. 9 is a diagram illustrating an example of a shopping mall page generated by an image processing apparatus of a personalized shopping mall system using a virtual camera according to the present invention. It can be seen that the shopping mall page 600 includes a virtual camera area 610, a product of interest area 620, a video chat area 630, and a store information area 640.

Meanwhile, according to an additional aspect of the present invention, the image processing apparatus 200 may further include a multiplexer 230 and an image transmission unit 240.

The multiplexer 230 converts ROI images reproduced by a plurality of user-specific virtual cameras into packet image signals originating from different ports and having the same network address, and multiplexing them into a single packet image stream. Convert.

The image transmission unit 240 transmits a single packet video stream multiplexed by the multiplexer 230 to user terminals through a single physical layer.

For example, ROI image signals output by a plurality of user-specific virtual cameras may be packet signals having respective network addresses. In this case, the multiplexer 230 uses a network address translation (NAT) function to convert the network addresses of each of the ROI video signals outputted by a plurality of user-specific virtual cameras into the same network address while converting a different port. It can be multiplexed by assigning a number.

For example, image signals of an ROI outputted by a plurality of virtual cameras for each user may be image stream data. In this case, the multiplexer 230 may convert and multiplex the video signals of each of the ROI video signals output by the plurality of user-specific virtual cameras into packets having the same network address and different port numbers.

For example, the address of an ROI video signal output by one virtual camera is assigned as '160,245,200,1:8091', and the address of an ROI video signal output by another virtual camera is '160,245,200,1:8092. Can be assigned to'. These multiplexed video packets having the same IP address may be output through the video transmission unit 240, which is a single physical layer (PHY).

Meanwhile, according to an additional aspect of the invention, the virtual camera management unit 220 may further include a control right collection unit 226. The control right collection unit 226 recovers the control right for the virtual camera granted to the user terminal that has been disconnected by the access management unit 221.

That is, in this embodiment, when a user terminal that has been granted control rights for a virtual camera by the control right granting unit 225 is disconnected, control rights for the virtual camera granted to the user terminal through the control right collecting unit 226 are granted. This is an embodiment to be recovered.

By implementing as described above, the present invention creates a virtual camera for each user with a different ROI from an image of an offline store photographed by a physical camera, includes it in a shopping mall page, and streams it to the user to shop at an actual offline store. It can give you the same feeling.

At this time, since Pan/Tilt control for the virtual camera is possible for each user, individual ROIs for each user can be overlapped with each other, so that the pan when streaming video using a single camera ( It can overcome the limitation that only one person can control Pan/Tilt.

10 is a block diagram showing a configuration of an embodiment of a virtual camera of a personalized shopping mall system using a virtual camera according to the present invention. As shown in FIG. 10, the virtual camera 700 included in the virtual camera area of the shopping mall page includes an ROI selection unit 710, an ROI reproducing unit 720, and an image processing unit 730.

In the first, second, and fourth embodiments, the region of interest selection unit 710 selects a region of interest from images of an offline store photographed by the physical camera 100. For example, it may be implemented such that an area in which popular products of an offline store are displayed is stored as an initial area of interest, and the area of interest selector 710 selects an area corresponding to the location of the initial area of interest as the area of interest.

In the case of the third embodiment, the region of interest selection unit 710 selects a region in which a specific 3D object is located on the offline store screen as the region of interest. For example, a specific 3D object through user selection for 3D objects detected by using stereo time difference of 2D images of an offline store each photographed by a plurality of physical cameras 100 disposed adjacent to the ROI selector 710 It may be implemented to select an area in which is located as an ROI.

In the first, second and fourth embodiments, the region of interest reproducing unit 720 reproduces the image of the region of interest selected by the region of interest selector 710. Accordingly, the user enjoys online shopping with the feeling of visiting an actual offline store and shopping while watching the video of the region of interest of the offline store streamed in real time by the virtual camera through online.

In the third embodiment, the region of interest reproducing unit 720 renders and displays the 3D object of the region of interest selected by the region of interest selection unit 710. Accordingly, the user enjoys online shopping with the feeling of visiting an actual offline store and shopping while watching the 3D object of the region of interest rendered and displayed in real time by the virtual camera through online.

In the first, second and fourth embodiments, the image processing unit 730 pans the image of the region of interest played by the region of interest reproducing unit 720 according to the manipulation of the user terminal. )/Tilt or Zoom In/Zoom Out.

In the case of the third embodiment, the image processing unit 730 rotates or zooms in/zooms out the 3D object rendered and displayed by the region of interest reproducing unit 720 according to the manipulation of the user terminal. Process (Zoom Out) or Pan/Tilt.

Meanwhile, when there are a plurality of 3D objects overlapping in the ROI, the ROI reproducing unit 720 may be implemented to display only a specific 3D object selected among them. That is, when another 3D object that overlaps the 3D object of interest to be displayed for rendering exists as a foreground or background object, the region of interest reproducing unit 720 selects and displays only the 3D object of interest to be displayed for rendering. The desired 3D object of interest may be selected by the region of interest selection unit 710.

In the case of the first, second, and fourth embodiments, the region of interest image selected by the region of interest selection unit 710 and reproduced by the region of interest reproducing unit 720 is a user who has been given control rights to the virtual camera. The virtual camera screen of the terminal is changed by Pan/Tilt and Zoom In/Zoom Out operations.

The image processing unit 730 pans/tilts the image of the region of interest according to the operation of pan/tilt and zoom-in/zoom-out of the camera screen of the user terminal. Or, zoom in/zoom out.

In the case of the third embodiment, the region of interest image selected by the region of interest selection unit 710 and played back by the region of interest reproducing unit 720 is a 3D object rotation of a user terminal that has been given control rights to the virtual camera. Alternatively, the camera screen is changed by zooming in/zooming out or pan/tilting.

The image processing unit 730 is included in the image of the region of interest according to the 3D object rotation of the user terminal or the camera screen zoom in/zoom out or pan/tilt operation. The 3D object is rotated or the image of the ROI is Pan/Tilt or Zoom In/Zoom Out.

Meanwhile, according to an additional aspect of the invention, the virtual camera 700 may further include a simulated pan tilt zoom processor 740. The simulated pan-tilt-zoom processing unit 740 displays the region of interest reproduced by the region of interest reproducing unit 720 according to a Pan/Tilt or Zoom In/Zoom Out operation signal. A virtual camera control command instructing to pan/tilt or zoom in/zoom out by software within the virtual camera area range is output to the image processing unit 730.

When a user manipulates Pan/Tilt or Zoom In/Zoom Out through the user terminal, the simulated pan/tilt zoom processing unit 740 responds to the user's manipulation, and the ROI playback unit ( 720) outputs to the image processing unit 730 a virtual camera control command instructing to pan/tilt or zoom in/zoom out the image of the region of interest played by the image processing unit 730, and the image processing unit 730 The region of interest is changed by pan/tilting or zooming in/out by software.

Meanwhile, according to an additional aspect of the present invention, the virtual camera 700 may further include an optical pan tilt zoom processing unit 750. The optical pan and tilt zoom processing unit 750 physically pans/tilts or zooms in/zooms out the physical camera 100 when the image of the region of interest reproduced by the region of interest reproducing unit 720 is out of the virtual camera region range. A physical camera driving command instructing to perform is output to the physical camera 100.

There is a limit to software pan/tilt or zoom-in/zoom-out control of the virtual camera by the simulated pan-tilt-zoom processor 740. If the image of the region of interest is out of the range of the virtual camera region, software pan/tilt or zoom-in/zoom-out control for the virtual camera is not possible. You need to control pan/tilt or zoom in/zoom out.

When the image of the region of interest is out of the virtual camera region range by pan/tilt or zoom in/zoom out operation through the user terminal, the optical pan/tilt zoom processor 750 A physical camera driving command instructing to physically pan/tilt or zoom in/zoom out is output to the physical camera 100, and the physical camera 100 moves to a position outside the range of the virtual camera area. The region of interest is changed by physically panning/tilting or zooming in/zooming out.

Meanwhile, according to an additional aspect of the invention, the virtual camera 700 may further include a setting information storage unit 760 and a virtual camera setting unit 770.

The setting information storage unit 760 stores virtual camera setting information. For example, the virtual camera setting information may include at least one of an initial region of interest location, region of interest size, resolution, quality, compression format, and virtual camera region pattern.

The virtual camera setting unit 770 receives virtual camera setting information from a user and stores it in the setting information storage unit 760. For example, the virtual camera setting unit 770 may be implemented to set at least one of an initial region of interest location, region of interest size, resolution, quality, compression format, and virtual camera region pattern as virtual camera setting information.

Meanwhile, according to an additional aspect of the invention, the image processing unit 730 may include a memory 731, an ROI extracting unit 732, an image scaler 733, and a codec 734.

The memory 731 stores store images photographed by a physical camera. For example, the memory 731 may be a semiconductor memory that buffers a store image signal captured by a physical camera and stores it for each frame. In this case, the size of the store image stored in the memory 731 may be equal to or smaller than the size of the store image photographed by a physical camera.

The region of interest extracting unit 732 extracts the image of the region of interest selected by the region of interest selection unit 710 from the store image stored in the memory 731 to the size of the region of interest set in camera setting information.

For example, the ROI extractor 732 may be implemented to extract an ROI image having a size of the ROI from store image frames buffered and stored in the memory 731 in real time. In this case, the size of the region of interest should be smaller than the size of the store image buffered and stored in the memory 731.

The image scaler 733 converts the ROI image extracted by the ROI extractor 732 into a resolution set in camera setting information. For example, if the resolution set in the camera setting information is '640*480', the image scaler 733 may convert the ROI image into '640*480' resolution.

The codec 734 compresses the ROI image converted by the image scaler 733 into a compression format set in camera setting information. For example, the codec 734 may compress and output an ROI image in a specific format such as MPEG-2.

Meanwhile, according to an additional aspect of the invention, the image processing unit 730 may further include a frame rate converter 735. The frame rate converter 735 converts the frame rate of the ROI image according to the quality set in the camera setting information. In this case, the quality set in the camera setting information may be set in stages such as high quality/medium quality/low quality, and the frame rate conversion unit 735 determines the relative frame rate as the quality of the ROI image is set to high quality. Converts to high.

On the other hand, according to an additional aspect of the invention, the image processing unit 730 extracts a background object and a foreground object from the image of the region of interest using video analysis technology, and the foreground object is removed. An image of the region of interest including only the background object may be reproduced through the region of interest reproducing unit 720.

For example, when a menu for'playing only the background image' is provided and the user selects a menu for'playing only the background image', the image processing unit 730 is an image of an ROI including only the background object from which the foreground object has been removed. Can be implemented to play.

In this embodiment, when a person passes or stays between a'product' displayed in a store and a camera, the product is hidden and invisible, so that a person as a foreground object is removed and reproduced so that only the product as a background object is visible.

Meanwhile, according to an additional aspect of the invention, the image processing unit 730 may be implemented to recognize a person from an image of an ROI, and to perform a masking process on the recognized person's face.

For example, the image processing unit 730 may be implemented to recognize a person among a plurality of objects existing in the region of interest, and mask the recognized person's face.

In this embodiment, a store image in which the faces of customers shopping in an offline store are masked is provided to online users, thereby making it impossible to identify customers shopping in an offline store, thereby protecting personal information.

The operation of providing a shopping mall page in a personalized shopping mall system using a virtual camera according to the present invention as described above will be described with reference to FIGS. 11 to 14.

FIG. 11 is a flowchart illustrating an example of an operation of providing a shopping mall page in the personalized shopping mall system using the virtual camera shown in FIG. 1.

First, when the user terminal is connected in step 810, the image processing apparatus generates a virtual camera to be provided to the user terminal.

In step 820, the image processing device configures a shopping mall page in which the virtual camera generated in step 810 is included in the virtual camera area.

In step 830, the image processing apparatus provides a shopping mall page in which the virtual camera generated in step 820 is included in the virtual camera area to the user terminal.

In step 840, the image processing apparatus grants control rights to the virtual camera included in the shopping mall page provided in step 830 to the corresponding user terminal.

Then, in step 850, the region of interest image among the images of the offline store captured in real time by the physical camera installed in the offline store is played back by the virtual camera included in the virtual camera area of the shopping mall page provided to the user terminal.

Then, in step 860, the pan/tilt of the ROI image according to the user's manipulation of the ROI image played by the virtual camera included in the virtual camera area of the user terminal that has been granted control rights to the virtual camera. ) Or a Zoom In/Zoom Out operation is performed.

By implementing in this way, the present invention creates a virtual camera for each user with a different ROI from an image of an offline store photographed by a physical camera, includes it in a shopping mall page, and streams it to the user to shop at an actual offline store. It can give you the same feeling.

12 is a flowchart illustrating an example of an operation of providing a shopping mall page in the personalized shopping mall system using the virtual camera shown in FIG. 2.

First, when a user terminal is connected in step 910, the image processing apparatus creates a virtual camera to be provided to the user terminal.

In step 920, the image processing device configures a shopping mall page in which the virtual camera generated in step 910 is included in the virtual camera area.

In step 930, the image processing apparatus provides a shopping mall page in which the virtual camera generated in step 920 is included in the virtual camera area to the user terminal.

In step 940, the image processing apparatus grants control rights to the virtual camera included in the shopping mall page provided in step 930 to the corresponding user terminal.

Then, in step 950, the virtual camera area of the shopping mall page in which the image of the region of interest is provided to the user terminal from among the offline store screens configured by sequentially connecting the images of the offline store respectively photographed in real time by a plurality of physical cameras installed in the offline store It is played by the virtual camera included in the.

Then, in step 960, pan/tilt of the ROI image according to the user's manipulation of the ROI image played by the virtual camera included in the virtual camera area of the user terminal to which the control right for the virtual camera is granted. ) Or a Zoom In/Zoom Out operation is performed.

By implementing in this way, the present invention creates a virtual camera for each user with a different ROI from an offline store screen formed by sequentially connecting images of an offline store each photographed by a plurality of physical cameras, and displays the virtual camera for each user on a shopping mall page. By streaming with inclusion, you can give users the feeling of shopping in an actual offline store.

13 is a flowchart illustrating an example of an operation of providing a shopping mall page in the personalized shopping mall system using the virtual camera shown in FIG. 5.

First, when the user terminal is connected in step 1010, the image processing apparatus creates a virtual camera to be provided to the user terminal.

In step 1020, the image processing apparatus configures a shopping mall page in which the virtual camera generated in step 1010 is included in the virtual camera area.

In this case, the virtual camera area of the shopping mall page may include a specific 3D object selected from among 3D objects detected using a stereo time difference of 2D images of an offline store respectively photographed by a plurality of adjacent physical cameras.

In step 1030, the image processing apparatus provides a shopping mall page in which the virtual camera generated in step 1020 is included in the virtual camera area to the user terminal. In this case, a 3D object is rendered and displayed through a virtual camera included in the shopping mall page.

In step 1040, the image processing apparatus grants control rights to the 3D object rendered and displayed through the virtual camera included in the shopping mall page provided in step 1030 to the corresponding user terminal.

Then, in step 1050, the image processing apparatus performs control on the 3D object rendered and displayed through the virtual camera according to a user manipulation from the user terminal that has been granted control rights for the 3D object.

In this case, the control of the 3D object may be rotation, zoom in/zoom out, or pan/tilt control for the 3D object rendered and displayed by the virtual camera. have.

By implementing in this way, the present invention is from an offline store screen including 3D objects detected using a stereo time difference of 2D images of an offline store respectively photographed by a plurality of physical cameras that are arranged adjacent to each other and photograph the image of each store in real time. By creating a plurality of different virtual cameras for each user using an area in which 3D objects are located as a region of interest (ROI), including them in a shopping mall page and streaming them, the user can feel like shopping in an actual offline store.

14 is a flowchart illustrating an example of an operation of providing a shopping mall page in the personalized shopping mall system using the virtual camera shown in FIG. 7.

First, when a user terminal is connected in step 1110, the image processing apparatus creates a virtual camera to be provided to the user terminal.

Then, in step 1120, the image processing device configures a shopping mall page in which the virtual camera generated in step 1110 is included in the virtual camera area.

In step 1130, the image processing apparatus provides a shopping mall page in which the virtual camera generated in step 1120 is included in the virtual camera area to the user terminal.

In step 1140, the image processing device grants control rights to the virtual camera included in the shopping mall page provided in step 1130 to the corresponding user terminal.

Then, the region of interest image among the images of the offline store captured in real time by the physical camera installed in the offline store is played back by the virtual camera included in the virtual camera area of the shopping mall page provided to the user terminal.

At this time, in step 1150, depending on the streaming load of the local server, the local server directly streams the ROI images played by the virtual cameras or indirectly through the streaming server, the ROI images played back by the virtual cameras. Stream.

Then, in step 1160, the pan/tilt of the ROI image according to the user's manipulation of the ROI image played by the virtual camera included in the virtual camera area of the user terminal that has been granted the control right for the virtual camera. ) Or a Zoom In/Zoom Out operation is performed.

By implementing in this way, the present invention generates a virtual camera for each user having a different ROI from an image of an offline store photographed by a physical camera, and includes it in a shopping mall page, and provides the virtual camera of the shopping mall pages for each user. By distributing and transmitting the streaming load of images of a region of interest played by virtual cameras for each user included in the region, it is possible to provide a shopping mall page that can give a user the feeling of shopping in an actual offline store at high speed.

Referring to FIG. 15, an operation of extracting an ROI from a virtual camera of a personalized shopping mall system using a virtual camera according to the present invention will be described. 15 is a flowchart illustrating an example of a procedure for extracting an ROI by the virtual camera shown in FIG. 10.

First, in step 1210, the virtual camera receives and stores virtual camera setting information from the user. For example, the virtual camera setting information may include at least one of an initial region of interest location, region of interest size, resolution, quality, compression format, and virtual camera region pattern.

Next, in step 1220, the virtual camera determines the initial region of interest location set in the virtual camera setting information as the region of interest, and extracts and reproduces a portion corresponding to the region of interest from the store image received from the physical camera as the size of the region of interest. .

Then, in step 1230, if a user manipulates Pan/Tilt or Zoom In/Zoom Out, the image of the region of interest played back by the virtual camera is software within the range of the virtual camera region. You can pan/tilt or zoom in/zoom out.

If the image of the region of interest played by the Pan/Tilt or Zoom In/Zoom Out operation from the user is out of the virtual camera area range (physical camera shooting range) In step 1240, a physical camera driving command instructing the virtual camera to physically pan/tilt or zoom in/zoom out is output to the physical camera.

As described above, the present invention creates a virtual camera for each user with a different ROI from an image of an offline store photographed by a single physical camera, includes it in a shopping mall page, and streams it to the user. It can give you the feeling of shopping at.

In addition, the present invention creates a virtual camera for each user with a different ROI from an offline store screen formed by sequentially connecting images of an offline store each photographed by a plurality of physical cameras, and includes it in a shopping mall page for streaming. By doing so, you can give users the feeling of shopping in an actual offline store.

In addition, the present invention provides 3D objects from an offline store screen including 3D objects detected using a stereo time difference of 2D images of an offline store respectively photographed by a plurality of physical cameras that are arranged adjacent to each other to capture images of each store in real time. By creating a plurality of different virtual cameras for each user with a region of interest as an ROI, including them in a shopping mall page and streaming them, the user can feel as if shopping in an actual offline store.

In addition, the present invention creates a virtual camera for each user with a different ROI from an image of an offline store photographed by a physical camera and includes it in a shopping mall page, but includes it in the virtual camera area of the shopping mall pages for each user. By distributing and transmitting the streaming load of images of the region of interest played by virtual cameras for each user, a shopping mall page that can give users the feeling of shopping in an actual offline store can be provided at a high speed.

Although the present invention has been described based on a preferred embodiment referenced by the accompanying drawings, it is apparent that various modifications are possible without departing from the scope of the present invention within the scope covered by the claims to be described later from this description. .

The present invention can be used industrially in an online shopping mall-related technical field and an applied technical field thereof.

100: camera
200: image processing device
210: image receiver
220: Virtual camera management unit
221: connection management unit
222: virtual camera generation unit
223: shopping mall page generation unit
224: shopping mall page providing unit
225: control right granting unit
226: control right collection unit
230: multiplexing unit
240: image transmission unit
300: local server
400: dedicated server for streaming
500: shopping mall management server
600: shopping mall page
610: virtual camera area
620: interest product area
630: video chat area
640: store information area
700: virtual camera
710: region of interest selection unit
720: region of interest playback unit
730: image processing unit
731: memory
732: Region of interest extraction unit
733: image scaler
734: codec
735: frame rate conversion unit
740: simulated pan tilt zoom processing unit
750: optical pan tilt zoom processing unit
760: setting information storage unit
770: virtual camera setting unit

Claims (20)

A plurality of physical fixed cameras for real-time shooting of an offline store image;
An integrated offline store screen is formed by sequentially connecting an image receiving unit for receiving images of an offline store respectively photographed by the plurality of physical fixed cameras and an offline store image received by the image receiving unit, and configured offline An image processing apparatus including a virtual camera management unit for generating a plurality of virtual cameras for each user having different ROIs from the store screen; including,
The virtual camera management unit:
A connection management unit that processes user terminal access and disconnection, and the images of offline stores each photographed by the plurality of physical fixed cameras are sequentially connected to form an integrated offline store screen, and a region of interest from the configured offline store screen A virtual camera generation unit that generates a plurality of user-specific virtual cameras having different (ROIs), a shopping mall page generation unit that configures a shopping mall page in which the virtual cameras generated by the virtual camera generation unit are included in the virtual camera area, and the A shopping mall page providing unit in which a virtual camera generated by the shopping mall page generating unit provides a shopping mall page included in the virtual camera area to a user terminal, and a control right for a virtual camera included in the shopping mall page provided by the shopping mall page providing unit Includes a control right granting unit for granting to the user terminal,
The virtual camera generated by the virtual camera generator outputs the image of the region of interest selected according to the Pan/Tilt or Zoom In/Zoom Out operation of the user terminal among the images of the configured offline store. However, a personalized shopping mall system using a virtual camera that prevents the background object from being obscured by the foreground object by removing the foreground object from the image of the region of interest to include only the background object. delete delete The method of claim 1,
The shopping mall page generated by the shopping mall page generator is:
Personalized shopping mall system using a virtual camera including a product of interest area in which an image of a product of interest uploaded by the manager terminal is registered. The method of claim 1,
The shopping mall page generated by the shopping mall page generator is:
Personalized shopping mall system using a virtual camera including a video chat area that processes video chat between a manager terminal and a user terminal. The method of claim 1,
The shopping mall page generated by the shopping mall page generator is:
Personalized shopping mall system using a virtual camera including a store information area that provides offline store information. The method of claim 1,
The virtual camera generation unit:
Instructs software to pan/tilt or zoom in/zoom out the image of the region of interest within the range of the virtual camera area according to the Pan/Tilt or Zoom In/Zoom Out operation signal. A personalized shopping mall system using a virtual camera that outputs a virtual camera control command to the image processing unit. The method of claim 7,
The virtual camera generation unit:
When the image of the region of interest played by the region of interest reproducing unit is out of the range of the virtual camera region, a virtual camera driving command instructing the physical camera to pan/tilt or zoom in/zoom out is output to the physical camera. Personalized shopping mall system using camera. The method of claim 1,
The virtual camera generation unit:
Personalized shopping mall system using a virtual camera that receives and stores virtual camera setting information from a user. The method of claim 9,
The virtual camera setting information is a personalized shopping mall system using a virtual camera including at least one of an initial region of interest position, a region of interest size, resolution, quality, compression format, and a virtual color region pattern. The method of claim 9,
The image processing device:
A memory for storing a store image photographed by a physical camera;
A region-of-interest extracting unit that extracts an image of the region of interest selected from the store image stored in the memory to the size of the region of interest set in camera setting information;
An image scaler for converting the region of interest image extracted by the region of interest extracting unit into a resolution set in camera setting information;
A codec for compressing the ROI image converted by the image scaler into a compression format set in camera setting information;
Personalized shopping mall system using the included virtual camera. The method of claim 11,
The image processing device:
A frame rate converter for converting a frame rate of an ROI image according to a quality set in camera setting information;
Personalized shopping mall system using a virtual camera further comprising. delete The method of claim 1,
The image processing unit:
A personalized shopping mall system using a virtual camera that recognizes a person from an image of a region of interest and masks the recognized person's face. The method of claim 1,
Regions of interest (ROIs) for each user selected by the region of interest selection unit are:
Personalized shopping mall system using virtual cameras that can overlap each other. The method of claim 1,
The image processing device:
A multiplexing unit for converting ROI images played by a plurality of user-specific virtual cameras into packet image signals originating from different ports and having the same network address, and multiplexing them into a single packet image stream;
An image transmission unit for transmitting a single packet image stream multiplexed by the multiplexing unit to user terminals through a single physical layer;
Personalized shopping mall system using a virtual camera further comprising. The method of claim 1,
The virtual camera management unit:
A control right recovery unit for recovering a control right for a virtual camera granted to a user terminal that has been disconnected by the connection management unit;
Personalized shopping mall system using a virtual camera further comprising. The method of claim 1,
The plurality of physical fixed cameras:
Personalized shopping mall system using virtual cameras installed in a row. The method of claim 1,
The plurality of physical fixed cameras:
Personalized shopping mall system using a virtual camera installed radially. The method of claim 1,
The virtual camera management unit:
Personalized shopping mall system using a virtual camera that connects some of the videos of the offline store to overlap.

Images