KR20230007792A - Clowd server, system comprising clowd server and client device and controlling method thereof - Google Patents

Abstract

Disclosed is a cloud server communicating with a plurality of security cameras. The cloud server, which can improve convenience of users, may include: a communication interface; a memory; and a processor which stores a plurality of images in the memory when the plurality of images are received from the plurality of security cameras through the communication interface, acquires a synthetic image including at least a part of the plurality of images based on synchronization information included in each of the received plurality of images to store the synthetic image in the memory, and transmits information related to the synthetic image to a client device through the communication interface when a request related to the synthetic image is received from the client device.

Description

Cloud server, system and its control method {CLOWD SERVER, SYSTEM COMPRISING CLOWD SERVER AND CLIENT DEVICE AND CONTROLLING METHOD THEREOF}

The present invention relates to a cloud server for processing an image obtained from a security camera, a system, and a control method thereof.

BACKGROUND OF THE INVENTION [0002] Recently, the spread of security cameras that take pictures of indoor spaces is increasing. A user may be provided with a plurality of images obtained from a plurality of security cameras, and to this end, the cloud server may process images acquired from the security cameras and transmit the images to the client device. However, in order for a user to receive an image obtained from a security camera, it is inconvenient for a user to make a plurality of requests corresponding to each camera, and protocols related to providing images obtained from each camera may be different. There was a problem in that high server traffic occurs when the server processes a plurality of requests sent by users.

Accordingly, there has been a continuous demand for a cloud server capable of acquiring a single composite image related to images acquired from a plurality of security cameras and providing the image requested by a user to a client device.

The present disclosure is in accordance with the above-described needs, and is to provide a cloud server, a system, and a control method thereof for obtaining a composite image based on images obtained from a plurality of security cameras and transmitting information related to the composite image to a client device. .

When a plurality of images are received from the plurality of security cameras through a communication interface, a memory, and the communication interface, the cloud server according to an embodiment of the present invention for achieving the above object displays the plurality of received images. Store in the memory, obtain a composite image including at least some of the plurality of images based on synchronization information included in each of the received plurality of images, store it in the memory, and acquire the composite image and the composite image from a client device. and a processor for transmitting information related to the synthesized image to the client device through the communication interface when a related request is received.

Here, the processor identifies at least some images captured at the same time among the plurality of images based on synchronization information included in each of the received plurality of images, and the at least some images are identified based on the output format of the client device. An output format of an image may be converted, and a composite image including at least some images of which the output format is converted may be obtained.

In addition, the processor, when receiving a request for information on the region of interest in the synthesized image from the client device, crops an image corresponding to the region of interest from the synthesized image to include the cropped image Information may be transmitted to the client device.

In addition, when a request related to the composite image is received from the client device, the processor converts an output format of the composite image based on an output format of the client device having received the request, and the output format is converted. Based on the synthesized image, information related to the synthesized image may be transmitted to the client device.

The processor obtains the synthesized image by arranging the at least some images at a location corresponding to the camera identification information based on camera identification information included in the at least some images, and transmits the synthesized image to the client device. When location information corresponding to a partial region of the composite image is received from the device, camera identification information corresponding to the location information is obtained, a composite image is obtained based on the obtained camera identification information, and the obtained composite image is obtained. may be transmitted to the client device.

In addition, the processor may arrange at least one of an image including a dynamic object among the at least some images, an image taken at a predetermined place among the at least some images, or an image having a threshold brightness or higher among the at least some images at a specific location to perform the A composite image can be obtained.

In addition, when a request related to the synthesized image is received from the client device, the processor determines a critical value from the client device among the plurality of security cameras based on location information of the client device and location information of each of the plurality of security cameras. A security camera located within the distance may be identified, and information including an image of an area other than an image area acquired from the identified camera may be transmitted to the client device in the composite image.

Meanwhile, when a plurality of images are received from a plurality of security cameras, a system including a cloud server and a client device according to an embodiment of the present invention stores the received plurality of images, and each of the received plurality of images Acquire and store a synthesized image including at least some of the plurality of images based on synchronization information included in the image, and when a request related to the synthesized image is received from the client device, information related to the synthesized image is transmitted to the client device. Displays the synthesized image included in the cloud server transmitted to the device and information received from the cloud server, and when a command for selecting a region of interest included in the synthesized image is received, the synthesized image is selected based on the received command It may include a client device that displays an image.

Here, the client device transmits information on the region of interest to the cloud server based on the received selection command, and the cloud server transmits information on the region of interest based on the information received from the client device. A synthesized image including the region of interest may be processed, and the processed synthesized image may be transmitted to the client device, and the client device may display the processed synthesized image received from the cloud server.

Also, the client device may process the synthesized image based on the received selection command and display the processed synthesized image.

Meanwhile, a control method of a cloud server according to an embodiment of the present invention includes, when a plurality of images are received from a plurality of security cameras, storing the received plurality of images, synchronizing each of the received plurality of images. Acquiring and storing a composite image including at least some of the plurality of images based on information and transmitting information related to the composite image to the client device when a request related to the composite image is received from a client device steps may be included.

Here, the acquiring and storing the composite image may include identifying at least some images captured at the same time among the plurality of images based on synchronization information included in each of the plurality of received images, and The method may include converting an output format of the at least some images based on an output format, and obtaining a composite image including the at least some images whose output formats are converted.

The transmitting may include cropping an image corresponding to the region of interest from the synthesized image when a request for information on the region of interest among the synthesized images is received from the client device to obtain the cropped image. The included information may be transmitted to the client device.

The transmitting may include, when a request related to the synthesized image is received from the client device, converting an output format of the synthesized image based on an output format of the client device having received the request, and Information related to the synthesized image may be transmitted to the client device based on the converted synthesized image.

In addition, the acquiring and storing the synthesized image may include arranging the at least some images at positions corresponding to the camera identification information based on camera identification information included in the at least some images to obtain the synthesized image; obtaining camera identification information corresponding to the location information when location information corresponding to a partial region of the composite image is received from the client device, and acquiring a composite image based on the obtained camera identification information. can do.

In addition, the acquiring and storing the synthesized image may include at least one of an image including a dynamic object among the at least some images, an image taken at a predetermined place among the at least some images, and an image having a brightness level higher than or equal to a threshold brightness among the at least some images. The synthesized image may be acquired by arranging the at a specific location.

The transmitting may include, when a request related to the synthesized image is received from the client device, the client device among the plurality of security cameras based on location information of the client device and location information of each of the plurality of security cameras. The method may include identifying a security camera located within a threshold distance from and transmitting information including an image of an area other than an image area obtained from the identified camera in the composite image to the client device.

According to various embodiments of the present disclosure, the cloud server may acquire a single composite image related to images obtained from a plurality of security cameras and provide the image requested by the user to the client device, thereby improving user convenience.

1 is a diagram schematically illustrating an operation of a cloud server according to an embodiment of the present disclosure.
2 is a block diagram illustrating a configuration of a cloud server according to an embodiment of the present disclosure.
3 is a diagram schematically illustrating an operation of obtaining a composite image according to an embodiment of the present disclosure.
4 is a diagram for explaining an operation of obtaining a composite image based on synchronization information according to an embodiment of the present disclosure.
5A and 5B are diagrams for explaining an operation of obtaining a composite image according to conversion of an output format of an image according to an embodiment of the present disclosure.
6 is a diagram for explaining an operation of providing a part of a composite image to a client device according to an embodiment of the present disclosure.
7 is a diagram for explaining an operation of obtaining a composite image according to another embodiment of the present disclosure.
8 is a diagram for explaining an operation of obtaining a composite image based on location information of a client device and a plurality of security cameras according to an embodiment of the present disclosure.
9 is a flowchart illustrating a control method according to an embodiment of the present disclosure.
10 is a sequence diagram for explaining an embodiment in which a cloud server and a client device work together to provide a composite image according to an embodiment of the present disclosure.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

The terms used in the embodiments of the present disclosure have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. . In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the disclosure. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

In the present disclosure, expressions such as “has,” “can have,” “includes,” or “can include” indicate the presence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

The expression at least one of A and/or B should be understood to denote either "A" or "B" or "A and B".

Expressions such as "first," "second," "first," or "second," used in the present disclosure may modify various elements regardless of order and/or importance, and may refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that an element may be directly connected to another element, or may be connected through another element (eg, a third element).

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "consist of" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In the present disclosure, a “module” or “unit” performs at least one function or operation, and may be implemented in hardware or software or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" are integrated into at least one module and implemented by at least one processor (not shown), except for "modules" or "units" that need to be implemented with specific hardware. It can be.

In the present disclosure, the term user may refer to a person using a client device.

1 is a diagram schematically illustrating an operation of a cloud server according to an embodiment of the present disclosure.

According to FIG. 1, a cloud server 100 communicates with a plurality of security cameras 11, 12, 13, and 14, and transmits images obtained from the plurality of security cameras 11, 12, 13, and 14 to a client device 201, 202, 203, 204).

The plurality of security cameras 11 , 12 , 13 , and 14 may obtain an image by capturing an area within a preset field of view (FoV). The plurality of security cameras 11 , 12 , 13 , and 14 may include a lens for focusing visible light and other optical signals received after being reflected by an object into an image sensor and an image sensor capable of detecting visible light and other optical signals. Here, the image sensor may include a 2D pixel array divided into a plurality of pixels.

The client devices 201, 202, 203, and 204 include smart phones, tablet PCs, mobile phones, video phones, e-book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, PDAs, and portable multimedia players (PMPs). , MP3 player, camera, wearable device, etc., but is not limited thereto, and any type of electronic device having a display for displaying an image may be used as a client device.

The client device may transmit a request for providing an image acquired from a specific security camera to the cloud server 100, and the cloud server 100 may transmit a request related to an image acquired from a specific security camera based on the request received from the client device. information may be transmitted to the client device on which the request was received.

For example, in order to receive an image obtained from a security camera 13 photographing a kitchen in a TV 201 located in a living room among client devices, the TV 201 acquires an image from the security camera 13 to the cloud server 100. You can send a request related to the provision of the image.

However, since the images obtained by the plurality of security cameras 11, 12, 13, and 14 may be images of different types, the client device provides images obtained from the plurality of security cameras 11, 12, 13, and 14. Receipt may require sending separate requests related to the processing of different types of images.

Here, the cloud server 100 may receive individual requests according to a plurality of different protocols or may receive individual requests according to RESTful API (Representational State Transfer Application Programming Interface). When the cloud server 100 receives individual requests according to a plurality of different protocols or APIs, excessive server traffic or computational burden may occur in the process of receiving the plurality of requests.

In order to reduce server traffic and computational burden, the cloud server 100 according to an embodiment of the present disclosure obtains a single composite image by synthesizing different types of images, and a A request related to an image may be received from the client device and an image obtained from at least one security camera may be provided to the client device.

Hereinafter, various embodiments in which a synthesized image including images obtained from a plurality of security cameras may be acquired and transmitted to a client device will be described in more detail.

2 is a block diagram illustrating a configuration of a cloud server according to an embodiment of the present disclosure.

According to FIG. 2 , the cloud server 100 may include a communication interface 110 , a memory 120 and a processor 130 . However, the configuration shown in FIG. 2 is an exemplary diagram for implementing the embodiments of the present disclosure, and suitable hardware and software configurations that are obvious to those skilled in the art may be additionally included in the cloud server 100.

The communication interface 110 may input and output various types of data. For example, the communication interface 110 is AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Bluetooth (Bluetooth), Zigbee (Zigbee), wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES/EBU (Audio Engineering Society/ European Broadcasting Union), Optical External device (eg, source device), external storage medium (eg, USB memory), external server (eg, web hard) and various types of data through communication methods such as , Coaxial, etc. can transmit and receive.

The memory 120 may store data necessary for various embodiments of the present disclosure. The memory 120 may be implemented in the form of a memory embedded in the cloud server 100 or in the form of a removable memory in the cloud server 100 according to a data storage purpose. For example, data for driving the cloud server 100 is stored in a memory embedded in the cloud server 100, and data for an extended function of the cloud server 100 is detachable from the cloud server 100. It can be stored in available memory. Meanwhile, in the case of memory embedded in the cloud server 100, volatile memory (e.g., DRAM (dynamic RAM), SRAM (static RAM), SDRAM (synchronous dynamic RAM), etc.), non-volatile memory (non-volatile memory) Examples: OTPROM (one time programmable ROM), PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, flash memory (such as NAND flash or NOR flash, etc.) ), a hard drive, or a solid state drive (SSD). In addition, in the case of a memory that is detachable from the cloud server 100, a memory card (for example, a compact flash drive (CF)) ), SD (secure digital), Micro-SD (micro secure digital), Mini-SD (mini secure digital), xD (extreme digital), MMC (multi-media card), etc.), external memory that can be connected to the USB port ( For example, it may be implemented in the form of a USB memory) or the like.

The processor 130 controls overall operations of the cloud server 100 . Specifically, the processor 130 may be connected to each component of the cloud server 100 to control the overall operation of the cloud server 100 . For example, the processor 130 may be connected to the communication interface 110 and the memory 120 to control the operation of the cloud server 100 .

According to an embodiment, the processor 130 includes a digital signal processor (DSP), a microprocessor, a central processing unit (CPU), a micro controller unit (MCU), and a micro processing unit (MPU). unit), NPU (Neural Processing Unit), controller, application processor (application processor (AP)), etc., but described as processor 130 in this specification.

The processor 130 may be implemented as a system on chip (SoC), large scale integration (LSI), or may be implemented as a field programmable gate array (FPGA). Also, the processor 130 may include volatile memory such as SRAM.

When a plurality of images are received from a plurality of security cameras through the communication interface 110, the processor 130 according to an embodiment of the present disclosure may store the received plurality of images in the memory 120.

Also, the processor 130 may acquire a composite image including at least some of the plurality of images based on the synchronization information included in each of the plurality of received images, and store the synthesized image in the memory 120 .

The synchronization information may be time information related to each frame included in a plurality of images. Synchronization information according to an example may be information about a time point at which a specific frame included in a plurality of images is acquired, but is not limited thereto.

When a request related to the composite image is received from the client device, the processor 130 may transmit information related to the composite image to the client device through the communication interface 110 .

The processor 130 identifies at least some images captured at the same time among the plurality of images based on the synchronization information included in each of the plurality of images received, and determines the output format of the at least some images based on the output format of the client device. can be converted Here, identifying at least some images captured at the same time may mean identifying at least one frame acquired at the same time among a plurality of frames included in a plurality of images.

The output format of the client device may include at least one of the type of image to be provided through the client device, the resolution of the image, or the frame rate of the image.

In addition, when a request for a region of interest among the synthesized images is received from the client device, the processor 130 crops an image corresponding to the region of interest from the synthesized image and transmits information including the cropped image to the client device. can

The region of interest in the synthesized image may refer to a region of a security importance level higher than a critical value among regions included in the synthesized image. The region of interest according to an example may be a region selected by a user by controlling the client device or a region determined according to an algorithm stored in the client device.

Also, the ROI may include an area including an image acquired from at least one camera among a plurality of security cameras or a partial area where an object of interest is located among the corresponding areas. The region of interest may have a rectangular shape, but is not limited thereto.

Cropping refers to an operation of acquiring information including an image included in a region of interest included in a synthesized image. Specifically, the processor 130 obtains image information including an image included in the region of interest, and transmits the acquired image information and user interface (UI) information related to the provision of the image information to the client device through the communication interface 110 can control.

When a request related to a synthesized image is received from a client device, the processor 130 according to an example may convert an output format of the synthesized image based on an output format of the client device receiving the request. Specifically, the client device may transmit a request related to at least one of a type of composite image to be provided from the cloud server 100, a resolution of the image, or a frame rate of the image to the cloud server 100.

For example, when the synthesized image obtained by the processor 130 has a resolution of 1080p and a frame rate of 120 fps, when a request for receiving a synthesized image having a resolution of 720p and a frame rate of 60 fps is received from a client device, the processor 130 may convert the output format of the composite image so that the composite image stored in the memory 120 has a resolution of 720p and a frame rate of 60 fps.

Also, the processor 130 may transmit information related to the synthesized image to the client device based on the synthesized image whose output format is converted.

The processor 130 may obtain a composite image by arranging at least some images at positions corresponding to the camera identification information based on camera identification information included in the at least some images, and transmit the obtained composite image to the client device.

In addition, the processor 130 obtains camera identification information corresponding to the location information when location information corresponding to a partial region of the synthesized image is received from the client device, obtains a synthesized image based on the acquired camera identification information, and The composite image can be sent to the client device.

The processor 130 may obtain a synthesized image by combining only images that satisfy certain conditions among a plurality of images acquired from a plurality of security cameras. For example, the processor 130 arranges and synthesizes at least one of an image including a dynamic object among at least some images, an image taken at a predetermined place among at least some images, and an image having a brightness level higher than a threshold among at least some images at a specific location. Images can also be obtained.

In addition, when a request related to a composite image is received from the client device, the processor 130 selects a security camera located within a threshold distance from the client device among a plurality of security cameras based on the location information of the client device and the location information of each of the plurality of security cameras. can be identified. Specifically, map data corresponding to an indoor space where a plurality of security cameras are located may be stored in the memory 120 of the cloud server 100 . The processor 130 identifies the location information of the client device for which the request related to the plurality of security cameras and the composite image has been received based on the map data, and the security camera located within a threshold distance from the client device based on the identified plurality of location information. can identify.

Subsequently, the processor 130 may transmit only information about images included in the composite image to the client device except for an area including an image obtained from a security camera located within a threshold distance from the client device.

3 is a diagram schematically illustrating an operation of obtaining a composite image according to an embodiment of the present disclosure.

The cloud server 100 according to an embodiment of the present disclosure may obtain a single composite image 310 based on a plurality of images 301 , 302 , 303 , and 304 obtained from a plurality of security cameras. Each of the plurality of images 301, 302, 303, and 304 may be images of different spaces captured by a plurality of security cameras located in the indoor space.

In addition, since each of the first to fourth images 301, 302, 303, and 304 may be a dynamic image (Video Stream) including several frames, the first to fourth images 301, 302, 303, and 304 The synthesized image 310 obtained based on may also be a dynamic image.

The cloud server 100 according to an example may obtain a single synthesized image 310 based on the first to fourth images 301, 302, 303, and 304 obtained from the first to fourth cameras, respectively. there is.

The first to fourth images 301, 302, 303, and 304 may be images of different types. For example, each frame constituting the first image 301 and the third image 303 is a jpg type image, and each frame constituting the second image 302 and the fourth image 304 is a png type may be an image of The cloud server 100 converts each frame constituting the first image 301 and the third image 303 into a png type image, and the converted first image 301 and the third image 303, Based on the second image 302 and the fourth image 304, a composite image 310 including a plurality of frames having a png type may be obtained.

The cloud server 100 may acquire the synthesized image 310 by arranging the first to fourth images 301 , 302 , 303 , and 304 in a specific location based on a preset layout. Here, the preset layout may include at least one of a regular arrangement and an irregular arrangement having an N*M shape.

According to the layout corresponding to the standard arrangement, the first to fourth images 301, 302, 303, and 304 are arranged in a 2*2 arrangement, and the first to fourth images 301 included in the composite image 310 , 302, 303, 304) may have the same size. To this end, the cloud server 100 may equally adjust the size of each of the first to fourth images 301, 302, 303, and 304, and obtain a synthesized image 310 based on each of the adjusted images. .

Meanwhile, according to the layout corresponding to the atypical arrangement, an area corresponding to at least one of the first to fourth images 301, 302, 303, and 304 may be larger than areas corresponding to the other images. In this case, the composite image 310 may include the first to fourth images 301 , 302 , 303 , and 304 disposed in regions of different sizes and blanks.

The cloud server 100 according to an example may acquire the synthesized image 310 by arranging the first to fourth images at locations corresponding to the camera identification information for each image based on the camera identification information. Specifically, the cloud server 100 may place the image 301 obtained from the first camera on the upper left side of the composite image 310 having a regular arrangement of 2*2 based on the camera identification information of the first camera. there is.

Similarly, the cloud server 100 places the image 302 obtained from the second camera on the upper right of the composite image 310 based on the camera identification information of the second to fourth cameras, and the image obtained from the third camera. The synthesized image 310 may be obtained by arranging 303 at the bottom left of the synthesized image 310 and the image 304 obtained from the fourth camera at the lower right of the synthesized image 310 .

In this case, the cloud server 100 may obtain location information corresponding to each image 301 , 302 , 303 , and 304 included in the obtained composite image 310 . If some or all of the images constituting the synthesized image 310 are changed, the cloud server 100 newly maps camera identification information and location information corresponding to one region of the synthesized image 310 to memory ( 120) can be stored. Specifically, when the image disposed in the upper left corner of the composite image 310 is replaced with an image obtained from the fifth security camera, the cloud server 100 provides location information corresponding to the upper left region of the composite image 310 and Camera identification information for the fifth security camera may be mapped and stored in the memory 120 .

4 is a diagram for explaining an operation of obtaining a composite image based on synchronization information according to an embodiment of the present disclosure.

The cloud server 100 according to an embodiment of the present disclosure may identify at least some images captured at the same time among a plurality of images based on synchronization information included in each of the plurality of images.

Synchronization information may include information about times at which a plurality of frames included in a plurality of images are acquired, but is not limited thereto.

For example, an image obtained from the first camera may include a plurality of frames 411, 412, 413, and 414. Likewise, each of the images obtained from the second to fourth cameras may include a plurality of frames 421 to 424 , 431 to 434 , and 441 to 444 .

The cloud server 100 may obtain the first frame 451 of the composite image based on the frames 411 , 421 , 431 , and 441 captured at the same time.

When the frame rate of the images acquired from the first camera and the third camera is 60 fps, and the frame rate of the images acquired from the second camera and the fourth camera is 120 fps, the second frame 452 of the composite image is taken at the same time It may include frames 412, 423, 432, and 443 obtained from the first camera to the fourth camera. In this case, the frame rate of the composite image may be 60 fps, which is the same as the frame rate of an image having a lower frame rate among the plurality of images.

5A and 5B are diagrams for explaining an operation of obtaining a composite image according to conversion of an output format of an image according to an embodiment of the present disclosure.

Referring to FIG. 5A , there may be cases where a user wants to reduce the output resolution of a synthesized image, such as when communication between a client device and the cloud server 100 is not smooth. In this case, the client device may transmit a request for setting the resolution of the synthesized image to 360p to the cloud server 100 . When a request related to a synthesized image is received from a client device, the cloud server 100 may change the output resolution of the synthesized image based on the output format of the client device receiving the request.

Specifically, the cloud server 100, which has previously stored synthesized images obtained based on the plurality of images 511, 512, 513, and 514 acquired from the first to fourth cameras, receives a request related to the synthesized image. If received, the resolution of the image included in the plurality of areas included in the composite image can be adjusted to 360p, and information related to the composite image 520 with the adjusted resolution can be transmitted to the client device.

According to FIG. 5B, a user's client device desiring to receive a composite image having a higher frame rate transmits a request for changing the frame rate of at least some of a plurality of images acquired from a plurality of security cameras to the cloud server 100. can be sent to

For example, when a user wants to receive a composite image having a resolution of 120 fps, the client device may transmit a request for setting the frame rate of the composite image to 120 fps to the cloud server 100 . When a request related to the composite image is received from the client device, the cloud server 100 may change the frame rate of the composite image based on the output format of the client device receiving the request.

Specifically, since the images obtained from the first camera and the third camera have a frame rate of 60 fps, and the images acquired from the second and fourth cameras have a frame rate of 120 fps, the cloud server 100 has a frame rate of 120 fps. And an interpolation frame may be inserted in relation to the image obtained from the third camera.

The cloud server 100 according to an example includes a first frame 541 included in a composite image based on a plurality of image frames 531, 532, 533, and 534 captured at a specific point in time from the first to fourth cameras. can be obtained.

In addition, the cloud server 100 provides an interpolation frame 531-1 identical to the frame 531 obtained at a specific point in time from the first camera and an interpolation frame 533 identical to the frame 533 obtained at a specific point in time from the third camera. -1) can be obtained.

In addition, the cloud server 100 based on the frames 532-1 and 534-1 obtained from the second camera and the fourth camera and interpolated frames 531-1 and 533-1 obtained at a point in time after a specific point in time. By doing so, the second frame 542 included in the composite image can be obtained. Here, the two frames 534 and 534-1 acquired by the fourth camera at different viewpoints may be the same when a dynamic object does not exist in a space photographed by the fourth camera.

Therefore, the composite image including the first frame 541 and the second frame 542 may have a frame rate of 120 fps, which is different from the description in FIG. 4 related to the operation of obtaining a composite image without inserting interpolation frames. The frame rate of an image having a high frame rate among images of , and the frame rate of a composite image may be set to be the same.

A request to set the resolution of an image to 360p may be transmitted to the cloud server 100 . When a request related to a composite image is received from a client device, the cloud server 100 may convert an output format of the composite image based on an output format of the client device receiving the request.

Specifically, the cloud server 100, which has previously stored synthesized images obtained based on the plurality of images 511, 512, 513, and 514 acquired from the first to fourth cameras, receives a request related to the synthesized image. If received, the resolution of the image included in the plurality of areas included in the composite image can be adjusted to 360p, and information related to the composite image 520 with the adjusted resolution can be transmitted to the client device.

Meanwhile, the cloud server 100 according to an embodiment different from that described in FIGS. 5A and 5B obtains a plurality of composite images having different resolutions corresponding to a plurality of levels and different frame rates corresponding to a plurality of levels, and stores the memory ( 120), and a composite image having a resolution and frame rate corresponding to the request received from the client device may be transmitted to the client device.

6 is a diagram for explaining an operation of providing a part of a composite image to a client device according to an embodiment of the present disclosure.

According to FIG. 6 , the cloud server 100 may receive a request 610 for information on a region of interest in a synthesized image from the client device 200 . Here, the information on the region of interest may mean information related to an image included in the region of interest. The region of interest according to an example may mean a region including an image obtained from at least one camera among synthesized images having a 2*2 standard arrangement layout, or a partial region where an object of interest is located among the corresponding regions.

If it is assumed that the region of interest has a rectangular shape, the region of interest may be expressed as information about the coordinates of the upper left corner of the rectangle and the horizontal and vertical sizes of the rectangle. For example, when the horizontal size of the composite image is 640 pixels and the vertical size is 480 pixels, the region of interest according to an example is (0, 0, 320, 240), (320, 0, 320, 240), (0, 240, 320, 240) or (320, 240, 320, 240). However, the above-described method of representing the region of interest is merely an embodiment of the present disclosure, and it goes without saying that the cloud server 100 may identify the region of interest in a method different from the above-described method.

According to an example, the client device 200 may identify a specific region of interest and transmit a request 610 for information on the identified region of interest to the cloud server 100 . In this case, the cloud server 100 may transmit information 620 on the region of interest identified by the client device 200 to the client device 200 based on the request 610 .

In addition, the client device 200 may transmit a request 610 including at least one of camera identification information corresponding to the identified region of interest or location information related to a synthesized image corresponding to the camera identification information to the cloud server 100. there is. In this case, the cloud server 100 identifies a region of interest included in the synthesized image based on at least one of camera identification information included in the request 610 and location information related to the synthesized image, and information on the identified region of interest ( 620) to the client device 200.

According to an embodiment of the present disclosure, when location information 610 corresponding to a partial region of a synthesized image is received from the client device 200, the cloud server 100 identifies a camera corresponding to the received location information 610. information can be obtained. For example, when location information 610 corresponding to the upper left area of the synthesized image is received from the client device 200, the cloud server 100 may obtain camera identification information corresponding to the location information of the corresponding area. The cloud server 100 may crop an image obtained from a security camera corresponding to the obtained camera identification information and transmit information 620 including the cropped image to the client device 200 .

Therefore, even when some or all of the images constituting the composite image are changed, the cloud server 100 provides camera identification information corresponding to a region of the composite image based on the location information 610 received from the client device 200. Based on this, an image that the user wants to be provided may be transmitted to the client device 200 .

According to another example, the client device 200 may request the cloud server 100 to identify an area of interest and transmit information on the area of interest. In this case, the cloud server 100 may identify a region of interest included in the synthesized image based on the received request 610 and transmit information 620 related to the image included in the region of interest to the client device 200 . .

The client device 200 may provide the user with a UI 630 displaying an image included in the region of interest based on the information 620 about the region of interest received from the cloud server 100 .

7 is a diagram for explaining an operation of obtaining a composite image according to another embodiment of the present disclosure.

According to FIG. 7 , the cloud server 100 may identify at least one image among a plurality of images 701 , 702 , 703 , 704 , 705 , 706 , 707 , 708 , and 709 included in a composite image 700 . there is. In addition, the cloud server 100 may obtain a new composite image 710 by disposing the identified image in a specific location.

Specifically, the cloud server 100 includes a plurality of images 701 , 702 , 703 , 704 , 705 , 706 , 707 , 708 , and 709 included in the composite image 700 , an image including a dynamic object, and a preset location. A new synthesized image 710 may be obtained by arranging at least one of an image taken at or an image having a threshold brightness or higher at a specific location.

For example, the cloud server 100 creates a new synthesized image 710 based on the remaining images except for the images 704 and 709 with less than a threshold brightness among areas included in the synthesized image 700 having a regular arrangement of 3*3. ) can be obtained. This is because the images 704 and 709 below the threshold brightness may be images of spaces in which the user is not interested.

However, the user may give a priority to the corresponding place even if it is a place corresponding to an image with less than the threshold brightness, and the client device sends a request related to a synthesized image including priority information about the place to the cloud server 100. can transmit In this case, the cloud server 100 may acquire a new composite image 710 including images corresponding to places to which priorities are assigned.

In addition, the cloud server 100 may acquire a new composite image 710 by arranging images 702 , 706 , 707 , and 708 including dynamic objects among the remaining images except for images having a brightness below a threshold in a specific location. . This is because the images 702 , 706 , 707 , and 708 including dynamic objects may be images of spaces of interest to the user.

Through this, the cloud server 100 acquires a new composite image 710 including at least one image of a space of interest to the user and provides information related to the new composite image 710 to the client device. can

8 is a diagram for explaining an operation of obtaining a composite image based on location information of a client device and a plurality of security cameras according to an embodiment of the present disclosure. In FIG. 8 , the same identification code is attached to a plurality of security cameras and images obtained from the plurality of security cameras.

According to FIG. 8 , the cloud server 100 receives location information of the client device 200 and a plurality of security cameras 801 , 802 , 803 , 804 , and 805 when a request related to a synthesized image is received from the client device 200 . A security camera 801 or 805 located within a critical distance from the client device 200 among the plurality of security cameras 801 , 802 , 803 , 804 , and 805 may be identified based on the location information of .

Since the security cameras 801 and 805 located within a critical distance from the location 800 of the client device 200 are likely to be cameras that capture the location 800 of the client device 200, the corresponding cameras 801 and 805 The image obtained from may be an image that the user does not want to be provided. The cloud server 100 determines the client device 200 based on line of sight information and distance information between the location 800 of the client device 200 and the locations 801 and 805 of security cameras located within a critical distance from the client device 200 . A camera 801 that directly captures the location 800 of ) may be identified. To this end, the cloud server 100 may utilize map data stored in the memory 120 .

In addition, the cloud server 100 obtains a composite image 810 based on images obtained from the security cameras 802, 803, 804, and 805 except for the identified security camera 801, and the obtained composite image ( Information related to 810 may be transmitted to the client device 200 .

Through this, the cloud server 100 can be provided with a synthesized image 810 including images obtained from security cameras 802, 803, 804, and 805 that capture a space other than the user's location 800. .

9 is a flowchart for explaining a control method according to an embodiment of the present disclosure.

According to the control method of the cloud server according to an embodiment of the present disclosure, when a plurality of images are received from a plurality of security cameras, the received plurality of images are stored (S910).

Then, based on the synchronization information included in each of the plurality of images, a composite image including at least some of the plurality of images is acquired and stored (S920).

Finally, when a request related to the composite image is received from the client device, information related to the composite image is transmitted to the client device (S930).

Here, the step of acquiring and storing the composite image (S920) includes identifying at least some images captured at the same time among a plurality of images based on synchronization information included in each of the plurality of received images, and output format of the client device. It may include converting output formats of at least some images based on and obtaining a composite image including at least some images whose output formats are converted.

In addition, in the step of transmitting information related to the synthesized image to the client device (S930), when a request for information on the region of interest in the synthesized image is received from the client device, an image corresponding to the region of interest is cropped and enlarged from the synthesized image. Information including the wrapped image may be transmitted to the client device.

In addition, in the step of transmitting information related to the synthesized image to the client device (S930), when a request related to the synthesized image is received from the client device, the output format of the synthesized image is converted based on the output format of the client device receiving the request, Based on the synthesized image whose output format is converted, information related to the synthesized image may be transmitted to the client device.

In addition, acquiring and storing the synthesized image (S920) includes acquiring a synthesized image by arranging at least some images at locations corresponding to the camera identification information based on camera identification information included in the at least some images, and obtaining a synthesized image from a client device. The method may include acquiring camera identification information corresponding to the location information when location information on some regions of the synthesized image is received, and obtaining a composite image based on the acquired camera identification information.

In addition, in the step of obtaining and storing the synthesized image (S920), at least one of an image including a dynamic object among at least some images, an image taken at a predetermined place among at least some images, and an image having a brightness level higher than or equal to a threshold brightness among at least some images is specified. A composite image can be obtained by placing it in a location.

In addition, in the step of transmitting information related to the synthesized image to the client device (S930), when a request related to the synthesized image is received from the client device, a plurality of security cameras are provided based on the location information of the client device and the location information of each of the plurality of security cameras. The method may include identifying a security camera located within a critical distance from the client device and transmitting information including images of areas other than an image area acquired from the camera identified in the composite image to the client device.

10 is a sequence diagram for explaining an embodiment in which a cloud server and a client device work together to provide a composite image according to an embodiment of the present disclosure.

According to FIG. 10 , a system according to an embodiment of the present disclosure may include a cloud server 100 and a client device 200 .

When a plurality of images are received from a plurality of security cameras, the cloud server 100 may store the received plurality of images (S1010). Subsequently, the cloud server 100 may obtain and store a synthesized image including at least some of the plurality of images based on synchronization information included in each of the plurality of received images (S1020).

In addition, when a request related to a composite image is received from the client device 200 (S1030), the cloud server 100 may transmit information related to the composite image to the client device 200 (S1040).

The client device 200 according to an example may display a composite image included in information received from the cloud server 100 . In addition, the client device 200 may transmit information on the region of interest to the cloud server 100 based on the selected scene for the region of interest included in the synthesized image (S1060). Here, the command for selecting a region of interest may refer to a command for requesting provision of an image corresponding to at least one region among regions included in the synthesized image. Also, the information on the region of interest may include at least one of information about an output format of a synthesized image, camera identification information of a security camera corresponding to the region of interest, or location information corresponding to the region of interest.

Next, the cloud server 100 may process the synthesized image including the region of interest based on the information on the region of interest (S1070). In addition, the cloud server 100 may transmit the processed synthesized image to the client device 200 (S1080).

When the processed synthesized image is received from the cloud server 100, the client device 200 may display the synthesized image included in the received information (S1090).

Meanwhile, the methods according to various embodiments of the present disclosure described above may be implemented in the form of applications that can be installed in the existing cloud server 100.

In addition, the methods according to various embodiments of the present disclosure described above may be implemented only by upgrading software or hardware of the existing cloud server 100 .

Meanwhile, various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor 130 itself. According to software implementation, embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

Meanwhile, computer instructions for performing the processing operation of the cloud server 100 according to various embodiments of the present disclosure described above may be stored in a non-transitory computer-readable medium. there is. When the computer instructions stored in the non-transitory computer readable medium are executed by the processor of the specific device, the processing operation in the cloud server 100 according to various embodiments described above is performed by the specific device.

A non-transitory computer readable medium is a medium that stores data semi-permanently and is readable by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specific examples of the non-transitory computer readable media may include CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Although the preferred embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the specific embodiments described above, and is common in the technical field belonging to the present disclosure without departing from the gist of the present disclosure claimed in the claims. Of course, various modifications are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or prospect of the present disclosure.

100: cloud server 110: communication interface
120: memory 130: processor
200: client device

Claims (17)

A cloud server communicating with a plurality of security cameras,
communication interface;
Memory; and
When a plurality of images are received from the plurality of security cameras through the communication interface, storing the received plurality of images in the memory;
Obtaining a composite image including at least some of the plurality of images based on synchronization information included in each of the received plurality of images and storing the synthesized image in the memory;
and a processor configured to transmit information related to the synthesized image to the client device through the communication interface when a request related to the synthesized image is received from a client device. According to claim 1,
the processor,
Identifying at least some images captured at the same time among the plurality of images based on synchronization information included in each of the received plurality of images;
Converting an output format of the at least some images based on an output format of the client device, and obtaining a composite image including at least some images of which the output format is converted. According to claim 1,
the processor,
When a request for information on a region of interest among the synthesized images is received from the client device, an image corresponding to the region of interest is cropped from the synthesized image, and information including the cropped image is transmitted to the client device. transmission, cloud server. According to claim 1,
the processor,
When a request related to the synthesized image is received from the client device, the output format of the synthesized image is converted based on the output format of the client device having received the request, and the output format is converted into the synthesized image based on the converted image. A cloud server that transmits information related to a composite image to the client device. According to claim 1,
the processor,
Based on camera identification information included in the at least some images, arranging the at least some images at a location corresponding to the camera identification information to obtain the composite image and transmit it to the client device;
When location information corresponding to a partial region of the synthesized image is received from the client device, camera identification information corresponding to the location information is obtained, and a synthesized image is obtained based on the obtained camera identification information. A cloud server that transmits the composite image to the client device. According to claim 1,
the processor,
Obtaining the synthesized image by arranging at least one of an image including a dynamic object among the at least some images, an image taken at a predetermined place among the at least some images, or an image having a threshold brightness or higher among the at least some images at a specific location, cloud server. According to claim 1,
the processor,
When a request related to the synthesized image is received from the client device, a security camera located within a threshold distance from the client device among the plurality of security cameras is selected based on the location information of the client device and the location information of each of the plurality of security cameras. identify,
The cloud server that transmits information including an image of an area other than an image area obtained from the identified camera in the synthesized image to the client device. In a system including a cloud server and a client device,
When a plurality of images are received from a plurality of security cameras, the received plurality of images are stored, and a composite image including at least some of the plurality of images based on synchronization information included in each of the received plurality of images a cloud server that acquires and stores , and transmits information related to the synthesized image to the client device when a request related to the synthesized image is received from the client device; and
a client device that displays the composite image included in the information received from the cloud server, and displays the composite image based on the received selection command when a selection command for a region of interest included in the composite image is received; Including, system. According to claim 8,
The client device,
Transmitting information on the region of interest to the cloud server based on the received selection command;
The cloud server,
processing a composite image including the region of interest based on information about the region of interest received from the client device, and transmitting the processed composite image to the client device;
The client device,
and displaying the processed composite image received from the cloud server. According to claim 8,
The client device,
Process the composite image based on the received selection command, and display the processed composite image. In the control method of the cloud server,
Storing the received plurality of images when a plurality of images are received from a plurality of security cameras;
obtaining and storing a synthesized image including at least some of the plurality of images based on synchronization information included in each of the plurality of received images; and
and transmitting information related to the synthesized image to the client device when a request related to the synthesized image is received from a client device. According to claim 11,
The step of acquiring and storing the composite image,
identifying at least some images captured at the same time among the plurality of images based on synchronization information included in each of the received plurality of images; and
Converting the output format of the at least some images based on the output format of the client device, and obtaining a composite image including the at least some images of which the output format is converted. According to claim 11,
The sending step is
When a request for information on a region of interest among the synthesized images is received from the client device, an image corresponding to the region of interest is cropped from the synthesized image, and information including the cropped image is transmitted to the client device. transmission, control method. According to claim 11,
The sending step is
When a request related to the synthesized image is received from the client device, the output format of the synthesized image is converted based on the output format of the client device having received the request, and the output format is converted into the synthesized image based on the converted image. A control method for transmitting information related to a composite image to the client device. According to claim 11,
The step of acquiring and storing the composite image,
obtaining the synthesized image by arranging the at least some images at positions corresponding to the camera identification information based on camera identification information included in the at least some images;
acquiring camera identification information corresponding to the location information when location information corresponding to a partial region of the synthesized image is received from the client device; and
Acquiring a synthesized image based on the obtained camera identification information; including, the control method. According to claim 11,
The step of acquiring and storing the composite image,
Obtaining the synthesized image by arranging at least one of an image including a dynamic object among the at least some images, an image taken at a predetermined place among the at least some images, or an image having a threshold brightness or higher among the at least some images at a specific location, control method. According to claim 11,
The sending step is
When a request related to the synthesized image is received from the client device, a security camera located within a threshold distance from the client device among the plurality of security cameras is selected based on the location information of the client device and the location information of each of the plurality of security cameras. identifying; and
and transmitting information including an image of an area other than an image area obtained from the identified camera in the synthesized image to the client device.

Images