CN109961472B

CN109961472B - Method, system, storage medium and electronic device for generating 3D thermodynamic diagram

Info

Publication number: CN109961472B
Application number: CN201711420490.0A
Authority: CN
Inventors: 谭傅伦
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2017-12-25
Filing date: 2017-12-25
Publication date: 2022-03-04
Anticipated expiration: 2037-12-25
Also published as: CN109961472A

Abstract

The embodiment of the invention provides a method, a system, a storage medium and electronic equipment for generating a 3D thermodynamic diagram, wherein the method comprises the following steps: the method comprises the steps of periodically obtaining RGB images and corresponding depth images from RGB videos and depth videos recorded by a depth camera, obtaining spatial coordinates of key points in a target scene according to the RGB images, the depth images and the spatial coordinates of the depth camera relative to the target scene, and generating a 3D thermodynamic diagram of the target scene based on the spatial coordinates of the key points in each period. Compared with thermodynamic diagrams in the prior art, the thermodynamic information of the position in any space in the target scene can be displayed, and therefore comprehensiveness of displaying the thermodynamic information through the thermodynamic diagrams is improved.

Description

Method, system, storage medium and electronic device for generating 3D thermodynamic diagram

Technical Field

The invention relates to the technical field of logistics and computers, in particular to a method and a system for generating a 3D thermodynamic diagram, a storage medium and electronic equipment.

Background

An enthusiasm is an illustration of a page area that is enthusiastic to visitors and the geographic area in which the visitors are located in a special highlighted form. The thermodynamic diagram may show what happens to the non-clickable areas.

Currently, store enthusiasm has been widely applied to store passenger flow analysis in the retail industry. By observing the color change in the thermodynamic diagram, the popularity region is judged according to the color depth, and the darker the color of the general region block, the more concentrated the population is, and the lighter the color is, the more dispersed the population is.

In the prior art, people flow is generally identified through monitoring videos, and the staying position of a customer in a shop is determined; in a store, the greater the number of customers who stay in the store, the longer the time, the darker the color in the thermodynamic diagram for that location. Fig. 1 is a prior art quotient thermodynamic diagram shown in an embodiment of the present invention.

In the process of implementing the invention, the inventor finds that at least the following technical problems exist in the prior art:

the display mode of the thermodynamic diagram in the prior art is 2d display, the thermodynamic information of the position in the space is not fully displayed, and the thermodynamic information of each layer of shelf in the whole quotient exceeds cannot be displayed like the thermodynamic diagram in fig. 1.

Therefore, a new method, system, storage medium, and electronic device for generating a 3D thermodynamic diagram are needed.

The above information disclosed in this background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not constitute prior art that is already known to a person of ordinary skill in the art.

Disclosure of Invention

In view of the above, the present invention provides a method, a system, a storage medium, and an electronic device for generating a 3D thermodynamic diagram, which can generate a more comprehensive display of position information in a space.

Additional features and advantages of the invention will be set forth in the detailed description which follows, or may be learned by practice of the invention.

According to a first aspect of the present invention, there is provided a method of 3D thermodynamic diagram generation, wherein at least one depth camera is included in a target scene, the method comprising:

periodically acquiring RGB images and corresponding depth images from the RGB video and the depth video recorded by the depth camera;

acquiring the spatial coordinates of key points in the target scene according to the RGB image, the depth image and the spatial coordinates of the depth camera relative to the target scene;

generating a 3D thermodynamic diagram of the target scene based on the spatial coordinates of the keypoints for each cycle.

According to some embodiments, obtaining spatial coordinates of a keypoint in the target scene from the RGB image, the depth image, and spatial coordinates of the depth camera with respect to the target scene comprises:

extracting RGB image coordinates of key points from the RGB image;

acquiring the depth coordinate of the key point based on the RGB image coordinate;

and acquiring the spatial coordinates of the key points in the target scene based on the RGB image coordinates and the depth coordinates of the key points and the spatial coordinates of the depth camera relative to the target scene.

According to some embodiments, extracting RGB image coordinates of keypoints from the RGB image comprises:

marking pedestrian frames from the RGB images based on a pedestrian detection algorithm;

extracting RGB image coordinates of five fingertips and wrists of the pedestrian according to the pedestrian frame and the human body key point detection algorithm;

and calculating the center or the gravity center of the palm according to the RGB image coordinates of the five fingertips and the wrist, and taking the RGB image coordinates of the center or the gravity center as the RGB image coordinates of the key points. .

According to some embodiments, when including a plurality of depth cameras in a target scene, based on spatial coordinates of the depth cameras relative to the target scene, comprising: acquiring an identifier of the depth camera; and searching the space coordinate of the depth camera relative to the target scene according to the identification.

According to some embodiments, before generating the 3D thermodynamic diagram of the target scene based on the spatial coordinates of the keypoints for each cycle, the method further comprises: judging whether the key points correspond to markers in the target scene or not according to the space coordinates of the key points in each period; and if so, generating a 3D thermodynamic diagram of the target scene based on the spatial coordinates of the key points.

According to some embodiments, generating a 3D thermodynamic diagram of the target scene based on the spatial coordinates of the keypoints for each cycle comprises: 3D reconstructing the target scene to acquire a 3D image of the target scene; and thermally marking the space coordinates of the key points of each period in the 3D image to generate a 3D thermodynamic diagram of the target scene.

According to a second aspect of the present invention, there is provided a system for 3D thermodynamic diagram generation, wherein at least one depth camera is included in a target scene, the system comprising:

the first acquisition module is used for periodically acquiring RGB images and corresponding depth images from the RGB video and the depth video recorded by the depth camera;

the second acquisition module is used for acquiring the spatial coordinates of the key points in the target scene according to the RGB image, the depth image and the spatial coordinates of the depth camera relative to the target scene;

a generating module, configured to generate a 3D thermodynamic diagram of the target scene based on the spatial coordinates of the key points of each period.

According to some embodiments, the second obtaining module comprises:

the extraction unit is used for extracting RGB image coordinates of key points from the RGB image;

the depth coordinate acquisition unit is used for acquiring the depth coordinate of the key point based on the RGB image coordinate;

and the space coordinate acquisition unit is used for acquiring the space coordinates of the key points in the target scene based on the RGB image coordinates and the depth coordinates of the key points and the space coordinates of the depth camera relative to the target scene.

According to some embodiments, the extracting unit is configured to mark a pedestrian frame from the RGB image based on a pedestrian detection algorithm, extract RGB image coordinates of five fingertips and wrists of a pedestrian according to the pedestrian frame and a human body key point detection algorithm, and calculate a center or a gravity center of the palm from the RGB image coordinates of the five fingertips and wrists, the RGB image coordinates of the center or the gravity center being RGB image coordinates of a key point.

According to some embodiments, the system further comprises: the judging module is used for judging whether the key points correspond to the markers in the target scene or not according to the space coordinates of the key points in each period; and the generating module is used for generating the 3D thermodynamic diagram of the target scene based on the space coordinates of the key points when the judgment result of the judging module is yes.

According to a third aspect of the invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, performs the method steps as set forth in the first aspect.

According to a fourth aspect of the present invention, there is provided an electronic apparatus, comprising: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out the method steps as described in the first aspect.

In the above embodiment of the present invention, a 3D thermodynamic diagram of a target scene can be generated by periodically obtaining RGB images and corresponding depth images from an RGB video and a depth video recorded by a depth camera, obtaining spatial coordinates of key points in the target scene according to the RGB images, the depth images, and spatial coordinates of the depth camera with respect to the target scene, and based on the spatial coordinates of the key points of each period. Compared with thermodynamic diagrams in the prior art, the thermodynamic information of the position in any space in the target scene can be displayed, and therefore comprehensiveness of displaying the thermodynamic information through the thermodynamic diagrams is improved.

Drawings

The above and other objects, features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

FIG. 1 is a prior art thermodynamic diagram of a quotient and excess shown in an embodiment of the present invention;

FIG. 2 is a flow diagram illustrating a method of 3D thermodynamic diagram generation in accordance with an exemplary embodiment;

FIG. 3 is a schematic diagram of the placement of a depth camera in a target scene shown in an embodiment of the present invention;

FIG. 4 is a flow diagram illustrating a method of obtaining spatial coordinates of keypoints in the target scene, according to an exemplary embodiment;

FIG. 5a is a schematic diagram of a spatial coordinate system of a target scene according to an embodiment of the present invention;

fig. 5b is a schematic diagram of a camera coordinate system with a depth camera as an origin according to an embodiment of the present invention;

FIG. 5c is a schematic diagram of an RGB image coordinate system according to an embodiment of the present invention;

FIG. 5d is a schematic diagram of a depth coordinate system according to an embodiment of the present invention;

FIG. 6 is a block diagram illustrating a system for 3D thermodynamic diagram generation in accordance with embodiments of the present invention;

fig. 7 is a schematic structural diagram of an electronic device according to an exemplary embodiment.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus, a repetitive description thereof will be omitted.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention may be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations or operations have not been shown or described in detail to avoid obscuring aspects of the invention.

The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

It should be noted that, although the terms, such as first/second, etc., are used to describe the processing module, the time period, and the sales data in the embodiment of the present invention, these terms are not intended to limit the processing module, the time period, and the sales data, and are only used to distinguish the processing module, the time period, and the sales data from each other.

Fig. 2 is a flow chart illustrating a method of 3D thermodynamic diagram generation in accordance with an exemplary embodiment. It should be noted that at least one depth camera is arranged in the target scene.

As shown in fig. 2, in S210, RGB images and corresponding depth images are periodically acquired from the RGB video and the depth video recorded by the depth camera.

According to an example embodiment, a target scene may be monitored using at least one depth camera, the placement of which is required to enable clear capture of each marker.

Fig. 3 is a schematic diagram of the arrangement of depth cameras in a target scene, which may be a convenience store, a supermarket or a store, and markers may be shelves, and by adjusting an angle α between a horizontal direction of each depth camera and a lens orientation, the two depth cameras can clearly shoot the shelves in the target scene.

According to an example embodiment, after the depth camera is arranged, the depth camera records a target scene, and simultaneously acquires an RGB video and a depth video, where it is to be noted that the RGB video corresponds to the depth video. Further, an RGB image is periodically acquired from the RGB video and the depth video, and a depth image corresponding to the RGB image is acquired. For example, the RGB video and the depth video may take images at the same time, and the obtained RGB image may correspond to the depth image.

In S220, obtaining spatial coordinates of a key point in the target scene according to the RGB image, the depth image, and spatial coordinates of the depth camera with respect to the target scene.

It should be noted that, in the embodiment of the present invention, in order to detect the thermal information of the spatial position in the target scene, a human hand is used as a key point to detect, it can be understood that in a business overload scene, a guest often has an interest in a commodity on a certain layer of a shelf and picks up the commodity to view the commodity. In the scheme of the invention, the human hand is used as a key point for detection, and the specific layer of the shelf for which the user stays for a long time can be detected, so that the 3D thermodynamic diagram for each layer is generated.

In S230, a 3D thermodynamic diagram of the target scene is generated based on the spatial coordinates of the keypoints for each cycle.

According to an example embodiment, before generating the 3D thermodynamic diagram of the target scene, whether a key point corresponds to a marker in the target scene may be determined according to the spatial coordinates of the key point of each period, and if so, the 3D thermodynamic diagram of the target scene is generated based on the spatial coordinates of the key point. If not, the key point is discarded.

It should be noted that in a business super-equal scene, the marker may be a shelf, the key point detected in the periodically acquired RGB image and the corresponding depth image may be that the user is walking, and at this time, the detected key point may be in a normal swinging process, and does not correspond to a certain shelf, and at this time, the spatial coordinate of the detected hand in the target scene has no meaning for generating the 3D thermodynamic diagram, so the key point may be discarded. And when the detected key point hand corresponds to a certain shelf and a certain commodity on the shelf is held in the hand, a 3D thermodynamic diagram needs to be generated according to the space coordinate pair of the detected hand in the target scene.

According to an example embodiment, when generating a 3D thermodynamic diagram of a target scene, the target scene may first be 3D reconstructed to obtain a 3D image of the target scene. Further, the spatial coordinates of the key points of each period are marked in the 3D image in a thermal mode, and a 3D thermodynamic diagram of the target scene is generated.

For example, each layer of each shelf is displayed in a 3D image, and further, after a key point of a target scene is acquired in each period, the spatial coordinates of the key point of the period are thermally marked in the 3D image according to the spatial coordinates of the key point in the target scene, so as to generate a 3D thermodynamic diagram of the target scene.

In the embodiment of the invention, the 3D thermodynamic diagram of the target scene can be generated by a method of periodically acquiring RGB images and corresponding depth images from RGB videos and depth videos recorded by a depth camera, acquiring the spatial coordinates of key points in the target scene according to the RGB images, the depth images and the spatial coordinates of the depth camera relative to the target scene, and based on the spatial coordinates of the key points of each period. Compared with thermodynamic diagrams in the prior art, the thermodynamic information of the position in any space in the target scene can be displayed, and therefore comprehensiveness of displaying the thermodynamic information through the thermodynamic diagrams is improved.

The following describes in detail a method for acquiring spatial coordinates of a key point in a target scene according to an RGB image, a depth image, and spatial coordinates of a depth camera with respect to the target scene in the embodiment of the present invention with reference to a specific embodiment.

FIG. 4 is a flow diagram illustrating a method of obtaining spatial coordinates of keypoints in the target scene, according to an exemplary embodiment. Fig. 5a is a schematic diagram of a spatial coordinate system of a target scene, where a dot of the coordinate system is preferably a corner of the target scene, and X, Y, Z three directions respectively extend in three directions, namely, a length direction, a width direction and a height direction with the corner as the dot. Fig. 5b is a schematic diagram of a camera coordinate system with a depth camera as an origin according to an embodiment of the present invention, and as shown in fig. 5b, the camera coordinate system may satisfy the following rule in a building process:

1. z1 is in the opposite direction from the Z-parallel direction;

2. x1 is parallel to X;

3. y1 is parallel to Y;

further, after the coordinate system of the camera is established, the included angles between the direction of the lens orientation of the camera and the X1, the included angles between the direction of the lens orientation of the camera and the X1 included angles and between the direction of the lens orientation of the camera and the Z1 included angles are respectively determined as

As shown in fig. 4, in S410, RGB image coordinates of the key points are extracted from the RGB image.

According to an example embodiment, it should be noted that, when extracting RGB image coordinates of key points from the RGB image, a passer box may be first marked from the RGB image based on a pedestrian detection algorithm, for example, yolo2 or DSSD algorithm. Further, according to a human key point detection algorithm, for example, a human key point monitoring algorithm of openposition, RGB image coordinates of five fingertips and wrists of a pedestrian are extracted from the pedestrian frame, and the center or the center of gravity of the palm is calculated according to the RGB image coordinates of the five fingertips and wrists, and finally, the RGB image coordinates of the center or the center of gravity are used as RGB image coordinates of the key points. Fig. 5c is a schematic diagram of an RGB image coordinate system according to an embodiment of the present invention. Wherein, a1 is the central point of the lens orientation of the camera formed in the RGB image, G is the key point extracted from the RBG image, and the distances from the central point on X2 and Y2 are W and H, respectively.

In S420, the depth coordinates of the key point are acquired based on the RGB image coordinates.

According to an example embodiment, the depth image corresponds to an RGB image, and each pixel point in the depth image represents its distance from the camera with depth. After the RGB image coordinates (x _ RGB, y _ RGB) of the key point are obtained, as shown in fig. 5d, it is a schematic diagram of a depth coordinate system provided in the embodiment of the present invention, and fig. 5d may obtain a depth value corresponding to the key point in the RGB coordinate system.

In S430, the spatial coordinates of the key point in the target scene are obtained based on the RGB image coordinates and the depth coordinates of the key point and the spatial coordinates of the depth camera with respect to the target scene.

After the depth coordinates of the key points are obtained, the actual distances from the key points to the distance center points in the RGB coordinate system may be calculated based on the depth values of the key points in the depth coordinates (i.e., the distances from the key points to the lens) and a preset comparison table of the distance from the lens to the actual distance of each pixel point.

For example, each point in the comparison table that is a distance from the lens may be preset, and then the actual distance of each pixel point corresponding to each point that is a distance from the lens may be obtained according to the actual size of each point and the pixel point occupied by the point in the depth coordinate system.

Distance from lens (m)	Actual distance per pixel (m)
		0-0.2	L_0-0.2
0.2-0.4	L_0.2-0.4
		……	……

TABLE 1

As can be seen from Table 1, when the distance from the lens is 0-0.2 m, the actual distance of each pixel point in the RGB coordinate system is 0-0.2 m.

Further, the actual distance corresponding to each pixel point on the RGB image at the corresponding depth in the comparison table can be found according to the depth value of the key point in the depth coordinate system from the found W and H occupying several pixel points, and the actual distance of the key point from the central point can be finally calculated.

Further, the spatial coordinates of the key points in the target scene can be calculated according to the following formula:

wherein the content of the first and second substances,

representing the depth value of the keypoint in a depth coordinate system,

the actual distance of each pixel point at the depth is represented, x, y and z respectively represent the spatial coordinates of the key point in the target scene,

a direction showing the lens orientation of the camera and X1,The angle between Y1 and Z1.

It should be noted that, when a target scene includes multiple depth cameras, an identifier of the depth camera may be first obtained, and further, according to the identifier, the spatial coordinates of the depth camera with respect to the target scene are found from the recorded spatial coordinates of each depth camera in the target scene.

It should be noted that, in the embodiment of the present invention, when depth cameras are deployed, it is required that ranges that can be shot by each depth camera do not overlap as much as possible, so that a utilization rate of the depth cameras can be improved, when a plurality of depth cameras acquire spatial coordinates of the same key point, one of the optimal depth cameras may be selected for calculation aiming at calculation of the spatial coordinates of the key point relative to a target scene, or the spatial coordinates of the key point relative to the plurality of depth cameras are acquired at the same time, and then fitting conversion is performed to acquire the spatial coordinates of the key point in the target scene.

In the embodiment of the invention, the RGB image coordinates of the key points are extracted from the RGB image, the depth coordinates of the key points are obtained based on the RGB image coordinates, the spatial coordinates of the key points relative to the depth camera are obtained based on the RGB image coordinates and the depth coordinates of the key points, and the spatial coordinates of the key points in the target scene are obtained based on the spatial coordinates of the depth camera relative to the target scene, so that the spatial coordinates of each key point in the target scene are obtained, and a basis is provided for constructing a 3D thermodynamic diagram.

It should be noted that, according to the method for generating a 3D thermodynamic diagram provided in the embodiment of the present invention, a person skilled in the art may simply modify the scheme, for example, obtain a spatial coordinate of each marker in a target scene in the target scene, detect a spatial coordinate of a key point based on the marker by using an RGB image and a depth image obtained by a depth camera, and further calculate the spatial coordinate of the key point in the target scene.

It should be clearly understood that the present disclosure describes how to make and use particular examples, but the principles of the present disclosure are not limited to any details of these examples. Rather, these principles can be applied to many other embodiments based on the teachings of the present disclosure.

The following are embodiments of systems of the present invention that may be used to perform embodiments of methods of the present invention. In the following description of the apparatus, the same parts as those of the foregoing method will not be described again.

Fig. 6 is a block diagram illustrating a system for 3D thermodynamic diagram generation in accordance with an embodiment of the present invention. Wherein at least one depth camera is included in the target scene, as shown in fig. 6, the system 600 includes:

the first obtaining module 610 is configured to periodically obtain RGB images and corresponding depth images from the RGB video and the depth video recorded by the depth camera;

a second obtaining module 620, configured to obtain spatial coordinates of a key point in the target scene according to the RGB image, the depth image, and spatial coordinates of the depth camera with respect to the target scene;

a generating module 630, configured to generate a 3D thermodynamic diagram of the target scene based on the spatial coordinates of the key points of each cycle.

According to some embodiments, the second obtaining module 620 comprises:

an extracting unit 622, configured to extract RGB image coordinates of a key point from the RGB image;

a depth coordinate obtaining unit 624, configured to obtain depth coordinates of the key points based on the RGB image coordinates;

a spatial coordinate obtaining unit 626, configured to obtain spatial coordinates of the key point in the target scene based on RGB image coordinates and depth coordinates of the key point and spatial coordinates of the depth camera with respect to the target scene.

According to some embodiments, the system further comprises: a judging module 640, configured to judge whether the key point corresponds to a marker in the target scene according to the spatial coordinate of the key point in each period;

the generating module 630 is configured to generate a 3D thermodynamic diagram of the target scene based on the spatial coordinates of the key points when the determination result of the determining module is yes.

As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to perform: periodically acquiring RGB images and corresponding depth images from the RGB video and the depth video recorded by the depth camera; acquiring the spatial coordinates of key points in the target scene according to the RGB image, the depth image and the spatial coordinates of the depth camera relative to the target scene; generating a 3D thermodynamic diagram of the target scene based on the spatial coordinates of the keypoints for each cycle.

Fig. 7 is a schematic structural diagram of an electronic device according to an exemplary embodiment. It should be noted that the electronic device shown in fig. 7 is only an example, and should not bring any limitation to the functions and the use range of the embodiment of the present application.

As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU)701, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data necessary for the operation of the system 700 are also stored. The CPU701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

The following components are connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program executes the above-described functions defined in the terminal of the present application when executed by the Central Processing Unit (CPU) 701.

It should be noted that the computer readable medium shown in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a first acquisition module, a second acquisition module, and a generation module. Wherein the names of the modules do not in some cases constitute a limitation of the module itself.

Exemplary embodiments of the present invention are specifically illustrated and described above. It is to be understood that the invention is not limited to the precise construction, arrangements, or instrumentalities described herein; on the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method of 3D thermodynamic diagram generation, comprising at least one depth camera in a target scene, further comprising a marker in the target scene, the marker comprising multiple layers, the method comprising:

acquiring the space coordinates of each period key point in the target scene according to the RGB image, the depth image and the space coordinates of the depth camera relative to the target scene;

judging whether the key points correspond to markers in the target scene or not according to the space coordinates of the key points in each period;

if not, discarding the key point;

if yes, determining the layer of the marker corresponding to the key point according to the spatial coordinates of the key point in each period, and thermally marking the spatial coordinates of the key point in each period at the position of the corresponding layer to generate a 3D thermodynamic diagram of the target scene, wherein the 3D thermodynamic diagram is used for displaying the thermal condition of each layer of the marker in the target scene;

acquiring the spatial coordinates of key points in the target scene according to the RGB image, the depth image and the spatial coordinates of the depth camera relative to the target scene, wherein the method comprises the following steps:

calculating the center or the gravity center of the palm according to the RGB image coordinates of the five fingertips and the wrist, and taking the RGB image coordinates of the center or the gravity center as the RGB image coordinates of the key points; acquiring the depth coordinate of the key point based on the RGB image coordinate;

acquiring the spatial coordinates of the key points in the target scene based on the RGB image coordinates and the depth coordinates of the key points and the spatial coordinates of the depth camera relative to the target scene;

acquiring the spatial coordinates of the key points in the target scene based on the RGB image coordinates and the depth coordinates of the key points and the spatial coordinates of the depth camera relative to the target scene comprises:

after the depth coordinates of the key points are obtained, determining the actual distances from the key points to the depth camera based on the depth values of the key points in the depth coordinates and a comparison table of the distance from a preset value to the depth camera and the actual distance of each pixel point;

calculating the spatial coordinates of the key points in the target scene according to the following formula:

wherein the content of the first and second substances,

wherein the content of the first and second substances,

representing the depth value of the keypoint in a depth coordinate system,

the actual distance of each pixel point at the depth is represented, x, y and z respectively represent the space coordinates of the key point in a space coordinate system XYZ of the target scene,

representing the angle of the camera's lens orientation in the camera coordinate system X1Y1Z1 with X1, Y1 and Z1, where Z1 is opposite to the Z-parallel direction, X1 is parallel to X, and Y1 is parallel to Y, where the depth coordinate system represents the coordinate system in which the depth values of the keypoints lie, where the camera's lens orientation forms a center point in an RGB image in the image coordinate system X2Y2The distances of the key point from the central point on X2 and Y2 are w and h, respectively.

2. The method of claim 1, when including a plurality of depth cameras in a target scene, based on spatial coordinates of the depth cameras relative to the target scene, comprising:

acquiring an identifier of the depth camera;

and searching the space coordinate of the depth camera relative to the target scene according to the identification.

3. The method of claim 1, wherein generating a 3D thermodynamic diagram of the target scene based on spatial coordinates of keypoints for each cycle comprises:

3D reconstructing the target scene to acquire a 3D image of the target scene;

and thermally marking the space coordinates of the key points of each period in the 3D image to generate a 3D thermodynamic diagram of the target scene.

4. A system for 3D thermodynamic diagram generation comprising at least one depth camera in a target scene, the target scene further comprising a marker comprising a plurality of layers, the system comprising:

the second acquisition module is used for acquiring the spatial coordinates of the key points in each period in the target scene according to the RGB images, the depth images and the spatial coordinates of the depth camera relative to the target scene;

the generating module is used for judging whether the key points correspond to the markers in the target scene or not according to the space coordinates of the key points in each period; if not, discarding the key point; if yes, determining the layer of the marker corresponding to the key point according to the spatial coordinates of the key point in each period, and thermally marking the spatial coordinates of the key point in each period at the position of the corresponding layer to generate a 3D thermodynamic diagram of the target scene, wherein the 3D thermodynamic diagram is used for displaying the thermal condition of each layer of the marker in the target scene;

wherein the second obtaining module includes:

an extraction unit, which is used for marking a pedestrian frame from the RGB image based on a pedestrian detection algorithm, extracting RGB image coordinates of five fingertips and wrists of a pedestrian according to the pedestrian frame and a human body key point detection algorithm,

calculating the center or the gravity center of the palm according to the RGB image coordinates of the five fingertips and the wrist, wherein the RGB image coordinates of the center or the gravity center are used as the RGB image coordinates of the key points;

the spatial coordinate acquisition unit is used for acquiring the spatial coordinates of the key points in the target scene based on the RGB image coordinates and the depth coordinates of the key points and the spatial coordinates of the depth camera relative to the target scene;

wherein the content of the first and second substances,

wherein the content of the first and second substances,

representing the depth value of the keypoint in a depth coordinate system,

representing the angle between the direction in which the camera lens is facing in the camera coordinate system X1Y1Z1 and X1, Y1 and Z1, where Z1 is opposite to the Z-parallel direction, X1 is parallel to X, and Y1 is parallel to Y, where the depth coordinate system represents the coordinate system in which the depth values of the key points are located, where the lens orientation of the camera lens forms a center point in an RGB image in the image coordinate system X2Y2, and the distances of the key points from the center point on X2 and Y2 are w and h, respectively.

5. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method steps of any one of claims 1 to 3.

6. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out the method steps of any of claims 1-3.