JP2023506990A - Systems and methods for displaying video data within a target environment - Google Patents

Abstract

A system for automatically displaying output information on a display surface based on activity within the target environment includes an image capture device for capturing data of the target environment in real time and processing the data to produce output information. and a processing unit for determining the content of the output information and a video output unit for displaying the output information on the display surface. The video output unit is configured to rotate 360°, and the rotation angle of the video output unit is controlled by the processing unit. The image capture device further captures the output information displayed on the display surface in real time and provides feedback data to the processing unit in real time so that the processing unit can determine the focus, orientation, and output displayed on the video output unit. Allows you to control information.

Description

FIELD OF THE DISCLOSURE This disclosure relates generally to systems and methods for displaying video data in a target environment, and more particularly to closed-loop artificial intelligence (AI) driven processes using real-time interactive visual and audio prompts. for displaying video data based on.

Businesses typically spend a large amount of money advertising their newly launched products by advertising on display screens in public places, such as shopping centers, train stations/bus stops, and the like. In order to maximize the company's profit and return on investment, the company wants the maximum number of customers to purchase the new product. However, existing display screens used for advertising only display pre-programmed data. In particular, existing display screens do not display promotions based on targeted customers.

Additionally, in Self Check-Out (SCO) stores, cashiers are not physically present at the billing counter to handle billing for customers. This has resulted in increased theft rates. Therefore, in SCO stores, there is a need for a system to provide audio/video alerts based on theft observations. Similarly, if a customer is unable to use the SCO system properly, the system should notify store management that the customer may need attention and assistance.

Additionally, in the event of natural disasters (e.g. earthquakes, fires, tsunamis, etc.), warnings may be necessary to inform people in public places of the disaster and to provide them with directions to safer places. . In an emergency situation, such as an attack on an army/naval base or a terrorist attack in a public place, current security procedures help to locate the attacker or otherwise help officials to apprehend the attacker. Does not support providing video/audio instructions.

Furthermore, current systems for teaching/training students do not observe the student's environment and adapt teaching/training accordingly. For example, current training systems follow pre-established procedures for training medical or veterinary students and do not take into account the environment of these students.

Therefore, in view of the above, a need exists for a system that takes into account the environment of one or more target users and provides automated audio/video output accordingly.

In one aspect of the present disclosure, a system is provided for automatically displaying output information on a display surface based on one or more activities within a target environment. The system includes an image capture device configured to capture image and video data of the target environment in real time to perceive one or more activities. The system may include a processing unit configured to process image and video data to determine locations for displaying output information and to determine content of output information. The system comprises a video output unit configured to display output information on a display surface, the video output unit configured to rotate 360°, the angle of rotation of the video output unit being determined by the processing unit It may further include a controlled video output unit. The image capture device captures output information displayed on the display surface in real time and provides feedback data to the processing unit in real time so that the processing unit can determine the focus, orientation, and output displayed on the video output unit. It is further configured to allow information to be controlled.

In another aspect of the present disclosure, a method is provided for automatically displaying output information on a display surface based on one or more activities within a target environment. The method includes capturing image and video data of the target environment in real time to recognize one or more activities. The method may further include processing the image and video data to determine locations for displaying the output information and determining content of the output information. The method may further include displaying the output information on the display surface with the video output unit, wherein the video output unit is configured to rotate 360 degrees. The method may further include controlling the rotation angle of the video output unit based on the captured image and video data. The method further comprises capturing the output information displayed on the display surface in real time and generating feedback data to control the focus, orientation, and output information displayed on the video output unit. can contain.

In yet another aspect of the present disclosure, a computer programmable product is provided for automatically displaying output information on a display surface based on one or more activities within a target environment. A computer programmable product comprises a set of instructions which, when executed by a processor, causes the processor to generate image and video data of a target environment in real time to perceive one or more activities. Capturing and processing image and video data to determine locations for displaying output information, determining content of the output information, and generating output information based on the processed data. and displaying the output information on a display surface by means of a video output unit. The video output unit is configured to rotate 360 degrees, and the rotation angle of the video output unit is controlled based on the captured image and video data. Additionally, the output information displayed in real time on the display surface is captured and feedback data is generated to control the focus, orientation, and output information displayed on the video output unit.

Various embodiments of the present disclosure provide a system that captures human behavior, interacts and communicates with humans, and instructs or informs humans in a manner appropriate to the process and environment being observed. The system can take visual, audio, and other sensor inputs, create visual and audio outputs, and form closed-loop interactions managed by software intelligence running in the background. The system can further act as an intelligent instructor/coach/manager, enabling automatic assurance of optimal performance against standards and prescribed processes. The system uses a camera for input and a projector for output, controlled by AI software, to create the opportunity to have two-way communication with the user.

The system is useful in scenarios where users need to be instructed or trained. A camera may observe environmental processes and use a projector to guide the user to act according to the desired result and keep a record of the training. Another example would be training medical or veterinary students to perform a particular procedure. Closed-loop feedback ensures that AI software is under real-time control and can change or modify processes and activities as they occur. Closed-loop AI-driven process control using real-time interactive visual and audio prompts/nudges to guide, control and/or ensure optimal process or behavioral outcomes.

It will be appreciated that the features of the disclosure are permissible to be combined in various combinations without departing from the scope of the disclosure as defined by the appended claims.

The above summary, and the following detailed description of illustrative embodiments, are better understood when read in conjunction with the accompanying drawings. For purposes of illustrating the present disclosure, exemplary structures of the present disclosure are shown in the drawings. However, the disclosure is not limited to the specific methods and instrumentalities disclosed herein. Additionally, those skilled in the art will appreciate that the drawings are not to scale. Wherever possible, similar elements are denoted by identical numbers.

1 is an illustration of an example system for displaying video data based on activity within a target environment in accordance with an embodiment of the present disclosure; FIG. FIG. 4 is a diagram of example operations for processing image frames captured by an image capture device, in accordance with an embodiment of the present disclosure; [0014] Fig. 4 is a diagram of an exemplary video output unit for projecting predefined image/video data generated by a processing unit onto a display surface, in accordance with an embodiment of the present disclosure; FIG. 10 is an illustration of an exemplary mechanism for rotating a motorized mirror about vertical and horizontal axes in the plane of the mirror, in accordance with an embodiment of the present disclosure; [0014] Fig. 4 illustrates an exemplary motor including two electrically controlled levers, in accordance with one embodiment of the present disclosure; 4 is an exemplary flow chart illustrating a method for automatically displaying video data on a display surface based on one or more activities within a target environment;

In the accompanying drawings, underlined numbers are employed to represent the item in which the underlined number is placed or the item to which the underlined number is adjacent. Non-underlined numbers relate to the item identified by the line connecting the non-underlined number to the item. If a number is not underlined and has an associated arrow, the non-underlined number is used to identify the general item to which the arrow points.

The following detailed description illustrates embodiments of the disclosure and the manner in which they may be implemented. While the best mode of carrying out the disclosure has been disclosed, those skilled in the art will recognize that other embodiments for making or practicing the disclosure are possible.

FIG. 1 is an illustration of an exemplary system 100 for displaying video data based on activity within a target environment 101, according to one embodiment of the present disclosure. In one embodiment of the present disclosure, the target environment 101 may relate to a self-checkout store (SCO), not shown, but with entities such as products, conveyors, industrial robots, etc., and activities such as operators enter or leave the scene, pick, drop, move, weigh or scan items, operate touch screen displays and pay via cash, mobile e-commerce or credit card and so on. However, it will be apparent to those skilled in the art that the target environment 101 may relate to general industrial areas, military/naval bases, training halls, security inspection areas, shopping centers, and restaurants. In addition to the above-identified target environments 101, the system 100 can also be used for retail automation, customer upselling, employee mentoring, employee training, logistics automation (import/export of goods), medical instructions such as EG surgical training, It can be useful in surgical specialist training with visual cues, emergency orders in the event of fire, earthquake, and gunfire.

System 100 includes a processing unit 102, an image capture device 104, an audio recording device 106, a sensor unit 108, and a video output unit 110 communicatively coupled together through a communication network. The communication network may be any suitable wired network, wireless network, a combination thereof or any other conventional network without limiting the scope of this disclosure. Some examples may include local area networks (LANs), wireless LAN connections, Internet connections, point-to-point connections, or other network connections and combinations thereof. In one example, the network may include a mobile communication network, such as a 2G, 3G, 4G, or 5G mobile communication network. A communication network may be coupled to one or more other networks, thereby providing coupling between a greater number of devices. Such may be the case, for example, when networks are coupled together through the Internet.

The image capture device 104 captures one or more images and videos of the target environment 101 in real time to recognize the actions of various entities such as humans, animals, and things in the image frames, and the captured It is arranged to transmit data to the processing unit 102 . Examples of image capture devices 104 are 360° cameras, Closed-Circuit Television (CCTV) cameras, high definition (HD) cameras, non-HD cameras, handheld cameras, traffic cameras, police vehicle cameras, and unmanned Including, but not limited to, cameras on unmanned aerial vehicles (UAVs).

Audio recording device 106 is configured to record audio data from target environment 101 and transmit the recorded audio data to processing unit 102 . Examples of audio recording device 106 include, but are not limited to, dynamic microphones, condenser microphones, piezoelectric microphones, or ribbon microphones.

The sensor unit 108 is configured to detect events or changes within the target environment 101 and transmit detected information to the processing unit 102 . In one embodiment of the present disclosure, target environment 101 is an army/naval base and sensor unit 108 may include one or more radars installed at the base to detect attacks on the base. In another embodiment, the target environment 101 is one or more residential/commercial buildings, and the sensor units 108 detect the occurrence of a natural disaster at or near the building and process the detected information. It may also include sensors installed in the building for transmission to unit 102 . In yet another embodiment, the target environment 101 is a security checkpoint area of an airport, shopping (or other) mall, or other similar location, and the sensor unit 108 detects people's luggage, bags, It may also include an X-ray device installed to detect the presence of suspect items in clothing or other aspects and transmit detected information to processing unit 102 .

Processing unit 102 is a central control unit that controls the operation of image capture device 104 , audio recording device 106 , sensor unit 108 and video output unit 110 . Processing unit 102 processes input data received from image capture device 104 , audio recording device 106 , and sensor unit 108 , generates output information based on the processed data, and outputs it to display surface 112 through video output unit 110 . configured to display the output information in the In one embodiment of the present disclosure, the output information may include predefined video and audio outputs including, but not limited to, alerts, notifications, advertisements, instructions, and training videos for the target environment 101. In one embodiment of the present disclosure, the video output unit 110 may include a projector, and the display surface 112 may be a white projection screen, a gray projection screen, or a white wall for displaying projected images from the projector. can contain. In one embodiment of the present disclosure, processing unit 102 is configured to control the focus and direction of video output unit 110 and also control the content displayed by video output unit 110 . In another embodiment of the present disclosure, video output unit 110 may include non-projection displays such as light emitting diodes (LEDs), liquid crystal displays (LCDs), and organic light emitting diodes (OLEDs).

In one embodiment of the present disclosure, image capture device 104 captures image/video data displayed on display surface 112 and audio recording device 106 records audio data being played on display surface 112 . Processing unit 102 uses the feedback data to improve the quality of the output information displayed on display surface 112 to enhance user feedback.

In one embodiment of the present disclosure, processing unit 102 includes an artificial intelligence (AI) platform capable of guiding visual information to precise locations within target environment 101 . Processing unit 102 may be implemented locally at a local computing device or at a remote processing server. In the context of this disclosure, processing unit 102 may include an AI-based processor, a graphics processing unit (GPU) for processing video/image data, memory for storing one or more instructions. Throughout this disclosure, the term "AI-based processor" relates to a computational element operable to respond to and process instructions stored within a corresponding memory. Optionally, the AI-based processor comprises a microprocessor, microcontroller, complex instruction set computing (CISC) microprocessor, reduced instruction set (RISC) microprocessor, very long instruction word ( VLIM (very long instruction word) including, but not limited to, a microprocessor, or any other type of processing circuitry. Processing unit 102 may include an initial set of predefined outputs that are programmed and extended with dynamic outputs created by corresponding AI software.

FIG. 2 is a diagram of exemplary operations for processing image frames 202 captured by image capture device 104 (as shown in FIG. 1), according to one embodiment of the present disclosure. Image frame 202 is a circular shape captured by the 360° lens of image capture device 104 . Processing unit 102 (as shown in FIG. 1) receives image frame 202 and creates a smoothed representation of image frame 202 to produce smoothed image 204 . In one embodiment of the present disclosure, the processing unit 102 (as shown in FIG. 1) compensates for fisheye distortion while creating the smoothed image 204 and grid view images of the smoothed image 204. 206 is generated.

Grid view image 206 includes a live grid with reference to target environment 101 (as shown in FIG. 1). In one implementation, the grid properties (ie number of rows and number of columns) are user configurable. In another implementation, grid properties are dynamically variable. In this case, the grid property is the processing unit 102 that analyzes the motion detected in the field of view of the image capture device 104 and ensures that the position of each individual element within the scene can be distinguished using the grid reference. determined by The grid reference is then used to inform the processing unit 102 about the activity occurring within the target environment 101 and the location of that activity.

In one example, if target environment 101 is a shopping center, image 204 may show a location that is typically crowded with people. The processing unit 102 may use this information to display a predefined video promotion at the corresponding location to draw the maximum number of people's attention to the promotion. In another example, if the target environment 101 is a security screening area of an airport, shopping (or other) mall, or another similar location, the images 204 may include one or more suspected prohibited It may relate to an X-ray image that reveals the presence of an article or item. Processing unit 102 may use this information to generate an audio/video alert to the relevant security administrator. Projecting video/image data onto a display surface is discussed in detail with reference to FIG. 3A.

FIG. 3A illustrates projection of predefined image/video data generated by processing unit 301 (similar to processing unit 102) onto display surface 310 (similar to display surface 112), according to one embodiment of the present disclosure. 3A is a diagram of an exemplary video output unit 300 (similar to video output unit 110) for

Video output unit 300 includes light source 302 , lens 304 , motorized focus system 306 , and motorized mirror 308 . Examples of light sources 302 may include, but are not limited to, lasers, lamps or LEDs (light emitting diodes). Lens 304 may include lenses with optical and digital zoom and focus capabilities. Motorized focusing system 306 is configured to direct light from light source 302 that is reflected by lens 304 toward motorized mirror 308 . Motorized mirror 308 is configured to be moved in each of the vertical and horizontal axes to project a predefined image/video onto display surface 310 at different tilts/positions. In one embodiment of the present disclosure, motorized mirror 308 rotates the predefined image/video up to 90 degrees to display the predefined image at a predefined location on display surface 310. Configured. The mechanism for rotating motorized mirror 308 is discussed in detail with reference to FIG. 3B.

FIG. 3B illustrates a mechanism 312 for rotating motorized mirror 308 about vertical and horizontal axes in the plane of the mirror. Mechanism 312 includes a motor 314 fixed to the ceiling and coupled to motorized mirror 308, and a physical hook 315 that ensures that motion rod 317 remains in a horizontal position. Motorized mirror 308 is fixedly mounted to the ceiling. Therefore, the normal axis to motorized mirror 308 is fixed, thereby limiting its movement along the depth axis.

FIG. 3C illustrates that motor 314 may include two electrically controlled levers 316a and 316b adapted to control rotation of motorized mirror 308 in horizontal and vertical directions.

Referring again to FIG. 3B, motor 314 is controlled in real time by processing unit 301 (similar to processing unit 102) such that when activity is relayed to processing unit 318 by a corresponding sensor unit, the processing unit 301 generates the associated video/image output and controls motor 314 to guide the video/image output to the correct location on display surface 310 . In the context of the present disclosure, motorized mirror 308 is configured to rotate 360° as directed by processing unit 318 .

Referring back to FIG. 1, image capture device 104 is configured to capture video/image data displayed on output surface 112 and provide the video/image data to processing unit 102 as feedback data. For example, if the projection is not sharp enough, processing unit 102 may adjust the focusing lens of video output unit 110 to improve the quality of the output image/video data.

In one embodiment of the present disclosure, based on past experience and software learning, the processing unit 102 predicts when such undesirable activity is likely to occur in order to proactively and prevent such undesirable activity. configured as The processing unit 102 may also build a risk profile to be used in “What If” scenario analysis. For example, processing unit 102 may detect that a process is not being tracked accurately and highlight this to the user. However, the processing unit 102 may simultaneously view this scenario as both a potential negative risk and a potential positive risk. The processing unit 102 can predict the expected outcome of both scenarios, and if it finds an unexpected benefit from a process deviation, it can automatically change the process to improve the change. can be incorporated into the process. This can be done on a standalone station to observe the results before forcing changes throughout the system. The processing unit 102 can remember past processes if it needs to go back to its original state for some unforeseen reason.

FIG. 4 is an exemplary flowchart illustrating a method 400 for automatically displaying video data on a display surface based on one or more activities within a target environment in accordance with this disclosure. The method is depicted as a collection of steps in a logical flow diagram, which represents a sequence of steps that can be implemented in hardware, software, or a combination thereof.

At step 402, image and video data of the target environment are captured in real time to enable recognition of one or more activities occurring within the target environment. Imagery and video data are captured from 360° cameras, closed circuit television (CCTV) cameras, high definition (HD) cameras, non-HD cameras, handheld cameras, traffic cameras, police vehicle cameras, and unmanned aerial vehicles (UAVs). captured using at least one of the cameras.

At step 404, the image and video data are processed and output information is generated based on the processed data. According to one embodiment of the present disclosure, video/image data is processed using a graphics processing unit (GPU). In one embodiment of the present disclosure, the output information includes predefined video and audio data including at least one of alerts, notifications, advertisements, instructions, and training videos. Further, according to one embodiment of the present disclosure, the output information is generated based on grid view images. A grid-view image is generated by receiving a circular input image frame, creating a smoothed representation of the circular input image frame, and generating a grid-view image of the smoothed representation. Additionally, the grid-view image provides information regarding the position of one or more entities within the circular input image frame.

At step 406, the output information is displayed on the display surface by the video output unit. In one embodiment of the present disclosure, the video output unit is configured to rotate 360 degrees. In one embodiment of the present disclosure, the video output unit includes a projector, and the display surface includes at least one of a white projection screen, a gray projection screen, and a white wall for displaying the projected image. .

At step 408, the rotation angle of the video output unit is controlled based on the captured image and video data. In one embodiment of the present disclosure, the video output unit includes a light source, lenses, motorized mirrors and a motorized focus system. The motorized mirror is configured to be moved horizontally and vertically to project a predefined image onto one or more positions of the display surface, the movement of the motorized mirror being based on the angle of rotation. are controlled by one or more electrically controlled levers. Additionally, the motorized focus system is configured to direct light from the light source reflected by the lens toward the motorized mirror.

At step 410, the captured output information is displayed on a display surface in real time, feedback data is generated to control the focus and orientation of the video output unit, and the output information displayed on the video output unit is further modified. controlled. In one example, if the projection is not sharp enough, the focus lens of the video output unit is adjusted to improve the quality of the displayed output information. This is achieved by having a feedback loop between the projected image and the camera input. AI software can detect out-of-focus images and send correction signals to the lens to improve image quality. This is an ongoing loop as the display surface may move or change shape.

Variations to the embodiments of the disclosure described above are possible without departing from the scope of the disclosure as defined by the appended claims. The terms "include," "comprise," "incorporate," "consist of," "have," "is," and the like are used to describe and claim the disclosure in a non-exclusive manner. is intended to be interpreted as such, i.e., to allow the presence of items, components or elements not explicitly described. References to the singular should also be construed to refer to the plural.

Claims (19)

A system for automatically displaying output information on a display surface based on one or more activities within a target environment, the system comprising:
an image capture device configured to capture image and video data of the target environment in real-time to perceive one or more activities;
a processing unit configured to process the image and video data to determine a location for displaying the output information and to determine the content of the output information;
a video output unit configured to display the output information on the display surface, the video output unit configured to rotate 360°, the rotation angle of the video output unit a video output unit controlled by
The image capture device captures in real-time the output information displayed on the display surface and provides feedback data to the processing unit in real-time so that the processing unit can determine focus, orientation, and the video output unit. The system is further configured to allow controlling the content of the output information displayed in the system. further comprising an audio recording device configured to record audio data of the target environment and transmit the recorded audio data to the processing unit, wherein the audio recording device converts audio data of the output information into feedback data; and providing the feedback data to the processing unit. 2. The system of claim 1, wherein the processing unit comprises an artificial intelligence (AI) platform configured to guide visual information to one or more predefined locations within the target environment. 2. The system of claim 1, wherein the processing unit comprises a graphics processing unit (GPU) for processing video/image data. 2. The system of claim 1, wherein the output information includes predefined video and audio data including at least one of alerts, notifications, advertisements, instructions, and training videos. 2. The video output unit of claim 1, wherein the video output unit comprises a projector and the display surface comprises at least one of a white projection screen, a gray projection screen, and a white wall for displaying projected images. system. The processing unit is configured to receive a circular input image frame from the image capture device, create a smoothed representation of the circular input image frame, and generate a grid view image of the smoothed representation. wherein said grid-view image provides information regarding the position of one or more entities within said circular input image frame, said processing unit generating said output information based on said grid-view image; 2. The system of claim 1, configured to display. further comprising a sensor unit configured to detect one or more events and changes in said target environment, said sensor unit including radar, x-ray, scanner, motion sensor, temperature sensor, gas sensor, and fire sensor 2. The system of claim 1, comprising at least one of: The video output unit is
a light source;
a lens;
A motorized mirror configured to be moved horizontally and vertically to project a predefined image onto one or more locations of said display surface, wherein movement of said motorized mirror is associated with said process. motorized mirrors controlled by one or more electrically controlled levers operated by the unit;
and a motorized focusing system configured to direct light from the light source reflected by the lens toward the motorized mirror. A method for automatically displaying output information on a display surface based on one or more activities in a target environment, the method comprising:
capturing image and video data of the target environment in real time to recognize one or more activities;
processing the image and video data to determine a location for displaying the output information and determining content of the output information;
displaying the output information on the display surface by a video output unit, the video output unit being configured to rotate 360°;
controlling a rotation angle of the video output unit based on the captured image and video data;
capturing in real time the output information displayed on the display surface and generating feedback data to control focus, orientation, and the content of the output information displayed on the video output unit; including, method. recording audio data of the target environment and transmitting the recorded audio data;
11. The method of claim 10, further comprising recording audio data of said output information as feedback data and providing said feedback data. 11. The method of claim 10, further comprising using an AI platform to guide visual information to one or more predefined locations within the target environment. 11. The method of claim 10, further comprising using a graphics processing unit (GPU) to process the video/image data. 11. The method of claim 10, wherein the output information includes predefined video and audio data including at least one of alerts, notifications, advertisements, instructions, and training videos. 11. The video output unit of claim 10, wherein the video output unit comprises a projector and the display surface comprises at least one of a white projection screen, a gray projection screen and a white wall for displaying projected images. Method. receiving a circular input image frame, creating a smoothed representation of the circular input image frame, and generating a grid-view image of the smoothed representation, the grid-view image being the circular and generating and displaying the output information based on the grid view image. the method of. Further detecting one or more events and changes in said target environment using a sensor selected from the group consisting of radar, x-ray, scanner, motion sensor, temperature sensor, gas sensor, and fire sensor. 11. The method of claim 10, comprising: The video output unit is
a light source;
a lens;
A motorized mirror configured to be moved horizontally and vertically to project a predefined image onto one or more locations of the display surface, wherein movement of the motorized mirror is performed by one or a motorized mirror controlled by a plurality of electrically controlled levers;
and a motorized focus system configured to direct light from the light source reflected by the lens toward the motorized mirror. A computer programmable product for automatically displaying output information on a display surface based on one or more activities within a target environment, said computer programmable product comprising a set of instructions. , the set of instructions, when executed by a processor, causes the processor to:
capturing image and video data of the target environment in real time to recognize one or more activities;
processing the image and video data to determine a location for displaying the output information and determining content of the output information;
displaying the output information on the display surface by a video output unit, the video output unit being configured to rotate 360°;
controlling a rotation angle of the video output unit based on the captured image and video data;
capturing in real time the output information displayed on the display surface and generating feedback data to control focus, orientation, and the content of the output information displayed on the video output unit; A computer-programmable product that causes

Images