BE1030666B1 - HAND MOVEMENT OR HAND POSE RECOGNITION FOR SMARTGLASSES - Google Patents

Abstract

The present invention concerns an improved method for managing audio intake, and in particular enabling it, during videoconferencing for a medical user in a sterile room.

Description

1 BE2022/5516

HAND MOVEMENT OR HAND POSE RECOGNITION FOR SMARTGLASSES

TECHNICAL DOMAIN

The invention relates to an improved method of managing video conferences in a hands-free manner.

STATE OF THE TECHNOLOGY

During medical interventions and other procedures in a sterile environment (such as certain actions in a clean room and similar situations), increasing emphasis is placed on contactless handling, and the persons present should avoid interacting with other objects as much as possible , in particular with their hands, as they must remain sterile to perform certain actions at a later time.

In a medical environment, it must be assumed that a surgeon must be sterile for a considerable time before the procedure begins, and then may no longer interact with items that are not sterile. However, current operating rooms contain a large number of medical instruments that can be interacted with, but this endangers sterility.

In addition, experts from multiple areas are increasingly being called upon for medical interventions, especially in the case of complex procedures. However, it is difficult for all of these to be physically present due to the limited space in an OR, but also because each additional person further compromises sterility and also makes the preparation for each procedure much more cumbersome and longer.

A solution that is increasingly being used is videoconferencing, where experts can assist and advise the person (or persons) physically present in the sterile room. In this application, smart glasses or other wearable devices with one or more image sensors are often used, so that the person on site has his hands free, and the remote experts have a similar image as the person on site, and thus better mutual understanding. can coordinate. It is also advantageous that they can clearly control the person on site due to the strongly overlapping field of view between the image sensor (what the remote experts see) and the view of the person on site.

2 BE2022/5516

However, it remains a problem to configure the videoconference, as it must initially be used by the person in the sterile environment, who has access to the instrumentation and who also has an overview of who is needed at the videoconference.

One of the problems recognized in this context is the difficulty in managing the video conference and peripherals hands-free for the (medical) user in the sterile room. Specifically, a difficulty lies in the fact that audio intake, and specific voice commands, must be disabled - temporarily - in some circumstances to prevent unauthorized users from manipulating certain equipment or the video conference, or from conflicting with ambient noise. On the other hand, there is also the wish for the person on site to no longer share their voice (and/or other ambient noise) in the video conference, either so as not to disturb the other parties, or for privacy reasons, or because the person on site feels uncomfortable about this.

In the above situation, however, it is not possible for the on-site user to re-enable voice command, as the on-site user must work hands-free and voice command is disabled. One option is to engage a so-called circulation nurse or other person who can use his hands to re-enable audio intake, and also voice command. Another alternative is to try to communicate the desire to re-enable this to a remote user who has a certain amount of control over the video conference and can re-enable it, for example via pre-arranged signals. However, this depends on this remote user and his control over the video conference, which is not desirable.

WO2022/035910 describes a system and method for holding video conferences in medical applications, where head movements are used to activate certain menu choices and other manipulations. However, this is not desirable since many simple head movements are already used during an operation, and the use of more complex head movements is all unnecessarily more difficult. Furthermore, this affects the positioning of the user's head on site, while their field of view should preferably be kept stable, whether to maintain a good overview themselves at all times, or to provide remote users with a stable and continuous view. keep giving.

3 BE2022/5516

The present invention aims to solve at least some of the above problems, and provides an intuitive, simple method that allows a person to manage a video conference hands-free.

SUMMARY OF THE INVENTION

The invention relates to an improved method for managing a video conference between a medical user located in a sterile room, preferably during medical treatment in the sterile room, and one or more remote users, wherein the user in the sterile room has a wears a portable, hands-free image sensor, wherein a second, preferably stationary, image sensor is provided in the sterile area, the second image sensor participating in the video conference, and wherein a microphone is provided in the sterile area, preferably on the portable hands-free image sensor, the microphone participating in the video conference, the method including following steps.

The method includes the following steps: a. sharing video input from the portable, hands-free image sensor in the video conference; b. sharing audio input from the microphone in the video conference; c. optionally sharing additional video and/or audio input from one or more of the remote users in the video conference; d. disabling audio input via the microphone or disabling audio input via the microphone, whereby the audio input from the microphone is not shared over the video conference when the audio input via the microphone is disabled; e. detecting a hand movement or hand pose with the second image sensor; f. recognizing a predetermined hand movement or predetermined hand pose in the hand movement or hand pose detected by the second image sensor; g. switching on the audio input via the microphone upon recognition of the predetermined hand movement or the predetermined hand pose with the second image sensor, whereby the audio input from the microphone is shared over the video conference when the audio input via the microphone is switched on.

4 BE2022/5516

Preferably, the wearable, hands-free image sensor is part of a head-worn device (HMD or head-mounted device), and in particular smart glasses.

By working on the basis of hand movements and/or hand poses, where predetermined (pre-programmed) movements/poses are associated with a certain action, here in particular switching on audio intake via the microphone in the sterile room, the person is able to on site to switch this back on without using his or her hands, thereby breaking sterility, for example.

It should be noted that not all participating devices/tools in the video conference are actually shared in the video conference. For example, data from a certain instrument can be passively available in the video conference, but can only be shared (temporarily) at the request of a user, or a certain audio feed can be used to control the video conference via voice commands, without this audio feed can be listened to effectively in the video conference. In addition, it is of course also possible to explicitly exclude certain feeds from participating devices/instruments from being shared in the video conference.

It should be noted that the application of the method is not necessarily limited to a medical context, but can also be used outside of it, such as in clean rooms or other contexts where manual contact with objects is undesirable, difficult or impossible, or avoided should be done, but also in applications where this is possible, such as a 'normal' video conference. The advantages of the present invention also apply in these contexts.

DESCRIPTION OF THE FIGURES

Figure 1 shows a schematic arrangement of a sterile room with users, instruments, and the remote users, according to an embodiment of the invention.

Figure 2 provides a schematic representation of the graphical presentation displayed by a remote user participating in a video conference, according to an embodiment of the invention.

DETAILED DESCRIPTION

Unless otherwise defined, all terms used in the description of the invention, including technical and scientific terms, have the meanings commonly understood by those skilled in the art of the invention. For a better appreciation of the description of the invention, the following terms are explicitly explained. “A”, “the” and “it” in this document refer to both the singular and the plural unless the context clearly suggests otherwise. For example, “a segment” means one or more than one segment.

When “about” or “around” is used in this document for a measurable quantity, a parameter, a period of time or moment, or the like, variations of +/-20% or less are meant, preferably +/-10% or less, more preferably +/-5% or less, even more preferably +/-1% or less, and even more preferably +/-0.1% or less than and of the quoted value, to the extent that such variations from are applicable in the described invention. However, this must be understood to mean that the value of the quantity for which the term “approximately” or “around” is used is itself specifically disclosed.

The terms “comprise”, “comprising”, “consist of”, “consisting of”, “providing”, “containing”, “containing”, “containing”, “containing”, “containing”, “containing” are synonyms and are inclusive or open terms that indicate the presence of what follows, and do not exclude or preclude the presence of other components, features, elements, members, steps, known or described in the prior art.

Instruments/devices participating in the video conference share their own data feed (video, audio and/or other) that is made available in the video conference, and may or may not be visualized, played or displayed in another way, and/or received the data feed from the video conference, typically from a server, and play it back (visual, auditory and/or other). In what follows it will become clear whether an instrument/device shares its own data feed and/or receives the data feed from the video conference, which often also follows logically from the type of instrument/device. Unless otherwise stated, it may be assumed that data-creating instruments/devices (e.g. electrocardiograph, endoscope, camera,

6 BE2022/5516 microphone, etc.) share their own created data feeds with the video conference and optionally also receive the data feed from the video conference, and that instruments/devices that do not clearly create their own data (e.g. screen, sound system) only receive the data feed from the video conference.

The terms “hand gestures”, “hand movements” and “hand poses” are used in the following as synonyms for simplicity unless explicitly stated otherwise, whereby both specific sequences of hand positionings (movements/gestures) and single positionings (poses) are considered to be included of this.

Quoting numerical intervals through the endpoints includes all integers, fractions, and/or real numbers between the endpoints, including these endpoints.

In a first aspect, a method for managing a video conference according to claim 1. A general context is outlined, among other things, in

PCT/EP2021/087293, which describes a specific methodology for hands-free setup and management of videoconferencing in a medical context, but the current method is not limited to this. The use of videoconferencing in a medical context is becoming increasingly common, partly because it makes it possible to make expertise from many sources available in a very simple way, without being bound by geographical proximity, sterility requirements, space for movement, preparation, etc., for the other participants. This way, specialists can be (temporarily) involved in every aspect during a medical procedure, without major impact on these specialists or on the location itself.

Just in terms of reducing the danger to a patient by reducing the number of staff present, this is incredibly valuable.

As already discussed, the present invention offers significant advantages in the context of a medical situation where the user wearing the hands-free image sensor can interact with the image sensor, as well as many other instruments, via voice commands only. In practice, it turned out that videoconferencing presented new problems, including in larger operating theaters where a number of other instruments are part of the videoconference, in the sense that output is shared in the videoconference, and/or in the sense that input from the video conference is updated accordingly. A number of the instruments in the sterile room, including often the image sensors themselves (such as in the form of smart glasses), increasingly have the option of

7 BE2022/5516 voice commands. However, it is also possible to control instruments that participate in the video conference via this link.

This can also be done for instruments that do not have voice command functionality themselves, by integrating the functionality into the video conference itself, preferably where this is limited to audio input that is picked up via the microphone in the sterile room, to avoid remote users have this option without any restrictions.

When a command is recognized for a specific instrument (picked up by the microphone), an action associated with the recognized command is communicated via the video conference link to the instrument, upon which the instrument acts.

Remote users, with or without certain control over matters in the video conference (as described in the above-mentioned application) in many cases also have the ability to communicate verbally with the other users in the video conference, and in particular with the user on site. For example, the user can wear headphones or earplugs on site, but there is often a sound system in the sterile room, which may or may not be integrated into another instrument. This also makes it possible for remote users to directly control instruments that use voice commands.

In addition, communication can also take place with everyone present in the sterile room, who is not necessarily participating in the video conference.

In some situations, be it for privacy reasons, for concentration, or simply because sharing the audio from the sterile room is not desired, the audio intake via the microphone in the sterile room is turned off.

Additionally, in some circumstances the on-site user may wish to disable the ability of other parties to control on-site instruments via voice commands, without however disabling the audio in the sterile room. This can easily be done by turning off all audio intake via the microphone in the sterile room.

In such situations, it is up to the on-site user, whose hands are proverbially tied, to re-enable this, as voice commands rely on audio intake via the microphone, which is disabled, and they cannot manually change settings.

In order not to be at the mercy of a remote user who can guess that this should be switched back on (if a remote user even already has this option), the system according to the invention has applied an improvement to the

8 BE2022/5516 detect and recognize a hand movement with a second image sensor, also participating in the video conference, and based on whether or not it is recognized, automatically reactivate audio intake via the microphone.

Recognition is done via recognition algorithms that analyze the detected images and compare them with the predetermined hand movements and/or hand poses.

Typically, a certain threshold is set for the degree of recognition. Below this threshold, the algorithm considers the detected hand movement/hand pose as not matching the predetermined one, and the recognition step is not satisfied, leaving the audio intake disabled.

It goes without saying that other hand movements and/or hand poses may be pre-programmed with different consequences (e.g. increasing/decreasing volume, changing a focus in the video conference to another feed, etc.).

The predetermined hand movement or hand pose is stored in a database, accessible to a server through which the video conference is held, and is compared with detected hand movements or hand poses. Preferably, layered detection methods are applied to reduce the computational burden and not to constantly perform comparisons with captured images from the image sensors. In this way, one can initially look for hands with sufficient visibility, and with further conditions such as whether or not they are stationary, depending on whether the predetermined trigger is a movement or a pose. Only if these conditions are met will the comparison be made with the predetermined hand movement or hand pose. Further conditions may be imposed, such as a minimum period of stationarity of the hand(s).

In certain embodiments, multiple microphones are provided in or around the sterile space, for example on one or more medical (imaging) instruments participating in the video conference.

In a preferred embodiment, the microphone is integrated into the wearable, hands-free image sensor, and the microphone and the wearable, hands-free image sensor preferably form part of smart glasses.

9 BE2022/5516

The use of smartglasses in operating rooms is increasingly on the rise, given the enormous benefits associated with this. For remote users, the operation can be viewed almost from the point of view of a surgeon or other medical user, which greatly alleviates perspective problems. In addition, smartglasses can be equipped with multiple cameras, a built-in microphone, sound system, etc. to enable all communication options to and from remote users.

In a preferred embodiment, the second image sensor is provided on or on a screen in the sterile room, wherein the screen displays at least part of a visual presentation of the video conference.

By explicitly providing the second image sensor on the screen in the sterile room that participates in the video conference, it is easier for the user on site to show a clearly visible hand gesture towards the second image sensor, which is stationary, and often at a strategically placed.

This is preferably directed towards the zone where the patient is typically positioned.

In further preferred embodiments, multiple second image sensors can be provided, spread across the sterile space, preferably redirected to a common area where a patient is typically positioned, so that the medical user can be captured from multiple perspectives and it is easier to navigate to at least one of the second image sensors to indicate the hand movement or hand pose, without the medical user having to drastically adjust his or her position, or having to focus specifically on a single second image sensor. Moreover, in this way it is also possible to receive multiple confirmation of recognizing the hand movement or hand pose, from multiple perspectives.

In a preferred embodiment, the method comprises the following steps: - detecting a hand movement or hand pose with the wearable, hands-free image sensor; and - recognizing a predetermined hand movement or predetermined hand pose in the hand movement or hand pose detected by the portable, hands-free image sensor;

10 BE2022/5516 whereby the audio intake is switched on upon recognition of the predetermined hand movement or the predetermined hand pose in the hand movement or hand pose detected by the portable, hands-free image sensor

In addition, the wearable image sensor can also be used to detect the hand gesture, and based on the images from the wearable image sensor, recognize it and enable the audio input. This is particularly useful for situations where the medical user cannot position himself in such a way that the second image sensor can detect the hand gesture.

In a preferred embodiment, the method comprises the following steps: - detecting a hand movement or hand pose with the portable, hands-free image sensor; and - recognizing a predetermined hand movement or predetermined hand pose in the hand movement or hand pose detected by the portable, hands-free image sensor; wherein the audio intake is switched on upon substantially simultaneous recognition of the predetermined hand movement or the predetermined hand pose in the hand movement or hand pose detected by the portable, hands-free image sensor and in the hand movement or hand pose detected by the second image sensor.

The above embodiment offers a higher degree of certainty that switching on is desired, by requiring that the detected hand movement or hand pose is simultaneously observed in the second and the hands-free image sensor. In this way, it is more likely that the hand gesture is a conscious action by the user on site, since the hand gesture must be performed specifically in the field of view of both image sensors. The additional condition also typically ensures that the hand gesture is looked at from two different perspectives, which significantly increases the reliability of the match between the predetermined hand movement or hand pose and the detected version of it.

In a preferred embodiment, a sound system is provided in or near the sterile room, for playing audio in the sterile room, wherein the sound system is coupled into the video conference for playing audio from the video conference.

11 BE2022/5516

The use of the sound system in the video conference provides a number of advantages in terms of communication, so that everyone present in and around the sterile room can be communicated with, but on the other hand it results in the aforementioned complication that remote users can use instruments in and around the sterile room via voice commands. could control spaces. As mentioned, by disabling audio intake from the microphone in the sterile room, this can be made impossible, with the provision of the invention to enable it hands-free again.

In a further preferred embodiment, switching on the audio intake via the microphone cannot be activated via voice command or voice recognition.

As indicated, in many situations there are sound systems present in the sterile room that participate in the video conference. To prevent remote users from giving voice commands to instruments and devices in the sterile room, voice commands can be disabled by definition.

In a preferred embodiment, the portable, hands-free image sensor is controllable via voice commands when audio intake via the microphone is enabled.

In certain variations, voice commands are allowed, but only if audio intake via the microphone is enabled. This offers the advantage, among other things, that remote users can control the local instruments if a sound system is present (which participates in the video conference), and can thus give commands on site via the microphone. On the other hand, in the absence of a sound system in the sterile room, it is ensured that remote users cannot control the local instruments, which offers advantages in terms of control of the on-site personnel, so that the instruments cannot receive conflicting or unwanted commands.

In a further preferred embodiment, the method comprises the following steps: - sharing additional audio input from one or more of the remote users in the video conference; and

12 BE2022/5516 - controlling the portable, hands-free image sensor via voice command or voice recognition applied to the additional audio input, preferably whereby controlling the image sensor via the additional audio input is only possible when audio intake via the microphone is enabled is.

In a further preferred embodiment, controlling the image sensor comprises at least manipulating focus and contrast of the image sensor.

For a remote user, the feed from the image sensors in the sterile room (can in principle be a hands-free, wearable image sensor, or a second image sensor) is the main way to view what is happening in the sterile room.

By offering remote users the option to control the image sensor via voice command, they are given more options to interact with the procedure in the sterile room, but the user on site can still deactivate this if desired.

Important parameters that can be manipulated for the image sensors include contrast, focus, but also zoom, additional lighting, or even orientation (e.g. second image sensor on a pivotable base).

In a preferred embodiment, at least one additional video input is shared in the video conference, the additional video input coming from a medical imaging instrument.

Video input from medical imaging instruments in the video conference can provide highly relevant additional information to the remote users. They can also use this information much more dynamically than the person on site, who typically has to work hands-free, while remote users can edit the information, provide annotations, etc.

In a preferred embodiment, audio input and/or video input from the portable, hands-free image sensor and the remote users are exchanged via an intermediary server between the medical user and the remote users during the video conference, preferably via a selective forwarding unit ( SFU) server. The recognition of the predetermined hand movement or the predetermined hand pose takes place on the intermediate server, on the basis of a pre-programmed signal database comprising hand movements and/or

13 BE2022/5516 hand poses and associated actions, where the actions involve manipulating the audio input and/or video input in the video conference.

The use of a pre-programmed signal database also allows other hand movements and hand poses to be recorded and actions to be associated with them.

In addition, providing an intermediary server is also much cheaper in terms of bandwidth used. The problems with peer-to-peer communication start with multi-party conversations. In a multi-party scenario, each participant must send his or her data to all other participants. If we assume that there are n participants in the call, the same data stream must be sent n-1 times to the n-1 participants. This requires a significant amount of bandwidth. In addition, there is also a significant computational cost for each client device as it must encode the same stream multiple times. In practice, direct peer-to-peer communication works well when the number of conversation participants is low.

A server, preferably a central SFU server, is therefore essential here to be able to connect multiple displays to the person's portable device in the sterile environment in an efficient manner. The SFU server allows the data stream of one participant out of n participants to be sent to the SFU server only once, after which the SFU server delivers this information to the n-1 other participants. Adding an extra screen is extremely efficient, because the SFU server itself can control the amount of data so that the bandwidth is not exceeded. The risk of bandwidth shortage is therefore much smaller, making the system more reliable for users.

In a preferred embodiment, the wearable, hands-free image sensor is integrated into smart glasses, the smart glasses further comprising a display, wherein the display of the smart glasses displays the video conference.

In a preferred embodiment, the recognition of the predetermined hand movement or the predetermined hand pose is based on comparing the detected hand movement or hand pose with a hand movement or hand pose model that is a three-dimensional model and/or includes at least two different perspective views on a three-dimensional hand movement or hand pose.

14 BE2022/5516

By working with different perspectives and/or a 3D model for recognition, it is ensured that recognition occurs independently of the user's orientation on site. In this way, hand poses and hand movements can also be recognized from the back, without forcing the user to reorientate himself or force the hand into strange bends.

In addition, it can be ensured that a certain weight is given to certain perspectives, and/or certain perspectives are excluded, so that incorrect recognition is not made on the basis of a very generic view (e.g. the image of the side of the hand with little pose in it). detectable can easily match for a large number of variations).

In some embodiments, multiple second image sensors may be provided in the sterile space, typically all directed toward a central, common area (operating table, for example). By working with multiple second image sensors, it can be ensured that at least one has a good view of the user's hand movement/hand pose on site, which can be matched to a perspective (of the three-dimensional model), again without having to place the user on site. forcing him or her to keep in mind the orientation of his or her hand during the movement or pose. In this way, multiple perspectives can also be used to have a higher degree of certainty in recognition, so that the required threshold for confirmation of recognition is achieved. By also giving predetermined weights to the perspectives, it can be decided more objectively whether the degree of certainty of recognition is sufficient to switch the audio intake back on. For example, a metric can be drawn up in which a threshold certainty , and for perspective M this is Zu. If Yn'Zn + Ym'Zm = X, the system considers the hand movement or hand pose as recognized, and audio intake is enabled. If additional perspectives are available, these can of course also be included in the above calculation, for example with different X values, depending on the number of perspectives (preferably X does not increase linearly with the number of perspectives).

In a further preferred embodiment, the method comprises the following steps:

15 BE2022/5516 - detecting a hand movement or hand pose with the portable, hands-free image sensor; and - recognizing a predetermined hand movement or predetermined hand pose in the hand movement or hand pose detected by the wearable, hands-free image sensor.

The audio intake is activated upon substantially simultaneous recognition of the predetermined hand movement or the predetermined hand pose in the hand movement or hand pose detected by the portable, hands-free image sensor and in the hand movement or hand pose detected by the second image sensor.

The audio intake is activated upon substantially simultaneous recognition of the predetermined hand movement or the predetermined hand pose in the hand movement or hand pose detected by the portable, hands-free image sensor and in the hand movement or hand pose detected by the second image sensor. Here, the hand movement or hand pose detected by the wearable, hands-free image sensor corresponds to a first perspective view of the hand movement or hand pose model, and the hand movement or hand pose detected by the second image sensor corresponds to a second perspective view of the hand movement or hand pose model, different from the first perspective view.

The use of images from the hands-free image sensor to further ensure recognition can be controlled with a specific model, as described above, so that a threshold value must be reached in aggregate in terms of recognition to confirm this.

In a preferred embodiment, the remote users can provide additional visual output to the graphical presentation. The remote users typically participate in the video conference via an electronic device with a display and input mechanisms, such as touchscreen, keyboard, mouse, joystick and/or others, and can provide annotations on the graphical presentation via the input mechanisms.

On the other hand, remote users can provide additional visual output, such as by sharing a screen of their PC, laptop or other, sharing a file, and/or via their own image sensor (webcam and the like).

In a preferred embodiment, the instruments whose output is shared in the video conference share the output via a wireless connection, preferably via Wi-Fi. Preferably a WAP or wireless access point is used for this purpose

16 BE2022/5516 provide the sterile environment, through which the instruments can share their output via Wi-Fi directly with the video conference participants. This avoids the need for the output of the instruments to pass through a local server.

In a preferred embodiment, remote users can provide augmented reality content in the shared visual output for marking objects and/or zones in the visual output, whereby said augmented reality content is anchored to the marked object and/or zone, and upon movement of the marked object and/or zone, the position of the augmented reality content is adjusted based on said movement.

By annotating certain images with augmented reality content (hereinafter referred to as annotation), remote users can choose that these annotations are anchored to a certain position or object in the image (for example, location of or in front of an incision), and are not static ( i.e. anchored on a particular pixel).

The system is configured to store the location/position of the annotation, with all relevant data stored there. The aim is that, when the position of the image sensor that captures the annotated image changes, the relative position is maintained, and the annotation therefore moves as the image moves. This ensures that with less stable image sensors, the annotation retains its desired meaning or purpose.

Adjusting the position of the augmented reality content in the displayed visual output can be achieved in several ways. On the one hand, this can be done by using gyroscopes and similar systems, which can determine the change in orientation/position of the image sensor, and on this basis the position of the augmented reality content can be recalculated. Alternatively or additionally, object recognition can also be opted for. By taking the area around it, or even the entire image into account when placing the annotation, this can be compared with images from new viewpoints, and based on this the position of the annotation can be updated.

In particular, the combination of the above two can be useful, whereby object recognition is preferably used as fine-tuning over the 'more cumbersome' recalculation based on determination of adjusted orientation/position via gyroscopes and related sensors.

The augmented reality content can take many forms, such as text, characters, a sequence of images, etc. One or more different annotations can be anchored to an object/position, while additionally non-anchored

17 BE2022/5516 annotations can be added (which are given a fixed position in the display of the image, for example for general comments).

In what follows, the invention is described by means of: non-limiting examples that illustrate the invention and are not intended or should be construed to limit the scope of the invention.

EXAMPLES

Figure 1 shows a schematic configuration of the actors in a video conference.

An operating table (120) is centrally positioned in or around the sterile space (100), in this case an operating room. A medical user such as a surgeon (110) is present, wearing smartglasses (112) that include a wearable, hands-free image sensor. The smartglasses (112) share and receive data in the video conference.

In addition, the sterile room (100) includes a number of medical instruments (140) that participate in the video conference and share therein, and optionally also receive, a certain data output (video, audio and/or other). In addition, a number of second image sensors (130, 130') are provided in the sterile room (100), oriented with their field of view (130, 130') on the operating table (120) in this case, in order to detect movements and/or to accommodate poses of the hand (111) of the medical user (110). A sound system (150) is also provided.

The aforementioned devices (112, 130, 130', 140, 150) share and/or receive data in the video conference via an intermediate server (200), through which remote users (301, 302, 303) can also share and receive data in the video conference. video conference.

In many cases, single-sided arrows can also become double-sided, if data (e.g. instructions) is shared from the video conference to the devices.

It is of course possible that additional medical instruments or other types of equipment are present in the sterile room (100) that do not participate in the video conference. However, these are not shown in the Figure.

Figure 2 shows a schematic representation of the graphical presentation visible to a remote user participating in a video conference.

The graphical presentation (1) is displayed on a screen of an electronic device of the remote user, for example a laptop, tablet or PC, equipped with a mouse, joystick, keyboard, touchscreen, or other such

18 BE2022/5516 input mechanisms. The presentation (1) can be maximized or displayed in a different format on the screen.

The graphical presentation (1) includes a number of sections (2a, 2b), including an enlarged section (2a), and a number of 'normal' sections (2b). Each of the sections (2a, 2b) represents a remote user, whether or not visibly displayed via its own image sensor or an imaging instrument.

In this case, the enlarged section (2a) displays the visual output of a zoom lens image sensor on the user's portable device on site, as can also be seen in a label (3) in said enlarged section (2a). The other sections (2b) also have such labels (3). In the graphical presentation (1), an anesthetist is present here, visible in the label (3) of the upper section (2b), and the visual output from a central camera on the user's hands-free device on site. Finally, the visual output of an ultrasound device is also available via the video conference in the lower section (2b).

The remote user (anesthetist in this case) can choose which visual output he wants to visualize in the enlarged section (2a), for example by touch interaction with the section that displays the visual output of the central camera.

The graphical presentation (1) also provides an action toolbar (4) for controlling or manipulating the shared visual output, especially that in the enlarged section (2a). This is often made possible via icons (4a-4g) in the toolbar, which can be set according to the wishes of the users.

In this case, the toolbar (4) has the possibility of controlling a lamp (4a) on the image sensor of the user on site, he can maximize the graphical presentation in his screen (4b), he can take a photo (4c) of the image in the enlarged section (2a), the audio level of the video conference can be adjusted (4d), the battery life of the user's image sensor is displayed on site (4e), settings can be changed (4f), the connection strength can also be displayed (4g), and finally annotations can be provided, both static (4h) and anchored (4). It should of course be realized that the foregoing is just an example, and that additional functionalities can be provided, such as adjusting contrast, zooming in, adjusting the orientation of the image sensor, etc.

19 BE2022/5516

Finally, the enlarged section (2a) has two annotated shapes, an anchored one (5a), which will move with the area on which it is mapped, and a stationary one (5b), which will be substantially in the same position in the enlarged section ( 2a) will remain.

It is understood that the present invention is not limited to the embodiments described above and that some modifications or changes may be added to the described examples without revising the appended claims. For example, the present invention has been described with reference to medical procedures, but it should be understood that the invention may be applied to e.g. hazardous operations, such as dismantling explosives or maintenance in high radioactivity environments, maintenance of deep-sea systems, difficult-to-access systems or equipment and others. In each of these situations, the expertise of several people would be extremely beneficial, but this is either impossible or too high an additional risk.

Claims (15)

20 BE2022/5516 CONCLUSIONS 1. Method for managing a video conference between a medical user located in a sterile room, preferably during medical treatment in the sterile room, and one or more remote users, wherein the user in the sterile room has a portable, hands-free image sensor, preferably on the head, and preferably the image sensor forms part of smart glasses, wherein a second, preferably stationary, image sensor is provided in the sterile space, the second image sensor participates in the video conference, and where a microphone is provided in the sterile room, preferably on the portable hands-free image sensor, the microphone participating in the video conference, the method includes the following steps: a. sharing video input from the wearable, hands-free image sensor in the video conference; b. sharing audio input from the microphone in the video conference; c. optionally sharing additional video and/or audio input from one or more of the remote users in the video conference; d. disabling audio input via the microphone or disabling audio input via the microphone, whereby the audio input from the microphone is not shared over the video conference when the audio input via the microphone is disabled; e. detecting a hand movement or hand pose with the second image sensor; f. recognizing a predetermined hand movement or predetermined hand pose in the hand movement or hand pose detected by the second image sensor; g. switching on the audio input via the microphone upon recognition of the predetermined hand movement or the predetermined hand pose with the second image sensor, whereby the audio input from the microphone is shared over the video conference when the audio input via the microphone is switched on. 2. Method according to the preceding claim 1, wherein the microphone is integrated into the portable, hands-free image sensor, and the microphone and the portable, hands-free image sensor preferably form part of smart glasses. 21 BE2022/5516 3. Method according to one of the preceding claims 1 or 2, wherein the second image sensor is provided on or on a screen in the sterile room, wherein the screen displays at least part of a visual presentation of the video conference. 4. Method according to any of the preceding claims 1 to 3, comprising a step of detecting a hand movement or hand pose with the wearable, hands-free image sensor, and a step of recognizing a predetermined hand movement or predetermined hand pose in the position determined by the hand movement or hand pose detected by the portable, hands-free image sensor, whereby the audio intake is switched on upon recognition of the predetermined hand movement or the predetermined hand pose in the hand movement or hand pose detected by the portable, hands-free image sensor. 5. Method according to any of the preceding claims 1 to 4, comprising a step of detecting a hand movement or hand pose with the wearable, hands-free image sensor, and a step of recognizing a predetermined hand movement or predetermined hand pose in the position determined by the portable, hands-free image sensor detected hand movement or hand pose, whereby the audio intake is switched on upon substantially simultaneous recognition of the predetermined hand movement or the predetermined hand pose in the hand movement or hand pose detected by the portable, hands-free image sensor and in the hand movement or hand pose detected by the second image sensor hand movement or hand pose. 6. Method according to any of the preceding claims 1 to 5, wherein a sound system is provided in or near the sterile room for playing audio in the sterile room, wherein the sound system is linked in the video conference for playing audio from the video conference. 7. Method according to the preceding claim 6, wherein switching on the audio intake via the microphone cannot be activated via voice command or voice recognition. 22 BE2022/5516 8. Method according to any of the preceding claims 1 to 7, wherein the portable, hands-free image sensor can be controlled via voice commands when audio intake via the microphone is enabled. A method according to the preceding claim 8, comprising a step of sharing additional audio input from one or more of the remote users in the video conference, and comprising a step of controlling the portable, hands-free image sensor via voice command or voice recognition applied on the additional audio input, preferably whereby controlling the image sensor via the additional audio input is only possible when audio intake via the microphone is enabled. 10. Method according to one of the preceding claims 8 or 9, wherein controlling the image sensor comprises at least manipulating focus and contrast of the image sensor. Method according to any one of the preceding claims 1 to 10, wherein at least one additional video input is shared in the video conference, the additional video input originating from a medical imaging instrument. 12. Method according to any of the preceding claims 1 to 11, wherein audio input and/or video input from the portable, hands-free image sensor and the remote users are exchanged via an intermediary server between the medical user and the remote users during the video conference, preferably via a selective forwarding unit (SFU) server, and where the recognition of the predetermined hand movement or the predetermined hand pose takes place on the intermediary server, on the basis of a pre-programmed signal database comprising hand movements and/or hand poses and associated actions , where the actions involve manipulating the audio input and/or video input in the video conference. Method according to any of the preceding claims 1 to 12, wherein the wearable, hands-free image sensor is integrated in smart glasses, further comprising a screen, wherein the screen of the smart glasses displays the video conference. 23 BE2022/5516 14. Method according to any of the preceding claims 1 to 13, wherein the recognition of the predetermined hand movement or the predetermined hand pose takes place on the basis of comparing the detected hand movement or hand pose with a hand movement or hand pose model that has been compared with a three-dimensional model. and/or includes at least two different perspective views of a three-dimensional hand movement or hand pose. 15. Method according to the preceding claims 5 and 14, wherein the audio intake is switched on upon substantially simultaneous recognition of the predetermined hand movement or the predetermined hand pose in the hand movement or hand pose detected by the portable, hands-free image sensor and in the hand movement or hand pose detected by the second image sensor-detected hand movement or hand pose, wherein the hand movement or hand pose detected by the wearable, hands-free image sensor corresponds to a first perspective view of the hand movement or hand pose model, and the hand movement or hand pose detected by the second image sensor corresponds to a second perspective view of the hand movement or hand pose hand pose model, different from the first perspective view.