Split-tunnel Teams media: make it right!

With the pandemic and the new work culture, work-from-home and VPNs are here to stay. I already wrote about SfB Split-tunneling, so the same logic applies to Teams: Media should reach the quickest path as possible to the other endpoint.

It’s up to the VPN solutions to configure the clients to use them. Microsoft has enough articles on how to deploy it globally for MS365 and VPN providers.

But due to some appliance limitations or ‘easier-to-deploy’ decisions, you might find cases were the split-tunnel rules based on application or process name. If you decide to allow split-tunnel for the ‘teams.exe’ processes you are allowing it to use the local internet breakout for any public IP instead of using the companies firewall inspector to protect the user from malicious sites.

Not a problem? let’s review Teams.

Teams is not just a communication app, but also allows you to embed other plugins and apps. But most important: it’s an ‘Electron based browser’. To make it simple and quick to understand here’s the ‘catch’:

1. Go to a Teams channel and click on the (+) sign of the Tab
2. Choose Website, type a name and a the URL website you want to open
3. You will now be able to navigate that site within the teams ‘embedded browser’ and directly on the Internet (since it’s allowed to bypass) without any corporate restrictions

This will also happens with any Teams app that can be attached to Teams

How bad can it be? Here’s the proof of concept:

1. using a normal web browser in you company you will obsviously get blocked when reaching a malicious web site, because you use your company internet breakout/firewall through the VPN tunnel.
2. If I embed the same site on a teams.exe-vpn-bypass web page, you I will be able to navigate through it.
And I don’t think that this chromium browser has too many security measures implemented like the standard browsers.

Risk: moderate

It’s not something that users will know how to do, but they can be deceived by someone who can craft a way to use ‘teams.exe’ to open a site or download a file (we still have the local EDR or AV to protect, right?). The other concern here is that users are able to access content that would be required to be inspected by a firewall / content filter (example: if someone invites the user to another Tenant that has an excel file or any other file hosted on sharepoint that contains a macro or other code)

Teams VPN media optimization: the right way

If you you want to optimize Teams media in a secure way, the simplest way is to create a VPN client rule allowing split tunnel for the Teams media gateways IP’s and ports on list #11 of the office MS documentation:
https://docs.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-worldwide#skype-for-business-online-and-microsoft-teams

You shall not migrate to Teams!!

So… you spend the last months planning and making a Pilot migration from SfB to Teams for a customer. 48 hours before the big day you think you have all ready until the moment you try the most infamous command to migrate SfB users to Teams…

Move-CsUser -Target sipfed.online.lync.com -BypassAudioConferencingCheck -UseOAuth -Verbose -userlist “someTextFile with user sip address”

Move-CsUser : HostedMigration fault: Error=(523), Description=(Moving users between your on-premises deployment and Teams now requires a newer version now that Skype for Business Online has reached end of life. To move users between Skype for Business Server and Teams, you must first upgrade to Skype for Business Server 2015 Cumulative Update 12 (KB3061064) or later.)

So…. what happened here?!

For this one you will not find an announcement. You will have to pay attention to MS documentation updates:

1. On July 2022 this ‘Move users’ documentation Prerequisites was updated

yeap. You might need these CU now

2. But what really triggered that message this and other issues, is (not quite) explained on this SfB Online retirement documentation updated 22 days ago with:

the ‘Implications’ link is wrong. Moved to here

Basically MS is silently removing the underlying SfBO infrastructure and replacing by something else for both hybrid and migration scenarios. Which means that you might have or not (yet) the issues.

Going back to the initial part of the text: you have now to update your 12 SfB server infrastructure in the next 48 hours or postpone the migration plan… or not 😉

Practical solution

In respect of all SfB engineers that invest years of their life support the best MS UCC solution, I will post here my solution after several loooong hours of work without getting into details of try/error messages.

1. Get a Windows 2016 domain server with internet access
Reason: it already include Powershell 5.1 required to SfB and Teams PS

2. Check if you have .Net Framework 4.7.2 or later. If not, install it.
Reason: you will need for SfB installation and later for Teams PS.

3. Get the SfB install ISO, and run the Install Local Configuration Store. This will install the SQL service RTCLOCAL and import the CMS to it.
Reason: the move-csuser requires the CMS configuration, but it looks only on the RTCLOCAL local database of the server that it run on.

4. Download the latest SfB CU depending on the version that you have, run and update the components

getting the most update-to-date move-csuser command

5. Install Teams Powershell
Reason: If you try to run the move-csuser it will complain about the lack of Teams PS

6. And finally… you can run the command right?
Not quite! if you already had an Hybrid setup made before MS “SfBO cleanup” you will get an error

Move-CsUser : {“code”:”Forbidden”,”message”:”Access Denied.”,”action”:”Provide different credential or request access.”}

You will need to use this undocumented parameter ‘UseLegacyMode’

The final SfB migration user command line should look like this:

Move-CsUser -Target sipfed.online.lync.com -MoveToTeams -BypassAudioConferencingCheck -Verbose -userlist “.\$Migwave.txt” -UseLegacyMode

Hard-core solution

I would not recommend this because it has some unexpected implications specially if you need to rollback the update.
But if you are in ‘panic-mode/need-to-migrate-this-right-now’. On the Front-end server that you want want to run the move-csuser

1. Check if you have .Net Framework 4.7.2 or later and Powershell 5.1 or later. If not, install them.
The server will required a reboot, so let’s hope that your SfB pool will not misbehave.
2. Download the latest SfB CU. Extract the contents of the package using the following command:

“SkypeServerUpdateInstaller.exe” /extractall

It will extract the contents to a subfolder named ‘Extracted’. You will need to manually update to components using these packages: ‘OcsCore.msp’ and (maybe) ‘UcmaRuntime.msp’

From the explorer you might get errors, so the command line is more efficient:
msiexec.exe /update “OcsCore.msp” /passive /norestart
msiexec.exe /update “UcmaRuntime.msp” /passive /norestart

4. Install Teams Powershell

Final thoughts

MS seems to be too much silent regarding they back-end cleanup actions and not informing customers and their own support about it. So far, I’ve faced another related case:

• SfB on-premises users could not reply directly to chats initiated by Teams external tenants
  MS SfB support confirmed to by a routing issue inside Teams for hybrid tenants stripped out of SfBO
  (Teams support refused to admit that it was their problem and closed the case on their side)

New licensing requirements for Teams resource accounts (AA/CQ)

Just noticed this MS document updated 2 days ago.
Before you only needed to assign Virtual Users licensing to resource accounts that: (1) needed to have a phone number, (2) needed to transfer calls to PSTN number.

They even mention that twice (!):

“Each resource account must be assigned a Microsoft Teams Phone Resource Account license to ensure they’re correctly identified by the system and properly function, regardless of whether the resource account will be assigned a telephone number“

….

Previously, a Teams Phone Resource Account license (once known as a Virtual user license) was only required when assigning a phone number to a resource account. Now, all resource accounts must be assigned a Teams Phone Resource Account license, regardless of whether they’ll be assigned a phone number or not“

Would this mean that AA/CQ that have assigned an unlicensed RA, will stop working ?

I guess MS really wants to start charging customers for this service

When Microsoft kicks you in the balls…

Sooner or later I would find me writing an article like this.
Many of you would say that this is about shamming MS, but ultimately this is my attempt to reach someone on MS that would listen and care with his customers and provide a better service.

Well,…. it an accumulate of small and medium issues that haunt Teams telephony administrators like me, this and escalated on last Friday.

On the 24th April 12:52 CET, MS started hiding the last 3 digits of all CDRs. Yes! on my GraphAPI based scripts that collect the Direct Routing CDRs
and also on the TAC PSTN usage reports

Above is the CDR that shows the exact moment that MS Teams started masking the last 3 digits. I changed the real numbers to protect privacy, but the MS ‘***’ are real.

Online you will not see the last 3 digits, and if you click on the export icon you will get the CSV files with the numbers masked.

I could not find any announcement of this action on the usual channels (just like on February that the data from the startDateTime was swapped with the inviteDateTime) and not even a reason to hide them now.

What is my (customers) problem with this ‘hidden number’ feature? Well, let think about some examples:

• Troubleshooting: how can you track down the number that called or was called? let’s think about a company that has a 1000 number block. To exactly whom did the call came from?
• Auditing/security: what exact number made that threatening call?
• Billing: how to charge a client support based on the number of support hours on the phone? and would you pay your Telecom company if the detailed invoice was hiding the numbers?

Solution(?)

What now? The usual:

• Open a tickets at MS we-almost-care Support. Still waiting for the office in India to reopen since there is no escalation/around the clock option. No feedback so far. Maybe because I could not send a recording (PSR) of the issue…
• Report on the MS365 admin center. Right! the one that closes automatically after 2 hours?
  
  

Time to ‘think out-of-the-box’!
If Teams CDR hides the last 3 digits of the callerID:

• let’s go to the SBC and add 3 extra digits to every inbound callerID before sending to MS…
• …MS Teams servers will record on the CDR these extra-long number…
• …and then we strip-it from the inbound call before it gets delivered to the users

Does it work? You still need to find a way to clean those extra ‘***’ from the CDRs,
but the numbers are back 🙂

And this is how MS made me lost my weekend!

Final thoughts

Teams telephony is still on is early stages running still in hybrid supported by SfB back-end servers, but I was expecting much more after this time. Let’s hope that Azure Communications would provide a new approach and more stable services.

My advice to all companies whose business rely on Telecommunications and plan to move to Teams is:

• Clearly identify all your critical needs and perform fully pilot tests
• Find a specialized partner on Telephony that can manage this components and SBC for you.
  Years of experience in “Teams”, on-boarding and migration doesn’t mean that some know how to handle the VoIP part.
  This quick workaround for this issue was not possible without this premise.

Updates

27.04.2022 – Opened an Incident asking MS why this happened

02.05.2022 – MS just updated today the PSTN reports documentation with a number obfuscation section for several countries. There are no links for the countries legal enforcement. And it’s still a total mess:

1. I am in Switzerland and I get 3 digits masked and not 4 as documented.
2. If you make a call forward or get calls to Call Queues, it will mask the callee (your own number) and keep the callerID (the external number that the call is forward)

09.05.2022 – MS support sent me the link of the number obfuscation. I asked why I was getting 3 digits masked, when the document mentions that Switzerland is masked with 4 digits

2022-05-10T09:36:00.055Z – MS just started today to obfuscate the CDRs in CH to 4 digits making it consistent with the documentation published 8 days ago.

Still not feedback from support regarding when was this change decided,announced and the legal references of the countries explaining that… still didn’t asked why it gets done in PROD ‘on-the-fly’.

ucCSI case: The ‘unmanageable Team’

When doing today a very simple task of adding a user to a Team, I could not find it on TAC

And using Teams Powershell I could not also find it

Alert mode: did someone delete it ?! No because:

1. You could locate the corresponding MS365 group on Azure portal and actually add the required member

2. And the users could access on their Teams clients and work on it.

So what is happening here?!

Investigation the cause

The hint came from the Office Admin Portal. The Group Teams status was blank!

Back again to powershell, a quick comparison with another Teams enabled group revealed the ‘glitch’. All the information for sharepoint, exchange, and naming where there, but the MS365 group association with Teams was missing. The field ResourceProvisioningOptions was empty (?)

Cause

Unknown. And I was not going to open a support case at MS for the ‘we-all-know’ reason !!

The solution

Obviously the Admin Portal option to create a Team would not be a good idea as you would not be sure if it will not just mix-up all with a new content and create a bigger problem

So let’s see if you can fix this using MSgraph:
DISCLAIMER: Do not do this by your own if you are not experienced with the GraphAPI calls

1. Using the graph explorer, set the method to PATCH and the URL of the MS365 group
   https://graph.microsoft.com/v1.0/groups/<groupID>/
2. On the Request body, set the field with the missing value with the following JSON
   {“resourceProvisioningOptions”: [ “Team”]}
3. You should get a 204 response which means that you were able to set it

And after a few seconds everything is back to normal 🙂

Teams resource accounts affected by MS back-end operations?

This is a fast-pace writing post to help those having the same issue as me and others that already confirmed. Sorry for my sentence construction.

Issue and symptoms

PSTN inbound calls to some Call Queues, Auto attendants or Bots are failing with ‘number unreachable’ or busy signal. There is nothing wrong with them, but with their associated resource accounts. You check that they have the number correctly associated.

How to check if you are affected?

• If you have access to the SBC, you can see for the call to your number a ‘404 Not Found’ response from MS Teams with the reason code ending in RNL.

• (possible) an error when trying to assigned the numbers to the resource account ‘user not authorized to perform the operation for the user <sid> in AADGraph API’

• But the most efficient way is to download the call CDRs from MS and look for the exact error. Check the sections ‘Actions and workarounds’

Possible cause

Some back-end action done by MS on the last two weeks has affected the BVD (Business Voice Directory) sync/association with the AAD (resource) accounts used by Teams .

On the CDR logs of an affected Call Queue that was not changed for the last months we could see different error messages until it stayed in ‘number not found’

Additionally some 3rd party providers that uses Bots (like Contact Center solutions) are instructing their partners and customers for urgent action and updates.

Actions and workarounds

Here’s what you can do. You need to download the CDRs from MS:

1. On Teams Admin Center go to the Usage Reports
2. Choose the report ‘PSTN and SMS (preview) usage’, and the timeline that cover the call that you know that failed
3. Click on ‘Run report’
4. Click on the ‘Export to Excel’ to download a zip file

5. Extract the 2 CSV files from the downloaded zip one is for Direct Routing and the other for Teams Calling records and open them on excel.

Now you can locate the call to your number and see the error details on the Field ‘Final SIP Phrase’

Known error messages and what you can do:

• “Getting user info by number from BVD failed” or “Getting user info by number from RuntimeAPI failed”
  You are in luck. Assigned another number to the resource account, wait ~15 minutes and assign the right number back. In another 15 minutes you should get the incoming calls working again
  
• “Getting user info by number from BVD failed – target id is empty Guid”
  Nothing to do. Open a case at Microsoft with the CDR message information
  
• “call to bot is rejected due to null user settings or user app settings”
  Contact your 3rd party solution provider. He knows how to proceed (i hope)

Does a Teams Admin needs a Teams license?

Teams license (MS365 license) is a cost, so we usually try to minimize them. For Teams admin roles we assume that we don’t need them since we are managing it and not using it (maybe if we want to make some tests).

Well… there is at least this case that you will find out but only if you look under the hood.

ISSUE

As you know, you can distribute the meetings backgrounds centrally from the Teams Admin Center meeting policies page:

But for some of you, will might get an error “We can’t get your images. Try again. If you continue to have problems, contact Microsoft customer support” and “We can’t load any data. Try again?”

You can scratch your head, contact MS support or peak inside the HTTP request to find out the real reason of the generic error message:

SOLUTION/WORKAROUND

1. Assign to your Teams Admin user a valid MS365 and/or Teams license (exploratory might work)
2. Wait and you would be able to access the page and manage the backgrounds now
3. (optional) free-up the license back from the Admin account

Additional note: If you read closely the documentation on ‘Use Microsoft Teams administrator roles to manage Teams‘ you will find a small remark below the table “²Microsoft Teams admin center³ Teams administrator account must have a valid Teams license”

Phishing with Teams: train your user’s awareness

Some colleagues and customers call me a security freak because I question all products and deployments on about their protected mechanism. Security is not on my role list, but I simply cannot ignore it. It’s my on my IT-DNA 🙂

We all had training sessions and got tested at our companies for e-mail phishing, social engineering. It is required as it’s the most common attempt for network infiltration.

But what I’ve observed over my many years on the UCC area is the lack of security concerns regarding the other collaborations tools. They can be also used by to users … and more easier and quicker than an email phishing. It has the same attack vectors as e-mail phishing: Social engineering, sending instant messages with malicious links or attachments, …

Microsoft has done a impressive job and provides for MS365 customers several built-in protection mechanisms (for both identity and phishing) and many other security tools to keep your users protected as long as you have a well-trained security IT team to manage it. But what about the human-factor?

Using Teams to fish

The example is from Teams, but it can apply to other vendors. It’s simulated and not evolving any other persons than myself

The best way to explain my point is with an example and ‘hacker mindset’

1. Create an Office365 demo tenant (it’s free)
2. Go to LinkedIn, pick an IT company with Teams, look for some engineers with profile pictures
3. Get the names of some of this company customer’s (usually the customer referrals, success stories
4. Email addresses are easy to find or calculate (first name last name, company email domain name)
   and confirm that the person exists and her display name, just by searching with your Teams client
   
   

Skipping the details, we can create an user with the same display name on our ‘phishing tenant’. It’s now time to reach the victim.

Instant message method

Sending an internet link or an file might seem a suspicious approach, so let’s try social engineering by being the helpful support engineer (Luis Ramos AVENIQ) trying to help a customer’s user (RA):

The external user has remote control of the user desktop. From this point now it’s up to your imagination.

On the above case, an attentive user can get suspicious if he’s paying attention to the aveniq.ch address.

But on the other hand, there are companies where the user email address (SMTP) doesn’t need to match his Teams address (SIP).

What about if we just call the user?

The incoming call toast notification will not help the user here:

And neither the main window, so now it’s up to the caller to convince the user:

Now the user needs to leave the mouse on the caller name to check if it’s really him!

Another: the missed call phishing

You actually don’t need to try and wait for the user to answer. Teams will save a missed call with the name for the user be able to call you back 🙂

Final comments

• If by now you are thinking ‘I never thought of that, what should I do from now one?’
  Then my post hit his goal 🙂
• Exploit success factor? depends on how tired and overwhelmed people are with confcalls, unread IM’s, homeoffice ‘burn-out’.
• Don’t get into the attitude: ‘never heard of such thing happening’ or “I will never fall for that!”
• This is an example of a social exploit
  But hjust like any other software, Teams can also be vulnerable to technical exploits overtime. This one got out of the news radar: “We found a dangerous vulnerability in Microsoft Teams, but the company fixed it only after two months”

That’s it!
Have a Happy Christmas and don’t forget to write to Santa. He might have been watching you all year

ucCSI case: The unknown IVR with a random DTMF number

We all love Mondays, but in the middle of the morning we just got a freaky urgent call. Fortunately this was a 2 hour episode and can be quickly explained

Part 1: The issue

The customer has an hybrid Contact Center solution working with Teams. The call flow is the following:

(1) the customers call a 0800 tool free number that goes to our SBCs. If the number matches a customer support number it sends the call to the Contact Center system
(2) the call flow will ring the call to the agent(s) of the queue back throw the SBC (with the real number of the queue) to MS Teams and it rings the agent (on his Teams client) that picks the call

The working call flow

Suddenly, the customer reported that some of the support numbers were getting initial answered by the queue music on hold and then answered by an IVR asking to type one or two numbers. Even more strange:

• The customer support number don’t have IVR with DTMF prompts
• Every time you dialed the 0800, the IVR will ask for other random DTMF numbers (!?)

Part 2: analyzing the evidence and identifying the suspect

After analyzing the SBC call flow logs, we narrowed down the source of the problem and the primary suspect. The calls were all fine until they are delivered to MS Teams and never rang the agent(s)… just this IVR

>> Something inside MS Teams service was intercepting the call !

Part 3: Freeze ! on the ground! now!

I was about to get on my nerves having to open another case on MS nightmare support service, when my ‘partner investigator’ remembered a past case some weeks ago with another customer that started getting calls in Teams with a toast notification with ‘Spam likely’.

After some google-fu, we just went through our Call Policies ‘Get-CsTeamsCallingPolicy‘ and confirmed that we had the default setting ‘SpamFilteringEnabledType‘ to Enabled

What is this parameter for?

A couple of minutes after we ran the command ‘Set-CsTeamsCallingPolicy -SpamFilteringEnabledType “Disabled” –Identity [your users assigned policy]‘ the calls started ringing the agent and all went back to normal.

Epilogue: The confession

The Call Spam filtering was announced on August 13st and quickly went GA by Microsoft around September 2021 (see feature ID: 85386) but it’s not documented how it works and closest official description can be found on the Teams PS command Set-CsTeamsCallingPolicy where you find a mention to an ‘Captcha IVR bot’

Based of the description, parameters naming and some scattered information of other internet posts:

• Microsoft has some internal score system based on the number of call a particular number makes to MS Teams per day / per hour
• After reaching a particular score it will starts displaying a yellow sign with an exclamation mark plus a text label saying “SPAM LIKELY” along with the calls
• When it reached a particular threshold it will start answering with IVR bot challenging the caller that ‘I am not a robot‘ and demands typing the number that it announces

On my customer case:

• We have a significant number of customer calls reaching the Contact center that presents the same queue number to all agents. This will trigger MS score
• The agent probably received the notification but ignore it and did not report to us
• When the IVR kicked-in DTMF is not supported (call flows between PSTN>Contact Center > SBC > Teams so the customer could not get through it to ring an agents
• RESULT: 2 hours without customer service 😠

I guess MS should have:

• included on the announcement “This Spam Call protection capability should help users save time by declining calls that might disturb their workflow” + ‘Disrupt your custom IVR and Contact Center solutions’
• and document how this works before releasing to GA !

ucCSI Case: The mysterious calls to the emergency services

Sometimes there are some SfB reported issues, involve understanding the human factor behind the logs. This case went throw several weekly episodes.

Episode One – The ‘issue’

Our customers SfB has a Contact Center integrated and one day he received a complain from an external associated partner that sometimes they received calls from the Agents and when the pickup the cal….l they were talking with a person from the national emergency services (Switzerland has several emergency numbers an in this case it was the Police).

The first step to be able to know where to search on the logs is to ask for reported situations: when (dates and times), who started (numbers), what happened next?. It was not easy since day one, because it involved at least 2 different calls (one inbound and another outbound), But I managed to identify the flow:
A client calls the SfB/CC number (1), an Agent picks the call (2), he calls the Partner support number (3) and he transfers the call of the client to that partner (4). Except that sometimes, the Agent of that partner instead of hearing the client get surprise by ‘Police ! what is your emergency?’

Episode 2 – Analyzing the facts and recreating the exact steps

The outermost challenge was to track the client call, which Agent picked the call and what did he do next to reach the Partner and transfer. Looking at the logs of several calls made to the partner number, I noticed that some Agents were using DTMF tones.

The Partner number is a Contact Center with an IVR! So now we have a more detailed call flow:
A client calls the SfB/CC number (1), an Agent picks the call (2), he calls the Partner number, goes throw the IVR (3) an Call Center Agent picks the call (4) and the customer Agent transfer the call (5) . Except that sometimes, the Agent of that partner instead of hearing the client get surprise by ‘Police ! what is your emergency?’

Time to include the Partner telephony provider for some potential issue on their side. But after some tests calls there was nothing on the call history the involved calling the authorities and the only active call was coming from our SfB Agent number. The other clue was that the Police that picked the calls is not from the same region as the partner but from our Customer. So the call is definitely triggered from SfB or our operator.

Episode 3 – The clue and the primary suspect

Looking now at the log history of all calls made to that IVR number, one call (that did not triggered the issue) caught my attention on the DTMF pattern. It was typing: 1,1,2. And 112 on a dialpad is…. the well-know emergency number!

Went back to the customer and the Partner and they confirmed that the affected Agents belong to a service that are behind the IVR menu: by dialing the IVR, listen the options and choose 1, then the sub-option 1 and finally the final option 2. There were no reported issues on other queues reached by other IVR menus.

Now we got a primary suspect: What if this DTMF ‘112’ sequence is triggering a call to the emergency services? but how does this is ‘accidentally’ transferred to them?

Episode 4 – Eliminating the suspects

There was still a loose end to clear out. The ‘DTMF 112’ theory issue could have two sources:
– The Customers Agent that we see on the previous logs.
– We noticed that some Agents will transfer the call to the Client before the IVR, and then you will only see the Client call trying to use DTMF.

Time to define a precise script of call testings to identify the one that is causing this mess. Using the minimum amount of variables (same client number to call, same customer agents queues, same partner IVR number:
1. Just to prove the theory: Calls to other IVR options 1,0 or 2,0 all working
2. If the agent transfer the Client that the IVR, we noticed that DTMF over this SfB call transfer doesn’t work (the IVR didn’t recognized the options): this one is excluded
3. Several call transfers using IVR menus 112 were also working until one particular agent call ended on the emergency services.

Time to look deeper to these last call logs and compare them. Nothing looked different, DTMF was ok, except… for the ones that worked, I could find the Client call transfer to the Partner, but the not for the mistery one.
So what happened? The answer involved looking back at all the calls on that time frame (this is a large company with a lot of calls….). And then one peculiar outbound call appear on the SBC log:
– from:<Partner IVR number>
– to :+41112 (+41 is the country prefix of Switzerland)
– referred-by: <Agent number>

Gotcha! the SfB client caused the call. But why and how?

Final episode 5 – The ‘crime scene’ and the perpetrator

We are still waiting for the ‘confession’ but, just by using a SfB Client we can present our delegations using any SfB Client:

1. The customer Agent receives a call from the client and click on ‘Consult’
   
2. The Client call is put on hold. A transfer window will appear were the Agent can search for internal contacts or type the Partner IVR number and click ‘Consult
   
3. A new call window will appear. The Agent has now 2 calls on his SfB: the customer on hold (on the left) and and active call the the Partner IVR. The Agent might not notice this because SfB will open the second call window in exactly in front of the one that is on hold
   1. The agent is now navigating throw the DTMF voice system and picks (type): the options 1,1 and 2 that start appearing on the dialpad text (A)
   2. To transfer the IVR call to the CLient he needs to click on the transfer call on the top right side of the call window (B)    >> if you press this one, SfB will transfer the call of the Client to IVR active call and both windows will close
   3. If he clicks on the ‘Transfer’ button above the dial pad, it will initiate a transfer of the IVR call to another new number (C)
      
      But it will still show the Search window with the numbers that were typed during the DTMF.
      You can clearly see the 112 as the default selected number. If he presses ‘Transfer’ now it will transfer the IVR call to the … Emergency service (112).
      
   4. The IVR call window will close but the Client call window will still there on hold (because there was not transfer to him). But the Agent unaware of his mistake, hangs up the customer call
      
      

End of episode:
…
<playing credits>
….
Disclaimer: no servers or software were harmed or fixed during this investigation

Are you really offline on Teams? (part 1)

I will start backwards by presenting the solution before the ‘glitches’.

Making sure that you are really offline for everyone

This is the traditional: ‘I’m off for today/this week, don’t contact me!’
This includes your colleagues and any external or federated contacts.

Just make sure that you choose ‘Sign out’ from all your Teams clients (Windows, Mobile, Linux, OSX, Browser)

• 

  Sign out on Windows,Linux,OSX or Web Browser client

• 

  On mobile client is a little ‘hidden’ down below on the settings

The users on your Tenant will see you offline and also the federated persons:

Both Teams federated tenant (on the left) and a SfB on-premise federated users will see me Offline and (hopefully) will not contact me

“Duh! we all know that. So why blog about this?”
You will not say this if you already had another user complaining that he doesn’t actually see your status offline. Now it’s time for the real topic… Why am I not offline for others?!

#1 Quit or shutdown might not put you to Offline status

Shutting down, hibernate your computer or specially using the Quit option on Teams sometimes will not trigger the Offline status. This seems to be related to client version or simply delays between the client and servers on the HTTPS sessions over VPN connection and corporate firewalls. Example:

I have my status Online and I clicked on the Teams task bar icon and quit the application…

…but all internal and federated users that have your contact will still see me as Available and will try to contact me

‘Appear offline’ will not work for all

Microsoft recently made the appear offline status available for Teams, which might be useful as a desperate ‘do not disturb me!!’ (might work for some who ignore the standard ‘DnD’).

Let’s test it!
(1) I have my status Available and I switch to ‘Appear Offline’

(2) My colleagues see me offline on Teams, check!

(3) If you still have an Hybrid federation with SfB onpremises, all the SfB user will see you as ‘Available’ or the last presence status that you had before changing to ‘Appear Offline’

(4) Your federated partners that use Teams will (most probably) see you ‘Offline’ but any SfB federated partner will also see your last status before you changed to ‘Appear Offline’

A Teams federated contact will see offline, but a SfB user will not and it will normally try to call you

If you check the SfB client logs and ran a trace on the SfB Edge server you will notice that all presence change status are sent from Teams to SfB except the ‘Appear Offline’.

Final thoughts

Let’s see if MS will find out and solve these two cases soon.

So… Are you sure that you are appearing offline to everyone right now? 🙂

“Skype for Business ends in July 2021. Migrate now to MS Teams!”… or not?

Some of my contacts and older customers have been coming to me because they were getting approached by some MS Partners that they must migrate their SfB infrastructure to Teams because ‘SfB is dead’ and ends in July 2021.

I will not finger-point no one here, but I want to clarify headers and speeches like this. I deliberate wrote my blog title like this but as friendly ‘click-bait’ 😉

Skype for Business Online will be retired on July 31, 2021

That is the correct heading. Only the SfB Online (SfBO) will be discontinued. SfB on-premises will continue to work and get support (much) after 31.07.2021

Where can read the retirement information from MS:
– UPDATE: Skype for Business Online retirement on July 31, 2021
– Skype for Business Online to Be Retired in 2021

If you have a plain and simple SfBO tenant your migration is simple (and probably automatically done by MS).

If you are still on SfBO and MS didn’t take any action, then it could mean that you have some an Hybrid SfB, PSTN or 3rd party connectivity:
“Skype for Business Online will be retired on July 31, 2021 after which the service will no longer be accessible. In addition, PSTN connectivity between your on-premises environment whether through Skype for Business Server or Cloud Connector Edition and Skype for Business Online will no longer be supported” – source

In these cases, you really need to take action and plan to move to Teams …or to a SfB On-premises 🙂

Skype for Business server 2015, 2019 are alive at least for 5 years

If you have a full SfB on-premises you can have support up to 14Oct2025. You can check that on the office MS Product Lifecycle

Current SfB lifecycle product support

Although SfB2015 mainstream support is now over, MS is still releasing updates. But to prevent MS to refuse support, either your extend (pay) it, or you should now upgrade to SfB 2019.

There will be a SfB vNext in 2021/22

Has announced by Microsoft there will be a next version of Skype for Business somewhere to be release in the second semester of 2021:
– (MS) The Next Version of Skype for Business Server
– (UC Today) Microsoft Reveals Fresh Details for Skype for Business Server 2022
– (Tom Talks) Skype for Business Server 2022 confirmed, only be available as a subscription

So, if you are comfortable with your reliable on-premises SfB infrastructure and you want to have full administration control as a critical business service, you know that you can have Skype for Business running until 2028 (?)

How to create Teams Live Events in Switzerland (and other regions)

About 3 weeks ago I posted the explanation why you could not schedule Teams live events in some regions like Switzerland and the options that you have to still use the Live Events: Can I make Teams Live Events in my country?

Now it’s time to show ‘option #3’: Unlock and schedule a Teams Live Event

Under the hood

Before explaining how, let’s give the technical explanation:

1. The Teams clients builds are the same worldwide (the same version is available from Microsoft downloads) and therefore have the same capabilities.
2. This means that for a specific Tenant or Region, the Teams client is being ‘instructed’ to block the ‘Live Event’ feature
   

After some days of detailed inspection on the traffic that the Client gets during the sign-in and loading, this setting caught my attention:

Since my Tenant region is ‘ch’ this could mean that the list membership triggers the client not to allow (better say ‘not show’) the scheduling of Live Events.
So…. let’s confirm the theory.

The setup

Here’s my automating cookbook.
(you could also use similar tools like Fiddler or Postman)

Step 1. Download Charles proxy, install and configure it to proxy and decode HTTPS. There are many tutorials on the internet. Here’s a detailed one.

Step 2. Now that we have a proxy inspector, lets intercept and rewrite those settings for the Teams Client:

^{On the tools menu, select Rewrite}

^{(1) Enable Rewrite, (2) Add a rule and (3) name it.}

^{(4) Add a Location. This is the URL request that we want to intercept.
Since the client version changes over time you might need to adjust the query
(example for versions released on year 2021: 1.0.0.2021??????)}

^{(5) Add a rewrite rule. We want to change the Body of the Response, by replacing the “broadcastSettings.supportedQuickStartRegions”:[ and add the region ‘ch’}

That’s it ! Let’s go for the Proof-of-concept

Unlocking the Teams Live Event

With Charles Proxy running (how will see HTTP traffic running), start your browser with the Teams Web Client (https://teams.microsoft.com) and sign-in. It should also work if you just re-launch your Windows client too as long as it uses the cached credentials.

When the client has loaded, you can look at Charles proxy to confirm that the interception has made is job

^{Hey Teams client! ‘ch’ is a supported Broadcast Region}

Go to the Teams calendar and… schedule a Live Event 😉

• 

• 

• 

• 

Once you have schedule the meeting you don’t need the Charles Proxy. Once it’s schedule you can click it on the calendar and make the adjustments.

Now Producers, Presenters and participant can participate on a Live Event from the Teams client.

^{Live Event Producer Teams Client}

Teams meeting recording button became unavailable

ISSUE

• Some day ago, users started reporting that they could not record a meeting. The record option was unavailable:

^{Start recording button dimmed?}

• The meeting policy for the users allow them to record meetings:

^{… Houston! we have a problem!}

CAUSE

Sooooo… what happened?
Time to investigate what has MS changed recently 🙂

Clue #1: Checking the Microsoft 365 message center
You can find MC222640 ‘Microsoft Teams: meeting recordings saved to OneDrive and SharePoint’ is in progress.
“October 19, 2020 (Complete) – You can enable the Teams Meeting policy to have meeting recordings saved to OneDrive and SharePoint instead of Microsoft Stream (Classic)”
But the rest of the info is not clear about what changes and if you need to take a specific action other than opt in/out to MS to change the recordings to OdB (OneDrive for Business) in 2021.

Clue #2: check the client settings received from the servers for new parameters.
Found a new one inside the callRecording: supportedOdbRegions with a list of countries, with “ch” is where my tenants are

"callingConstants.callRecording": {
    "downloadRecordingExpirationDays": 20,
    "supportedOdbRegions": [
        "ae",
        "ch",
        "de",
        "no",
        "sa",
        "br",
        "sg"
    ]
},

Clue #3: Let’s have a closer look at the meeting policies
Nothing has seem to have changed. Recordings are allowed (AllowCloudRecording) and using MS Stream (RecordingStorageMode), but…

^{… what will that parameter AllowRecordingStorageOutsideRegion=false do?}

SOLUTION

As you know, Stream is not available in all regions like CH, so if AllowRecordingStorageOutsideRegion is getting honored, you cannot record if you are not allowed to save on the region that your account is? Solution(s):

1. RecordingStorageMode=OneDriveForBusiness
   and/or
2. AllowRecordingStorageOutsideRegion=true

My preference goes to #1 using the PS command

Set-CsTeamsMeetingPolicy ‘Global’ -RecordingStorageMode OneDriveForBusiness

Problem solved! you got your recording button back and your recordings are now available on OdB

• 

• 

Final notes and references

Looking at that client settings list, the Tenants on the regions United Arab Emirates (ae), Switzerland (ch), Germany (de), Norway (no), South Africa (sa), Brasil (br) and Singapore (sg) could also experience the same issue.

• Microsoft Official and details Docs
  Use OneDrive for Business and SharePoint or Stream for meeting recordings
• Microsoft 365 Roadmap
  67138 – Microsoft Teams: meeting recordings saved to OneDrive and SharePoint
• Microsoft 365 message center
  Message ID MC222640

Teams ‘Busy on Busy’ causes missed call storm notifications

Recently in my company we decided to rollout the ‘Busy on Busy’ feature for all users, to suppress the annoying ring sound that comes on your headsets when you are on a call (or a meeting) and this makes it difficult to hear others.
But after a few days, some users started reporting a new symptom.

ISSUE

Scenario:
– Teams Tenant using Direct Routing
– User enabled for ‘Busy on Busy’
– User is on the busy state ‘in a call’ (PSTN, Teams-to-Teams or meet/confcall)

When a PSTN number calls that user, he will hear the busy tone. But the user will get a storm of missed calls notifications from the same number:

Storm of missed calls notification from a single call

Cause

When checking the SBC (Audiocodes) logs I could confirm several calls from the PSTN provider for every Busy answer of Teams:

The caller seems to be dialing every second when it gets a busy

Knowing that the user and his phone do not redial that fast, I decided to open an inquiry on the Telecom Provider.
But when I was collecting the SIP call logs, something came up when I was checking the Q.850 reason cause code:

All busy fields seems OK, but what’s that code=34 ?!

The ITU-T Q.850 codes specifies that 34 is ‘NORMAL_CIRCUIT_CONGESTION‘. The SBC sends this SIP ‘Busy Here’ to the Telecom providers and depending on the Telecom provider of the caller, it will retry the connections several times until it hangs-up.
Know you know why you get so many missed calls in a few seconds.

SOLUTION

From the ITU-T Q.850 codes table, the reason code for Busy should be 17.
Let’s fix this Teams Busy here ‘hiccup’.
Since I am using an Audiocodes this is my solution:

• Create a Message manipulation rule that will detect if the ‘SIP 486 Busy here’ has the incorrect reason code. If so change it to ’17’.

Busy code not ’17’ ? let’s change it

• Add or assign this message manipulation to the Teams inbound routing

Now when Teams send the Busy here, your SBC will send to the Telecom provider SIP message with the correct busy code, and they will not call again 😉

Hi real Telecom providers, the called number is busy here

Final notes and references

After the troubleshooting and solution I ‘googled’ for the right keywords and looks like that there are other blogs with the same and similar situations with different reason codes numbers:

• (msXfqd.de) DirectRouting und Q.850 Codes
• (erik’s blog) Call loops after enabling busy on busy for Microsoft Teams Calling with Audiocodes Mediant
• (Luca Vitali) How to fix Microsoft Teams SIP 486 Busy Here to Q.850 mappings in Ribbon SBC