ucCSI Case: The mysterious calls to the emergency services

Sometimes there are some SfB reported issues, involve understanding the human factor behind the logs. This case went throw several weekly episodes.

Episode One – The ‘issue’

Our customers SfB has a Contact Center integrated and one day he received a complain from an external associated partner that sometimes they received calls from the Agents and when the pickup the cal….l they were talking with a person from the national emergency services (Switzerland has several emergency numbers an in this case it was the Police).

The first step to be able to know where to search on the logs is to ask for reported situations: when (dates and times), who started (numbers), what happened next?. It was not easy since day one, because it involved at least 2 different calls (one inbound and another outbound), But I managed to identify the flow:
A client calls the SfB/CC number (1), an Agent picks the call (2), he calls the Partner support number (3) and he transfers the call of the client to that partner (4). Except that sometimes, the Agent of that partner instead of hearing the client get surprise by ‘Police ! what is your emergency?’

Episode 2 – Analyzing the facts and recreating the exact steps

The outermost challenge was to track the client call, which Agent picked the call and what did he do next to reach the Partner and transfer. Looking at the logs of several calls made to the partner number, I noticed that some Agents were using DTMF tones.

The Partner number is a Contact Center with an IVR! So now we have a more detailed call flow:
A client calls the SfB/CC number (1), an Agent picks the call (2), he calls the Partner number, goes throw the IVR (3) an Call Center Agent picks the call (4) and the customer Agent transfer the call (5) . Except that sometimes, the Agent of that partner instead of hearing the client get surprise by ‘Police ! what is your emergency?’

Time to include the Partner telephony provider for some potential issue on their side. But after some tests calls there was nothing on the call history the involved calling the authorities and the only active call was coming from our SfB Agent number. The other clue was that the Police that picked the calls is not from the same region as the partner but from our Customer. So the call is definitely triggered from SfB or our operator.

Episode 3 – The clue and the primary suspect

Looking now at the log history of all calls made to that IVR number, one call (that did not triggered the issue) caught my attention on the DTMF pattern. It was typing: 1,1,2. And 112 on a dialpad is…. the well-know emergency number!

Went back to the customer and the Partner and they confirmed that the affected Agents belong to a service that are behind the IVR menu: by dialing the IVR, listen the options and choose 1, then the sub-option 1 and finally the final option 2. There were no reported issues on other queues reached by other IVR menus.

Now we got a primary suspect: What if this DTMF ‘112’ sequence is triggering a call to the emergency services? but how does this is ‘accidentally’ transferred to them?

Episode 4 – Eliminating the suspects

There was still a loose end to clear out. The ‘DTMF 112’ theory issue could have two sources:
– The Customers Agent that we see on the previous logs.
– We noticed that some Agents will transfer the call to the Client before the IVR, and then you will only see the Client call trying to use DTMF.

Time to define a precise script of call testings to identify the one that is causing this mess. Using the minimum amount of variables (same client number to call, same customer agents queues, same partner IVR number:
1. Just to prove the theory: Calls to other IVR options 1,0 or 2,0 all working
2. If the agent transfer the Client that the IVR, we noticed that DTMF over this SfB call transfer doesn’t work (the IVR didn’t recognized the options): this one is excluded
3. Several call transfers using IVR menus 112 were also working until one particular agent call ended on the emergency services.

Time to look deeper to these last call logs and compare them. Nothing looked different, DTMF was ok, except… for the ones that worked, I could find the Client call transfer to the Partner, but the not for the mistery one.
So what happened? The answer involved looking back at all the calls on that time frame (this is a large company with a lot of calls….). And then one peculiar outbound call appear on the SBC log:
– from:<Partner IVR number>
– to :+41112 (+41 is the country prefix of Switzerland)
– referred-by: <Agent number>

Gotcha! the SfB client caused the call. But why and how?

Final episode 5 – The ‘crime scene’ and the perpetrator

We are still waiting for the ‘confession’ but, just by using a SfB Client we can present our delegations using any SfB Client:

1. The customer Agent receives a call from the client and click on ‘Consult’
   
2. The Client call is put on hold. A transfer window will appear were the Agent can search for internal contacts or type the Partner IVR number and click ‘Consult
   
3. A new call window will appear. The Agent has now 2 calls on his SfB: the customer on hold (on the left) and and active call the the Partner IVR. The Agent might not notice this because SfB will open the second call window in exactly in front of the one that is on hold
   1. The agent is now navigating throw the DTMF voice system and picks (type): the options 1,1 and 2 that start appearing on the dialpad text (A)
   2. To transfer the IVR call to the CLient he needs to click on the transfer call on the top right side of the call window (B)    >> if you press this one, SfB will transfer the call of the Client to IVR active call and both windows will close
   3. If he clicks on the ‘Transfer’ button above the dial pad, it will initiate a transfer of the IVR call to another new number (C)
      
      But it will still show the Search window with the numbers that were typed during the DTMF.
      You can clearly see the 112 as the default selected number. If he presses ‘Transfer’ now it will transfer the IVR call to the … Emergency service (112).
      
   4. The IVR call window will close but the Client call window will still there on hold (because there was not transfer to him). But the Agent unaware of his mistake, hangs up the customer call
      
      

End of episode:
…
<playing credits>
….
Disclaimer: no servers or software were harmed or fixed during this investigation

Are you really offline on Teams? (part 1)

I will start backwards by presenting the solution before the ‘glitches’.

Making sure that you are really offline for everyone

This is the traditional: ‘I’m off for today/this week, don’t contact me!’
This includes your colleagues and any external or federated contacts.

Just make sure that you choose ‘Sign out’ from all your Teams clients (Windows, Mobile, Linux, OSX, Browser)

• 

  Sign out on Windows,Linux,OSX or Web Browser client

• 

  On mobile client is a little ‘hidden’ down below on the settings

The users on your Tenant will see you offline and also the federated persons:

Both Teams federated tenant (on the left) and a SfB on-premise federated users will see me Offline and (hopefully) will not contact me

“Duh! we all know that. So why blog about this?”
You will not say this if you already had another user complaining that he doesn’t actually see your status offline. Now it’s time for the real topic… Why am I not offline for others?!

#1 Quit or shutdown might not put you to Offline status

Shutting down, hibernate your computer or specially using the Quit option on Teams sometimes will not trigger the Offline status. This seems to be related to client version or simply delays between the client and servers on the HTTPS sessions over VPN connection and corporate firewalls. Example:

I have my status Online and I clicked on the Teams task bar icon and quit the application…

…but all internal and federated users that have your contact will still see me as Available and will try to contact me

‘Appear offline’ will not work for all

Microsoft recently made the appear offline status available for Teams, which might be useful as a desperate ‘do not disturb me!!’ (might work for some who ignore the standard ‘DnD’).

Let’s test it!
(1) I have my status Available and I switch to ‘Appear Offline’

(2) My colleagues see me offline on Teams, check!

(3) If you still have an Hybrid federation with SfB onpremises, all the SfB user will see you as ‘Available’ or the last presence status that you had before changing to ‘Appear Offline’

(4) Your federated partners that use Teams will (most probably) see you ‘Offline’ but any SfB federated partner will also see your last status before you changed to ‘Appear Offline’

A Teams federated contact will see offline, but a SfB user will not and it will normally try to call you

If you check the SfB client logs and ran a trace on the SfB Edge server you will notice that all presence change status are sent from Teams to SfB except the ‘Appear Offline’.

Final thoughts

Let’s see if MS will find out and solve these two cases soon.

So… Are you sure that you are appearing offline to everyone right now? 🙂

“Skype for Business ends in July 2021. Migrate now to MS Teams!”… or not?

Some of my contacts and older customers have been coming to me because they were getting approached by some MS Partners that they must migrate their SfB infrastructure to Teams because ‘SfB is dead’ and ends in July 2021.

I will not finger-point no one here, but I want to clarify headers and speeches like this. I deliberate wrote my blog title like this but as friendly ‘click-bait’ 😉

Skype for Business Online will be retired on July 31, 2021

That is the correct heading. Only the SfB Online (SfBO) will be discontinued. SfB on-premises will continue to work and get support (much) after 31.07.2021

Where can read the retirement information from MS:
– UPDATE: Skype for Business Online retirement on July 31, 2021
– Skype for Business Online to Be Retired in 2021

If you have a plain and simple SfBO tenant your migration is simple (and probably automatically done by MS).

If you are still on SfBO and MS didn’t take any action, then it could mean that you have some an Hybrid SfB, PSTN or 3rd party connectivity:
“Skype for Business Online will be retired on July 31, 2021 after which the service will no longer be accessible. In addition, PSTN connectivity between your on-premises environment whether through Skype for Business Server or Cloud Connector Edition and Skype for Business Online will no longer be supported” – source

In these cases, you really need to take action and plan to move to Teams …or to a SfB On-premises 🙂

Skype for Business server 2015, 2019 are alive at least for 5 years

If you have a full SfB on-premises you can have support up to 14Oct2025. You can check that on the office MS Product Lifecycle

Current SfB lifecycle product support

Although SfB2015 mainstream support is now over, MS is still releasing updates. But to prevent MS to refuse support, either your extend (pay) it, or you should now upgrade to SfB 2019.

There will be a SfB vNext in 2021/22

Has announced by Microsoft there will be a next version of Skype for Business somewhere to be release in the second semester of 2021:
– (MS) The Next Version of Skype for Business Server
– (UC Today) Microsoft Reveals Fresh Details for Skype for Business Server 2022
– (Tom Talks) Skype for Business Server 2022 confirmed, only be available as a subscription

So, if you are comfortable with your reliable on-premises SfB infrastructure and you want to have full administration control as a critical business service, you know that you can have Skype for Business running until 2028 (?)

How to create Teams Live Events in Switzerland (and other regions)

About 3 weeks ago I posted the explanation why you could not schedule Teams live events in some regions like Switzerland and the options that you have to still use the Live Events: Can I make Teams Live Events in my country?

Now it’s time to show ‘option #3’: Unlock and schedule a Teams Live Event

Under the hood

Before explaining how, let’s give the technical explanation:

1. The Teams clients builds are the same worldwide (the same version is available from Microsoft downloads) and therefore have the same capabilities.
2. This means that for a specific Tenant or Region, the Teams client is being ‘instructed’ to block the ‘Live Event’ feature
   

After some days of detailed inspection on the traffic that the Client gets during the sign-in and loading, this setting caught my attention:

Since my Tenant region is ‘ch’ this could mean that the list membership triggers the client not to allow (better say ‘not show’) the scheduling of Live Events.
So…. let’s confirm the theory.

The setup

Here’s my automating cookbook.
(you could also use similar tools like Fiddler or Postman)

Step 1. Download Charles proxy, install and configure it to proxy and decode HTTPS. There are many tutorials on the internet. Here’s a detailed one.

Step 2. Now that we have a proxy inspector, lets intercept and rewrite those settings for the Teams Client:

^{On the tools menu, select Rewrite}

^{(1) Enable Rewrite, (2) Add a rule and (3) name it.}

^{(4) Add a Location. This is the URL request that we want to intercept.
Since the client version changes over time you might need to adjust the query
(example for versions released on year 2021: 1.0.0.2021??????)}

^{(5) Add a rewrite rule. We want to change the Body of the Response, by replacing the “broadcastSettings.supportedQuickStartRegions”:[ and add the region ‘ch’}

That’s it ! Let’s go for the Proof-of-concept

Unlocking the Teams Live Event

With Charles Proxy running (how will see HTTP traffic running), start your browser with the Teams Web Client (https://teams.microsoft.com) and sign-in. It should also work if you just re-launch your Windows client too as long as it uses the cached credentials.

When the client has loaded, you can look at Charles proxy to confirm that the interception has made is job

^{Hey Teams client! ‘ch’ is a supported Broadcast Region}

Go to the Teams calendar and… schedule a Live Event 😉

• 

• 

• 

• 

Once you have schedule the meeting you don’t need the Charles Proxy. Once it’s schedule you can click it on the calendar and make the adjustments.

Now Producers, Presenters and participant can participate on a Live Event from the Teams client.

^{Live Event Producer Teams Client}

Teams meeting recording button became unavailable

ISSUE

• Some day ago, users started reporting that they could not record a meeting. The record option was unavailable:

^{Start recording button dimmed?}

• The meeting policy for the users allow them to record meetings:

^{… Houston! we have a problem!}

CAUSE

Sooooo… what happened?
Time to investigate what has MS changed recently 🙂

Clue #1: Checking the Microsoft 365 message center
You can find MC222640 ‘Microsoft Teams: meeting recordings saved to OneDrive and SharePoint’ is in progress.
“October 19, 2020 (Complete) – You can enable the Teams Meeting policy to have meeting recordings saved to OneDrive and SharePoint instead of Microsoft Stream (Classic)”
But the rest of the info is not clear about what changes and if you need to take a specific action other than opt in/out to MS to change the recordings to OdB (OneDrive for Business) in 2021.

Clue #2: check the client settings received from the servers for new parameters.
Found a new one inside the callRecording: supportedOdbRegions with a list of countries, with “ch” is where my tenants are

"callingConstants.callRecording": {
    "downloadRecordingExpirationDays": 20,
    "supportedOdbRegions": [
        "ae",
        "ch",
        "de",
        "no",
        "sa",
        "br",
        "sg"
    ]
},

Clue #3: Let’s have a closer look at the meeting policies
Nothing has seem to have changed. Recordings are allowed (AllowCloudRecording) and using MS Stream (RecordingStorageMode), but…

^{… what will that parameter AllowRecordingStorageOutsideRegion=false do?}

SOLUTION

As you know, Stream is not available in all regions like CH, so if AllowRecordingStorageOutsideRegion is getting honored, you cannot record if you are not allowed to save on the region that your account is? Solution(s):

1. RecordingStorageMode=OneDriveForBusiness
   and/or
2. AllowRecordingStorageOutsideRegion=true

My preference goes to #1 using the PS command

Set-CsTeamsMeetingPolicy ‘Global’ -RecordingStorageMode OneDriveForBusiness

Problem solved! you got your recording button back and your recordings are now available on OdB

• 

• 

Final notes and references

Looking at that client settings list, the Tenants on the regions United Arab Emirates (ae), Switzerland (ch), Germany (de), Norway (no), South Africa (sa), Brasil (br) and Singapore (sg) could also experience the same issue.

• Microsoft Official and details Docs
  Use OneDrive for Business and SharePoint or Stream for meeting recordings
• Microsoft 365 Roadmap
  67138 – Microsoft Teams: meeting recordings saved to OneDrive and SharePoint
• Microsoft 365 message center
  Message ID MC222640

Teams ‘Busy on Busy’ causes missed call storm notifications

Recently in my company we decided to rollout the ‘Busy on Busy’ feature for all users, to suppress the annoying ring sound that comes on your headsets when you are on a call (or a meeting) and this makes it difficult to hear others.
But after a few days, some users started reporting a new symptom.

ISSUE

Scenario:
– Teams Tenant using Direct Routing
– User enabled for ‘Busy on Busy’
– User is on the busy state ‘in a call’ (PSTN, Teams-to-Teams or meet/confcall)

When a PSTN number calls that user, he will hear the busy tone. But the user will get a storm of missed calls notifications from the same number:

Storm of missed calls notification from a single call

Cause

When checking the SBC (Audiocodes) logs I could confirm several calls from the PSTN provider for every Busy answer of Teams:

The caller seems to be dialing every second when it gets a busy

Knowing that the user and his phone do not redial that fast, I decided to open an inquiry on the Telecom Provider.
But when I was collecting the SIP call logs, something came up when I was checking the Q.850 reason cause code:

All busy fields seems OK, but what’s that code=34 ?!

The ITU-T Q.850 codes specifies that 34 is ‘NORMAL_CIRCUIT_CONGESTION‘. The SBC sends this SIP ‘Busy Here’ to the Telecom providers and depending on the Telecom provider of the caller, it will retry the connections several times until it hangs-up.
Know you know why you get so many missed calls in a few seconds.

SOLUTION

From the ITU-T Q.850 codes table, the reason code for Busy should be 17.
Let’s fix this Teams Busy here ‘hiccup’.
Since I am using an Audiocodes this is my solution:

• Create a Message manipulation rule that will detect if the ‘SIP 486 Busy here’ has the incorrect reason code. If so change it to ’17’.

Busy code not ’17’ ? let’s change it

• Add or assign this message manipulation to the Teams inbound routing

Now when Teams send the Busy here, your SBC will send to the Telecom provider SIP message with the correct busy code, and they will not call again 😉

Hi real Telecom providers, the called number is busy here

Final notes and references

After the troubleshooting and solution I ‘googled’ for the right keywords and looks like that there are other blogs with the same and similar situations with different reason codes numbers:

• (msXfqd.de) DirectRouting und Q.850 Codes
• (erik’s blog) Call loops after enabling busy on busy for Microsoft Teams Calling with Audiocodes Mediant
• (Luca Vitali) How to fix Microsoft Teams SIP 486 Busy Here to Q.850 mappings in Ribbon SBC

SfB Online portal is gone: how do I manage user phones now?

For the last 28 hours I confirmed reports that the SfB Legacy portal disappeared from the Teams Admin Center (TAC). Probably as part of the retirement plan or just a temporary bug because today is back 🙂

This new article explains where to find the Skype for Business settings on the TAC, but…
…what about managing Enterprise Voice users on Tenants with Direct Routing (DR)? Would could do this on the SfBO portal, but there is currently no UI on TAC to do this.

Workarounds

1. Use powershell (very user-unfriendly for 1st line support agents) to manage the users for voice

Import-Module MicrosoftTeams
# this will prompt for the login modern authentication
$sfbOnline=New-CsOnlineSession
Import-PsSession $sfbOnline
  
#User to configure $theUser=@{'SipAddress'='sip:first.last@company.ch';'LineURI'='tel:+41333444555'}
#ex: enable the user for enterprise voice and/or set the number
Set-CsUser -Identity $theUser.SipAddress -OnPremLineURI $theUser.LineUri -EnterpriseVoiceEnabled $true
#ex: disable the user for enterprise voice 
Set-CsUser -Identity $theUser.SipAddress -OnPremLineURI $null -EnterpriseVoiceEnabled $false

2. Locate and use the SfBO legacy portal
The tab item is gone, but the portal is still there ;). Use this powershell script to find the right one for your Tenant

# tenant base url (when it was created)
$tenantBaseUrl = "<customID>.onmicrosoft.com"
$sfbOnline=New-CsOnlineSession -OverrideAdminDomain $tenantBaseUrl Import-PsSession $sfbOnline
(Get-CsOnlinePowerShellEndpoint -TargetDomain $tenantBaseUrl).host

It will return the admin URL where your Tenant is hosted. So, for example if the return value is https://admined4.online.lync.com/, your SfBOnline admin center is https://admined4.online.lync.com/lscp

Open it on a browser and you got your SfBO admin center back 🙂

NOTES
– to use the powershell, make sure that you have installed the Teams Powershell Module
– Don’t forget to assign new users Phone System licenses (or E5) or you will get an error like: Cannot modify the parameter: “OnPremLineURI” because it is restricted for the user service plan: MCOMEETADD, MCOProfessional.

Can I make Teams Live Events in my country?

As I work with Swiss customers, this topic would sooner or later end on my ‘homedesk’. As a simple question became a challenge, it’s time to share it.

The customer has a MS365 Tenant in Switzerland and with these ‘COVID-19 age’ it asked: ‘I have Office E3+E5 licenses and I want to create a Teams Live Event for my 550 employees, but I cannot find the button’ (?!)

Traditionally you use Teams for interactive meetings up to 300 participants, but for more participants (up to 20’000) and also more professional presentations, Microsoft provides to his customers Live Events.

If you follow the official documentation, you find the instructions on how to do it using Teams. But if your tenant is based on Switzerland, Germany or France you will get stuck on ‘Step 1. Schedule a live event‘

• 

  The guide shows where to schedule a live event

• 

  But your client only has ‘New meeting’

You call your Teams admin and he confirms that your user has been granted Live Event permission policy … but the ‘Live Event’ option is still not available.

You will scratch your head and try on the Windows, web, Mac client until you ask Microsoft support or google until you find my blog or the official Microsoft link:

Regional availability

You can use Teams live events in multiple regions across the world. The following information shows availability for event team members and attendees.
…
These countries/regions and clouds aren’t supported
– Germany, France, Norway. South Africa, South Korea
Switzerland, UAE, Government Community Cloud (GCC)-H, DOD

Case closed! You cannot make Teams Live Events if your tenant is on one of these countries/regions.

There are no technical limitations or GDPR reasons as other EU countries have (like Portugal or Italy). I believe it’s more a political/legal concerns involving companies,governments, CISO’s and where sensitive data is stored like a Live Event recording.

And now what?

‘But I don’t share that concern and I am paying for a subscription that includes that!’

In that case you can still schedule Live Events…. They are documented 🙂 Here’s 2 of the my 3 ‘workarounds’:

Yammer

Use or create a Yammer group, grant admin permission to the organizers, switch to the classic Yammer style… there it is! 🙂

how to create a Live Event from Yammer

To organize and conduct the event you can google for some videos or follow Microsoft documentation: Organize a live event in classic Yammer

The main advantage of conducting throw Yammer is that users can interact throw the Q&A channel (I am not a particular fan of this UI method) and it will stay organized for future view on a yammer channel.

Microsoft Stream

Microsoft Stream is the back-end of Teams and Yammer Live Events. When you schedule them, it will be used to Broadcast and record the Live Event video produced by you to the audience.

Has long as your account has Live Events permissions (it’s allowed by default to all Tenant users) it’s a ‘one-click’ way from https://web.microsoftstream.com

How to create a Live Event from Stream

Live Events created from Stream are broadcasts mainly but it the easiest tool to create. You compose the event information, set access permissions, send a nice looking email to the participants and you can record it make adaptations and include captions to share it later :). People receive and click link, it opens the browser and… voila.

Again use Google or just the Microsoft documentation ‘Create a live event in Microsoft Stream‘ to prepare yourself and the presenters.

• 

  Participants view

• 

  Organizer and Producers view

About the options

Depending on the final audience (employees only, guests, anyone) please check the tables on the ‘features breakdown by service and event type‘. Depending on the Production setup choice, there are limitations to know on how can people watch, interact or even produce

About other Microsoft Options

While you are still Teams-limited here’s some additional information regarding Microsoft additional solution:

• Moving your tenant to another region is not possible
  
• In August/2020 MS launched a Team add-on “Advanced communications“
  After a contact with MS support, this feature just increases the number of participants and allows logo upload of a Live Event.
  It will not remove your regional restrictions, so don’t waist your money
  
• On December CY2020 Microsoft is rolling out Large Meeting support for Teams (up to 1’000 interactive participants)
  
• As a plan C you can always create a small dedicated tenant on a ‘supported’ region and create a Teams Live Event and invite your employees 😉

Producing the content

All looks simple, right? Well… Live Events are not meetings. It’s much more than a ‘create meeting’ click and send invites to the participant.

They are similar to a TV show. You need to plan in advance the content, the audience communication, invites to send train in advance your participants and … Produce it. You are going to need a producer tool (and a person with know-how) that will aggregate all the audio and video sources and send them to Microsoft Stream to broadcast to your audience.

Creating and scheduling from Teams

This is the big advantage of Teams. You schedule the Live Event with all details, select the audience, presenters and the producers and send the invite. For the producers, the Teams client will turn into a Producer software. Presenter will join the meeting you place the content to share and start the event from there and change the content and video throw the event.

Teams client in Live Event Producer mode

Creating and scheduling from Yammer

When scheduling live events from Yammer it will ask you for the producer to use: An external/3rd party (same as Stream) or Teams (it will not be electable if you are on an ‘unsupported’ zone)

Creating and scheduling from Stream

When scheduling live events from Stream you can only use a 3rd party solution to send the content to Stream for broadcasting. You will get a list of support hardware/software solution providers or use another compatible solution

There is this great free software solutions OBS Studio, that grab so many sources (cameras, mobile phones, capture desktop, web browser, applications and event Teams -with the NDI option ON-) and so easy to use that the only limit is your imagination and skills.

Composing this post while filming with the laptop and mobile cameras. (I should have shaved for this ocassion…)

There you go my Swiss friends! You can now Produce and Stream live events.

Let’s see if this post will gets 1’000 likes/shares to publish an ‘eastern egg’ regarding just Teams Live Events 😉

Polycom CCX family experience

Just had the pleasure to have the newest Microsoft family models of Poly on my lab: CCX400, CCX500 and the CCX600 for a short time.

This generation of phones are android-based, touch-screen only and have two hardware versions: with or without an handset (except CCX400). The audio quality is simply… Poly at is finest :): HD Voice, Acoustic Clarity technology providing full-duplex conversations, acoustic echo cancellation and background noise suppression.

The black design is stylish, high-resolution crystal clear screens and an optimal user interface.
Apologize for my low photography skills that don’t make justice with my review.

CCX400 (on the left) with the big brother CCX600

CCX400: with 5-inch color LCD (720 x 1280 pixel)
9:16 aspect ratio

CCX600 with 7” color LCD (1024 x 600 pixel), 16:9 aspect ratio

The larger screen of the CCX600 allows you to view more details like a Meeting appointment

Just like all the other models, you have can:

• Activate a management Web interface were you can remotely manage the phones on all details

• Remote capture a live screen (and even remote control the phone)
• Centrally manage and provision them (Poly RPRM, ZTE, …)

The phones are OpenSIP, Skype for Business and also support, off course, MS Teams.

But you don’t get much control of the phone throw its Webadmin UI, since it’s basically now running a Microsoft cloud based client managed by the Team admin center:

• 

  Admin UI disable features

• 

  Centrally managed phones using the MS Team Admin center

On the age of softphones, Poly continues to provide a great set of solutions where a deskphone companion still helps: a reception, a meeting room, a secretary or common area phones.

You can get all the details on theses phone family on the official Poly CCX site, but here’s some brief differences:

CCX400	CCX500	CCX600
5” color LCD (720 x 1280 pixel) 9:16 aspect ratio	5” color LCD (720 x 1280 pixel) 9:16 aspect ratio	7” color LCD (1024 x 600 pixel) 16:9 aspect ratio
One USB type-A port	1 USB type-A port 1 USB Type-C port	1 USB type-A port 1 USB Type-C port
	Bluetooth 4.2	Bluetooth 4.2
		Wifi

full comparison details here

NOTE: there is also a CCX700 not mentioned here because it supports the OpenSIP firmware only

Teams new meeting experience breaks call queues answering

Just recently enable several call queues on customers and experienced this situation. Here’s the workaround for those who were scratching the head like me.

ISSUE

When a user receives and picks a call from a Call Queue from the Windows desktop client, the call window closes immediately after it opens and the call drops

Affected users/clients:

• Only users receiving calls from call queues (CQ)
• Users with Teams client v1.3.0019173 (64bit) and above running on Windows 10

Unaffected users/clients:

• Normal inbound calls not coming from CQ (or Autoattendants)
• Mobile or Web clients can answer calls from CQ

Cause

(not confirmed) Issue related to call queues created between mid to 29^th of  September

(confirmed) The issue is caused by the new meeting experience feature launched in July/2020

(23.10.2020) Fixed

Since today that the issue has disappeared from both Tenants. Maybe something done by MS on them since I only detected the issue on these two.
The latest client version update was on 16th October, so doesn’t seem to have been the solution and the cause.

Workarounds

1. Disable the ‘new meeting experience’ feature and restart the Teams client

2. Or, use the Web Client to answer call queues

NOTES

• This feature is being rolled out gradually to Tenants, so some might not have received and noticed the issue
  
• For customers that don’t control version rollout (i.e. users can install Teams by their own) – It seems that the automatic clients updates (it reinstalls Teams) will enable this feature automatically

Reference(s):

https://microsoftteams.uservoice.com/forums/908686-bug-reports/suggestions/41119102-if-new-meeting-experience-is-enabled-telephone

https://techcommunity.microsoft.com/t5/microsoft-teams-blog/new-meeting-and-calling-experience-in-microsoft-teams/ba-p/1537581/page/6#comments

Why SfB fails to join meetings?

It’s time to explain the logic of a Skype for Business client joining a meeting and a ‘hidden secret’. I will not go throw all the details. Some parts of the process are not include to keep the content less boring and confusing.

Nowadays, with the majority of users in Homeoffice environments, the company networks have ‘extended’ and included different type of secure/VPN remote access. They were also forced to open SfB external access for collaboration with employees and business partners.

This has exposed one particular behavior to the end-user thats your SfB infrastructure has a problem while connecting to meetings.

The SfB Join meeting logic

A typical and formal SfB meeting has the following sequence. Here’s an overview of the process before the detailed explanation:

(1) The presenter creates an Outlook invitation (using the SfB meeting plugin). This generates a meeting link url where participants can click and join. The presenter can also (should) adjust the meetings settings and permissions and then (2) send the email to the participants.

(3) The participants just need to press the link to join the meeting (or dial the phone numbers), right?
Now it all depends on a series of factors from the computer software to the network where the user is. The meeting url is a web link, so 99,5% of the participants will be able to open it. What happens next is ‘SfB sweet magic’

• If you have a SfB client installed it will launch it to join. If not, then the participant can use the web browser to install and launch the ‘Skype Meeting app’ plugin to join the meeting
• If you are using a personal computer at home, or the SfB client on your mobile, the probability to join the meeting is very high
• If you are joining a meeting from a colleague and you have a company computer the probability is also very high
• If you are joining a meeting hosted by another company, then a series of conditions will trigger the SfB client behaviour.

This last situation is the one I want to explain, either if you are a SfB user or system administrator to understand why sometimes you will not be able to join the meetings.
The SfB federation/meeting guest policies define if and how the users can join meeting.

(3a) If both companies SfB are allowed to federate, the participant SfB client will try to reach the SfB servers (throw the Edge server and then to the internal servers hosting the conference)

(3b) one or both companies are not allowing federation with each other, but the the meeting policies allow guest participants, then the SfB client will try to join as a guest. Internally it launches an instance as anonymous so it can bypass server validation. You can see this on the Client logs (at it also appears on the Monitoring reports)

(3c) of course, if neither federation and meetings guest access is allowed, then participants from other companies will be unable to join.

The ‘security and network policies’ factor

As you could read, SfB has a lot of resources to be able to help users to join a meeting. But the scenario 3b presents a new challenge when the user is behind the company network security architecture:

• If you connect to your company network ‘on-demand’ (you can connect/disconnect the VPN) or if you have a split-tunnel VPN in place, the probability to join the meeting from other companies is very good
• But if you inside your company network of if you have a allway-on VPN (you cannot disconnect it and use you home internet connection) with a forced Tunnel (all your computer traffic must go throw the company network firewall), then the probability to join the meeting from other companies is very low

To explain this let’s use the same meeting flow diagrams with the network.

With federation allowed between companies, users will join the conference. The audio and video will go either directly (homeoffice) or throw the SfB Edge servers (VPN and LAN users)

But if federation is not allowed between companies, the SfB client will try to join as a guest. But now the audio/video must go directly to the Presenter Edge server as it cannot use the Participant’s Edge servers (not authenticated).
Why? because the companies network firewalls usually block any desktop client attempts to access directly the internet. Understandable, because the audio and video ports are sometimes dynamic and cannot be properly inspected.

The bad image

As an IT engineer you now know why the client will fail joining meetings.
But for the less informed user, all that he sees is the yellow warning/error information when SfB fails to join a meeting. And since the initial part of the joining is web traffic, the client might actually open and join, but then the audio fails and the meeting ‘dies on the beach‘.

For Sysadmins the SfB is working fine, but for the frustrating Presenters and Participants SfB is just failing: ‘SfB is a *”&*ç%, VIPs escalate incidents,…

Many companies rushed users to homeoffice, asked the network teams for VPN access but forgot to involve the UCC teams on the process, flooding them with tickets and complains

The workaround

There is no 100% solution for this and the issue is actually related to processes:

• Allow federation between companies
  If not using open federation, you need to allow it the domain that is blocking it
• Solve the internal firewall blocking
  It’s more a political/security issue than a technical one 🙂
• (or) allow VPN Split tunnel
  it might solve not just this, but other issues when trying to join meetings from other 3rd parties
• Sometimes it takes two sides to solve the problem
  Ex: you SfB sysadmin might solve the problem of you to join external meeting, but for external parties to join your meetings it requires solution from the SfB/network admins from the other party
• Keep the 1st line of enduser support aware of the new network complexity and how to troubleshoot
  They now have not just to check the LAN and VPN, but also any mix of homeoffice internet access, private computers,… 😦

Final note: “Microsoft Teams is better” (?)

By this time and after these and other ‘issues’ every SfB Admin already heard everyone commenting: “MS Teams is better”. “I don’t have these problems with Teams”, “other companies are better with teams… we are stuck with this limited SfB”

Well, this particular “issue” will also happens if you use Teams and if your network is configured the same way.
And it would be even worse: You would not be able to use audio or conferencing!
Why? because the Teams client also uses the same audio/video logic for ports. The firewall will block the same way as is does for SfB.

Because of this, a participant that doesn’t have Teams or SfB will not be able to join any meeting invitation if they are inside their company LAN/VPN.
(There are actually companies that use other UCC solutions other than MS, you know 😉 ?)

But it’s not failing, why?

This is the unfortunately difference between the company SfB engineer and an Official Microsoft consultant.

Microsoft has documented pre-requisites for Office365 and Cloud services. Between them, the requirement to allow audio/video ports access from internal networks to O365 media servers.
No one will block/object this against MS, but the SfB engineer as to struggle internally to get the same results.

 

 

Workplace contingency plans: the hidden issue

The Covid-19 pandemic caused an worldwide cause for concern. The best way to contain it is to reduce people direct interaction.
Some governments already imposed travel bans, forbid crowded events and closing schools.
Companies also limited travelling and ultimately send people to home office.

This is a great case scenario for companies to have the right UCC solution in place.
People can still collaborate, arrange meetings on the ‘safety’ of their home without the risks of public transport travelling, office doors knobs, next desk colleague or customer meetings.
Now, Skype for Business and others become a critical tool for companies.

But there is a ‘unexpected catch’ for companies to send half or more of their workers home: How do workers access the company internal resources? usually using a VPN.

Suddenly, companies have a large number of people using the internet speed and bandwidth to fighting for access to the systems (and also the Internet) -and it’s probably not the 1Gpbs per user as on the office LAN –

Now this old feature topic raised again.

The issue

Besides the issue of available bandwidth (including the one at home), how this can this get worth?

Some companies have VPN policies (either to security reasons or simplified administration) to enforce all their managed PC to send all the traffic throw the VPN (let’s call it ‘Forced-tunnel VPN’).
This includes applications traffic, emails, files, internet browsing including video content and… Skype for Business (SfB)!

As you already imagine people expect audio, video and the shared contents to be real-time but the SfB client is competing with:

• Other applications loading files, email, video from the same tunnel
• Double encryption/decryption: SfB encrypts his traffic and the VPN encrypt the traffic that is sent over the internet

If not well planned or prepared, IT support is going to have a flood of disgruntled users complaining about voice quality issues, failures, and unsuccessful meetings.

‘Force-tunnel VPN’ creates an additional problem for real-time protocols. Instead of delivering the packets to the shortest route possible, it will take a very long path in some cases. Let’s use the following picture to show you that:

There are two evident situations:

• The calls between two home office worker of the same company will go first to the VPN server. And the call might get encrypted/decrypted twice
• If another ‘SfB enabled’ company also uses Forced-tunneling the traffic will (1) get encrypted/decrypted until the SfB Edge server (2) to the other company Edge server and encrypted/decrypted again.

Now you have SfB traffic getting encrypted on an (overloaded) VPN tunnel traveling between several other systems and networks.

End user calling: “Skype for Business is a sh***. Totally useless”

Is there a solution?

Ask the CFO that you need to increase the internet bandwidth 🙂

Or… implement a Split-tunnel VPN.
SfB takes advantages of protocols like ICE and STUN/TURN to pass through routers and firewalls and get the shortest path to the other endpoint.

Let’s see the same picture now, where users don’t use a VPN or there is a complete Split-tunnel configuration:

Differences?:

• Home Office calls are going directly throw the Internet and encrypted only once (native done by SfB)
• The other SfB call and conferencing will go to the internal LAN throw the SfB Edge server (and encrypted only once)
• All SfB traffic will not consume VPN bandwidth

Is it important? as the Covid-19 continues to spread, more and more companies will adopt, someway or another, home office policies.
If 5% of home office of the users complaining about calls issues might not be important, but if you suddenly have 50-75% of your staff at home a SfB issue would make you look at a different perspective.

How to implements a split-tunnel for SfB?

There are many resources on the Internet to implement split-tunneling. I will not enumerate them because you need to understand how your VPN is implemented and the Windows configurations in place (local firewall, group policies, QoS)

The main concept is to ensure that all the SfB traffic can bypass the VPN. You need to:

• Ensure that the home office client can reach and route traffic to the Edge servers
• Block media ports from reaching the internal front-end servers
• And let the SfB client do the rest!

Almost there! this will get you a ‘half-split-tunnel’. Unless your VPN client is smart enough to allow the SfB client to reach any public IP address, the above solution allows them to reach the Edge servers. The traffic will bypass the VPN, and it will use the Edge servers:

To get to the complete split-tunnel solution, you actually need to configure the VPN client to route only the internal company addresses and let the remaining apps to reach the internet.
Advantages: your VPN will only have traffic for the internal applications, Skype for Business calls will go throw the fastest path.

This solution also place another challenge for companies with stricter security rules: ‘all companies PC traffic must go throw the VPN’. A good opportunity to rethink on newer security solutions 😉

And before you decide to optimize the SfB calls,  here’s my IT usual recommendations:

• test first before rolling out to users: worst than some call quality issues is having no calls at all
• Ensure that you have enough resources on the help-desk to support users troubleshooting their Home LAN and the router

You can now a happy ‘home office quarantine’ 🙂

Final notes:

• This is not an issue/solution specific for SfB. You will face the same situation either if you are using Cisco on-premises, MS Teams, Webex, ….
• Keep safe! Careless is as bad as Panic.

 

One Uppercase letter+one misconfiguration=4 hours quest

Just a normal a day with a SfB on-premises (yes, there are still some installed in the world) after migrating the RGS from another domain to this one and you decide to look around on the Monitoring services if everything is OK…

ISSUE

Just go to the Monitoring Reports and pick ‘Response Group Usage Report’:

And… come on !! really?

Let summarize my last 4 hours:
(1) Go to the the SSRS, <program files>\Microsoft SQL Server\LogFiles and you will find this ‘self explanatory’ error message (whaaaat?):
processing!ReportServer_0-41!f30!10/14/2019-14:54:52:: w WARN: Data source ‘CDRDB’: Report processing has been aborted.
processing!ReportServer_0-41!f30!10/14/2019-14:54:52:: e ERROR: Throwing … —> Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Query execution failed for dataset ‘MainDS’. —> System.Data.SqlClient.SqlException: Invalid column name ‘TCTIme’.

(2) Start a SQL Server profiler session for the LcsCDR database, repeat the ‘query’ and you will get the call to a storage procedure:

(3) Open the LcsCDR database and manually execute that stored procedure dbo.CdrRGSUsageTrend. Gotcha!

(4) edit (Modify) the store procedure and you will find one line where the temporary column ‘TCTime’ with the ‘I’ in uppercase (the only one on the entire SQL statement)

CAUSE

(1) The uppercase ‘I’ is a long-term MS bug.
If you go to the <program files>\Common Files\Skype for Business Server 2015\DBSetup and open the ‘CdrDb.sql’ you will find it defined like that.

(2) Check the LcsCDR properties, and you might find that the collation is not Case Insensitive (CI) which means that ‘i’ <> ‘I’

WORKAROUND

Modify the ‘I’ to lower case and save the store procedure.
This will solve the problem… until you update SfB databases, because the CdrDB.sql will replace the store procedures with the uppercase ‘I’… unless MS fix this on the next CU

THE SOLUTION

Change the DB SQL collation to Case Insensitive (CI), like the default ‘Latin1_General_CP1_CI_AS’

You might now say that you really need an ‘I’ (or two) to troubleshoot this one 😉

Skype for Business security challenges – part 3

This is the third part of the topic: ‘enhancing Skype for Business environments’. In case you miss, check part 1 and part 2 to get the full picture.

On the previous topic post I focused on the ‘challenges’ on exposing user accounts and service from unauthorized access or DDOS. But now let’s see the scenario with authorized users using the collaboration features.

As mentioned before SfB is a tool that enables collaboration between people: any device, anytime and anywhere. And with federation you just extend all these capabilities with people across the world (specially with open federation).

But the ‘openness’ of the features can sometimes expose more information than people want to unintentionally share, or worse: intentionally!

 

Let’s use a reverse example: in a traditional meeting room you share information with a specific group of people. If it’s confidential you want to make sure that the information keeps private (closed room), that only the right participants are present, no open doors and all whiteboards, slides erased before leaving the meeting.

It should be the same on a SfB Meeting, right? But from my experience, who checks if the meeting URL is private and no available for guest access? Did you select ‘End Meeting’ when it finished? Did you remove all the shared content that was uploaded?

Another unintentional situation: How many of you sent a username and password using a chat session ? I did 🙂 (and regret it sometimes)

Federation is a great capability of SfB (I loved it, really!), but it can also go against you. Others can see your presence, ‘chat-noying’ and, on extreme cases, it can show more than you think.

Let’s use the picture of a federated test contact that I have on my SfB. This is what you can see from you contact when he changes the privacy level:

	(1) external contacts	(2) Colleagues	Workgroup	Friends and family
Presence	X	X	X	X
SIP	X	X	X	X
Email	X	X	X	X
Title	X	X	X	X
Company	X	X	X	X
Department	X	X	X	X
Office		X	X	X
Work phone		X	X	X
Mobile		X	X	X
Time zone			X	X
Home Phone (3)				X
Other Phone (3)				X

(1) default when adding federated contacts

(2) default for internal company contacts

(3) values set manually by the user on the SfB client

‘So what?’ some people ask
What if the customer finds out that you are an outsourcer, when you mentioned that you work on the main contractor? What if someone based on an email looks on the recipients and locates one the VIP? Why contact you for urgent matters if they can ‘escalate’?

The most extreme example is in fact a risk: Would you allow collaborators on a bank to share their desktop with outside participants and give remote control. The quickest corporate espionage is based on a rogue employee exposing sensitive data to competitors. SfB and other similar tools can be a good tool.

I can hear the readers thinking: this guy is paranoid !!
Answer: I’m not 😉  My out-of-the-box thinking always covers security aspects of the projects that I participate

SfB provides to SysAdmins several features to control and limit on how people collaborate, but in some situations it lacks of granularity. Let’s see some examples:

• You can limit the modalities (can share desktop, application, remote control, use audio/video) on a per user basis. BUT… not per group of users
• Remember the extreme case of undesired desktop/application sharing? You can block with policy. BUT… what if the end-user support is outsourced and you want your users to share the desktop with them ? or with any ‘company group’ domain partner?
• You can in fact block your contact card and presence, by setting that only users on your contact list. BUT… it will do it for all external and internal contacts

Other examples of situations that you can think when administrate SfB:

• Limit federated contacts to reach VIP’s or specific departments
• Block showing internal presence status to all external users.
• Prevent an internal user to share an application but allow external user to share with that user.

• Scan file transfer for virus/malware

And then you get your Legal department with security concerns and compliance policies:

“We need to prevent disclosure of confidential data (ex: block or alert in case confidential project code names, share customer data that violates GDPR rules)”

This 3rd part was also the last one on enumerating the challenges. The next one(s) would be on how to mitigate them.

Skype for Business security challenges – part 2

This is second part of the topic: ‘enhancing Skype for Business environment’. In case you miss, check part 1 to get the full picture.

This one will be shorter and quick to read. It’s about authentication of your user accounts. I will write in form of questions and not go through descriptions the idea is to make you reflect on the topic.

………..

And now how can I:

• enable multi-factor authentication (ex: RSA keys, biometrics, passwordless, …) on SfB Clients?
• limit specific mobile devices to connect to SfB (ex: iPhone 10.2.1 only, block Huawei devices)?
• login in SfB with different credentials than my Domain?
• prevent the user to save credentials locally on any device?
• restrict a user can sign-in on a maximum of two different mobile devices?
• prevent two or more users to sign-in from the same mobile device?
• limit users to sign-in from specific locations/networks (ex: employees service tablet to only inside the sales store Wifi) ? and block from specific countries?

A little side topic: If by this time you have the idea that ‘Skype for Business’ and MS are unsecured,… well most of this challenges can be also observed on the main competitors 🙂

Take me to part 3 >>