Why SfB fails to join meetings?

It’s time to explain the logic of a Skype for Business client joining a meeting and a ‘hidden secret’. I will not go throw all the details. Some parts of the process are not include to keep the content less boring and confusing.

Nowadays, with the majority of users in Homeoffice environments, the company networks have ‘extended’ and included different type of secure/VPN remote access. They were also forced to open SfB external access for collaboration with employees and business partners.

This has exposed one particular behavior to the end-user thats your SfB infrastructure has a problem while connecting to meetings.

The SfB Join meeting logic

A typical and formal SfB meeting has the following sequence. Here’s an overview of the process before the detailed explanation:

(1) The presenter creates an Outlook invitation (using the SfB meeting plugin). This generates a meeting link url where participants can click and join. The presenter can also (should) adjust the meetings settings and permissions and then (2) send the email to the participants.

(3) The participants just need to press the link to join the meeting (or dial the phone numbers), right?
Now it all depends on a series of factors from the computer software to the network where the user is. The meeting url is a web link, so 99,5% of the participants will be able to open it. What happens next is ‘SfB sweet magic’

• If you have a SfB client installed it will launch it to join. If not, then the participant can use the web browser to install and launch the ‘Skype Meeting app’ plugin to join the meeting
• If you are using a personal computer at home, or the SfB client on your mobile, the probability to join the meeting is very high
• If you are joining a meeting from a colleague and you have a company computer the probability is also very high
• If you are joining a meeting hosted by another company, then a series of conditions will trigger the SfB client behaviour.

This last situation is the one I want to explain, either if you are a SfB user or system administrator to understand why sometimes you will not be able to join the meetings.
The SfB federation/meeting guest policies define if and how the users can join meeting.

(3a) If both companies SfB are allowed to federate, the participant SfB client will try to reach the SfB servers (throw the Edge server and then to the internal servers hosting the conference)

(3b) one or both companies are not allowing federation with each other, but the the meeting policies allow guest participants, then the SfB client will try to join as a guest. Internally it launches an instance as anonymous so it can bypass server validation. You can see this on the Client logs (at it also appears on the Monitoring reports)

(3c) of course, if neither federation and meetings guest access is allowed, then participants from other companies will be unable to join.

The ‘security and network policies’ factor

As you could read, SfB has a lot of resources to be able to help users to join a meeting. But the scenario 3b presents a new challenge when the user is behind the company network security architecture:

• If you connect to your company network ‘on-demand’ (you can connect/disconnect the VPN) or if you have a split-tunnel VPN in place, the probability to join the meeting from other companies is very good
• But if you inside your company network of if you have a allway-on VPN (you cannot disconnect it and use you home internet connection) with a forced Tunnel (all your computer traffic must go throw the company network firewall), then the probability to join the meeting from other companies is very low

To explain this let’s use the same meeting flow diagrams with the network.

With federation allowed between companies, users will join the conference. The audio and video will go either directly (homeoffice) or throw the SfB Edge servers (VPN and LAN users)

But if federation is not allowed between companies, the SfB client will try to join as a guest. But now the audio/video must go directly to the Presenter Edge server as it cannot use the Participant’s Edge servers (not authenticated).
Why? because the companies network firewalls usually block any desktop client attempts to access directly the internet. Understandable, because the audio and video ports are sometimes dynamic and cannot be properly inspected.

The bad image

As an IT engineer you now know why the client will fail joining meetings.
But for the less informed user, all that he sees is the yellow warning/error information when SfB fails to join a meeting. And since the initial part of the joining is web traffic, the client might actually open and join, but then the audio fails and the meeting ‘dies on the beach‘.

For Sysadmins the SfB is working fine, but for the frustrating Presenters and Participants SfB is just failing: ‘SfB is a *”&*ç%, VIPs escalate incidents,…

Many companies rushed users to homeoffice, asked the network teams for VPN access but forgot to involve the UCC teams on the process, flooding them with tickets and complains

The workaround

There is no 100% solution for this and the issue is actually related to processes:

• Allow federation between companies
  If not using open federation, you need to allow it the domain that is blocking it
• Solve the internal firewall blocking
  It’s more a political/security issue than a technical one 🙂
• (or) allow VPN Split tunnel
  it might solve not just this, but other issues when trying to join meetings from other 3rd parties
• Sometimes it takes two sides to solve the problem
  Ex: you SfB sysadmin might solve the problem of you to join external meeting, but for external parties to join your meetings it requires solution from the SfB/network admins from the other party
• Keep the 1st line of enduser support aware of the new network complexity and how to troubleshoot
  They now have not just to check the LAN and VPN, but also any mix of homeoffice internet access, private computers,… 😦

Final note: “Microsoft Teams is better” (?)

By this time and after these and other ‘issues’ every SfB Admin already heard everyone commenting: “MS Teams is better”. “I don’t have these problems with Teams”, “other companies are better with teams… we are stuck with this limited SfB”

Well, this particular “issue” will also happens if you use Teams and if your network is configured the same way.
And it would be even worse: You would not be able to use audio or conferencing!
Why? because the Teams client also uses the same audio/video logic for ports. The firewall will block the same way as is does for SfB.

Because of this, a participant that doesn’t have Teams or SfB will not be able to join any meeting invitation if they are inside their company LAN/VPN.
(There are actually companies that use other UCC solutions other than MS, you know 😉 ?)

But it’s not failing, why?

This is the unfortunately difference between the company SfB engineer and an Official Microsoft consultant.

Microsoft has documented pre-requisites for Office365 and Cloud services. Between them, the requirement to allow audio/video ports access from internal networks to O365 media servers.
No one will block/object this against MS, but the SfB engineer as to struggle internally to get the same results.

 

 

Workplace contingency plans: the hidden issue

The Covid-19 pandemic caused an worldwide cause for concern. The best way to contain it is to reduce people direct interaction.
Some governments already imposed travel bans, forbid crowded events and closing schools.
Companies also limited travelling and ultimately send people to home office.

This is a great case scenario for companies to have the right UCC solution in place.
People can still collaborate, arrange meetings on the ‘safety’ of their home without the risks of public transport travelling, office doors knobs, next desk colleague or customer meetings.
Now, Skype for Business and others become a critical tool for companies.

But there is a ‘unexpected catch’ for companies to send half or more of their workers home: How do workers access the company internal resources? usually using a VPN.

Suddenly, companies have a large number of people using the internet speed and bandwidth to fighting for access to the systems (and also the Internet) -and it’s probably not the 1Gpbs per user as on the office LAN –

Now this old feature topic raised again.

The issue

Besides the issue of available bandwidth (including the one at home), how this can this get worth?

Some companies have VPN policies (either to security reasons or simplified administration) to enforce all their managed PC to send all the traffic throw the VPN (let’s call it ‘Forced-tunnel VPN’).
This includes applications traffic, emails, files, internet browsing including video content and… Skype for Business (SfB)!

As you already imagine people expect audio, video and the shared contents to be real-time but the SfB client is competing with:

• Other applications loading files, email, video from the same tunnel
• Double encryption/decryption: SfB encrypts his traffic and the VPN encrypt the traffic that is sent over the internet

If not well planned or prepared, IT support is going to have a flood of disgruntled users complaining about voice quality issues, failures, and unsuccessful meetings.

‘Force-tunnel VPN’ creates an additional problem for real-time protocols. Instead of delivering the packets to the shortest route possible, it will take a very long path in some cases. Let’s use the following picture to show you that:

There are two evident situations:

• The calls between two home office worker of the same company will go first to the VPN server. And the call might get encrypted/decrypted twice
• If another ‘SfB enabled’ company also uses Forced-tunneling the traffic will (1) get encrypted/decrypted until the SfB Edge server (2) to the other company Edge server and encrypted/decrypted again.

Now you have SfB traffic getting encrypted on an (overloaded) VPN tunnel traveling between several other systems and networks.

End user calling: “Skype for Business is a sh***. Totally useless”

Is there a solution?

Ask the CFO that you need to increase the internet bandwidth 🙂

Or… implement a Split-tunnel VPN.
SfB takes advantages of protocols like ICE and STUN/TURN to pass through routers and firewalls and get the shortest path to the other endpoint.

Let’s see the same picture now, where users don’t use a VPN or there is a complete Split-tunnel configuration:

Differences?:

• Home Office calls are going directly throw the Internet and encrypted only once (native done by SfB)
• The other SfB call and conferencing will go to the internal LAN throw the SfB Edge server (and encrypted only once)
• All SfB traffic will not consume VPN bandwidth

Is it important? as the Covid-19 continues to spread, more and more companies will adopt, someway or another, home office policies.
If 5% of home office of the users complaining about calls issues might not be important, but if you suddenly have 50-75% of your staff at home a SfB issue would make you look at a different perspective.

How to implements a split-tunnel for SfB?

There are many resources on the Internet to implement split-tunneling. I will not enumerate them because you need to understand how your VPN is implemented and the Windows configurations in place (local firewall, group policies, QoS)

The main concept is to ensure that all the SfB traffic can bypass the VPN. You need to:

• Ensure that the home office client can reach and route traffic to the Edge servers
• Block media ports from reaching the internal front-end servers
• And let the SfB client do the rest!

Almost there! this will get you a ‘half-split-tunnel’. Unless your VPN client is smart enough to allow the SfB client to reach any public IP address, the above solution allows them to reach the Edge servers. The traffic will bypass the VPN, and it will use the Edge servers:

To get to the complete split-tunnel solution, you actually need to configure the VPN client to route only the internal company addresses and let the remaining apps to reach the internet.
Advantages: your VPN will only have traffic for the internal applications, Skype for Business calls will go throw the fastest path.

This solution also place another challenge for companies with stricter security rules: ‘all companies PC traffic must go throw the VPN’. A good opportunity to rethink on newer security solutions 😉

And before you decide to optimize the SfB calls,  here’s my IT usual recommendations:

• test first before rolling out to users: worst than some call quality issues is having no calls at all
• Ensure that you have enough resources on the help-desk to support users troubleshooting their Home LAN and the router

You can now a happy ‘home office quarantine’ 🙂

Final notes:

• This is not an issue/solution specific for SfB. You will face the same situation either if you are using Cisco on-premises, MS Teams, Webex, ….
• Keep safe! Careless is as bad as Panic.

 

One Uppercase letter+one misconfiguration=4 hours quest

Just a normal a day with a SfB on-premises (yes, there are still some installed in the world) after migrating the RGS from another domain to this one and you decide to look around on the Monitoring services if everything is OK…

ISSUE

Just go to the Monitoring Reports and pick ‘Response Group Usage Report’:

And… come on !! really?

Let summarize my last 4 hours:
(1) Go to the the SSRS, <program files>\Microsoft SQL Server\LogFiles and you will find this ‘self explanatory’ error message (whaaaat?):
processing!ReportServer_0-41!f30!10/14/2019-14:54:52:: w WARN: Data source ‘CDRDB’: Report processing has been aborted.
processing!ReportServer_0-41!f30!10/14/2019-14:54:52:: e ERROR: Throwing … —> Microsoft.ReportingServices.ReportProcessing.ReportProcessingException: Query execution failed for dataset ‘MainDS’. —> System.Data.SqlClient.SqlException: Invalid column name ‘TCTIme’.

(2) Start a SQL Server profiler session for the LcsCDR database, repeat the ‘query’ and you will get the call to a storage procedure:

(3) Open the LcsCDR database and manually execute that stored procedure dbo.CdrRGSUsageTrend. Gotcha!

(4) edit (Modify) the store procedure and you will find one line where the temporary column ‘TCTime’ with the ‘I’ in uppercase (the only one on the entire SQL statement)

CAUSE

(1) The uppercase ‘I’ is a long-term MS bug.
If you go to the <program files>\Common Files\Skype for Business Server 2015\DBSetup and open the ‘CdrDb.sql’ you will find it defined like that.

(2) Check the LcsCDR properties, and you might find that the collation is not Case Insensitive (CI) which means that ‘i’ <> ‘I’

WORKAROUND

Modify the ‘I’ to lower case and save the store procedure.
This will solve the problem… until you update SfB databases, because the CdrDB.sql will replace the store procedures with the uppercase ‘I’… unless MS fix this on the next CU

THE SOLUTION

Change the DB SQL collation to Case Insensitive (CI), like the default ‘Latin1_General_CP1_CI_AS’

You might now say that you really need an ‘I’ (or two) to troubleshoot this one 😉

Skype for Business security challenges – part 3

This is the third part of the topic: ‘enhancing Skype for Business environments’. In case you miss, check part 1 and part 2 to get the full picture.

On the previous topic post I focused on the ‘challenges’ on exposing user accounts and service from unauthorized access or DDOS. But now let’s see the scenario with authorized users using the collaboration features.

As mentioned before SfB is a tool that enables collaboration between people: any device, anytime and anywhere. And with federation you just extend all these capabilities with people across the world (specially with open federation).

But the ‘openness’ of the features can sometimes expose more information than people want to unintentionally share, or worse: intentionally!

 

Let’s use a reverse example: in a traditional meeting room you share information with a specific group of people. If it’s confidential you want to make sure that the information keeps private (closed room), that only the right participants are present, no open doors and all whiteboards, slides erased before leaving the meeting.

It should be the same on a SfB Meeting, right? But from my experience, who checks if the meeting URL is private and no available for guest access? Did you select ‘End Meeting’ when it finished? Did you remove all the shared content that was uploaded?

Another unintentional situation: How many of you sent a username and password using a chat session ? I did 🙂 (and regret it sometimes)

Federation is a great capability of SfB (I loved it, really!), but it can also go against you. Others can see your presence, ‘chat-noying’ and, on extreme cases, it can show more than you think.

Let’s use the picture of a federated test contact that I have on my SfB. This is what you can see from you contact when he changes the privacy level:

	(1) external contacts	(2) Colleagues	Workgroup	Friends and family
Presence	X	X	X	X
SIP	X	X	X	X
Email	X	X	X	X
Title	X	X	X	X
Company	X	X	X	X
Department	X	X	X	X
Office		X	X	X
Work phone		X	X	X
Mobile		X	X	X
Time zone			X	X
Home Phone (3)				X
Other Phone (3)				X

(1) default when adding federated contacts

(2) default for internal company contacts

(3) values set manually by the user on the SfB client

‘So what?’ some people ask
What if the customer finds out that you are an outsourcer, when you mentioned that you work on the main contractor? What if someone based on an email looks on the recipients and locates one the VIP? Why contact you for urgent matters if they can ‘escalate’?

The most extreme example is in fact a risk: Would you allow collaborators on a bank to share their desktop with outside participants and give remote control. The quickest corporate espionage is based on a rogue employee exposing sensitive data to competitors. SfB and other similar tools can be a good tool.

I can hear the readers thinking: this guy is paranoid !!
Answer: I’m not 😉  My out-of-the-box thinking always covers security aspects of the projects that I participate

SfB provides to SysAdmins several features to control and limit on how people collaborate, but in some situations it lacks of granularity. Let’s see some examples:

• You can limit the modalities (can share desktop, application, remote control, use audio/video) on a per user basis. BUT… not per group of users
• Remember the extreme case of undesired desktop/application sharing? You can block with policy. BUT… what if the end-user support is outsourced and you want your users to share the desktop with them ? or with any ‘company group’ domain partner?
• You can in fact block your contact card and presence, by setting that only users on your contact list. BUT… it will do it for all external and internal contacts

Other examples of situations that you can think when administrate SfB:

• Limit federated contacts to reach VIP’s or specific departments
• Block showing internal presence status to all external users.
• Prevent an internal user to share an application but allow external user to share with that user.

• Scan file transfer for virus/malware

And then you get your Legal department with security concerns and compliance policies:

“We need to prevent disclosure of confidential data (ex: block or alert in case confidential project code names, share customer data that violates GDPR rules)”

This 3rd part was also the last one on enumerating the challenges. The next one(s) would be on how to mitigate them.

Skype for Business security challenges – part 2

This is second part of the topic: ‘enhancing Skype for Business environment’. In case you miss, check part 1 to get the full picture.

This one will be shorter and quick to read. It’s about authentication of your user accounts. I will write in form of questions and not go through descriptions the idea is to make you reflect on the topic.

………..

And now how can I:

• enable multi-factor authentication (ex: RSA keys, biometrics, passwordless, …) on SfB Clients?
• limit specific mobile devices to connect to SfB (ex: iPhone 10.2.1 only, block Huawei devices)?
• login in SfB with different credentials than my Domain?
• prevent the user to save credentials locally on any device?
• restrict a user can sign-in on a maximum of two different mobile devices?
• prevent two or more users to sign-in from the same mobile device?
• limit users to sign-in from specific locations/networks (ex: employees service tablet to only inside the sales store Wifi) ? and block from specific countries?

A little side topic: If by this time you have the idea that ‘Skype for Business’ and MS are unsecured,… well most of this challenges can be also observed on the main competitors 🙂

Take me to part 3 >>

 

Skype for Business security challenges – part 1

So…! You have roll-out Skype for Busines (SfB) on you company. Either it’s a simple or more complex (HA, PSTN connectivity,…) it contains 3 components: a Front-end on an Active Directory and an Edge and a Reverse Proxy. These last two are, hopefully, isolated by firewalls.

Skype for Business topology example

Then you configure several policies to allow remote user access (PC, mobiles clients, ), federation with other SfB domains and conferences.

SfB is about enabling collaboration between people: any device, anytime and anywhere. Once you enable all these abilities to your users, you also create new security challenges to your company.

I would prefer not to call them ‘security risks’ at this time, BUT… ignoring them after reading and knowing them, it would change subject title!

Since there is quite a significant amount of content to cover, I divided this topic into smaller post. Initially I will expose the challenges and after them, I will describe how to mitigate them.

Keep also in notice that, although:

• I use an On-premises SfB as the example, you will see very similar challenges using Office 365, in either Skype for Business Online or Microsoft Teams;
• the challenges might be related to external user connectivity, they can be replicated using just inside your corporate LAN.

Let’s start with the first topic:

Part 1 – Denial of Services and exploits

 

1.1. Denial of Services by account lockout

Requirements:

1. knowing a SIP address of an user. How difficult is that? In 90% of the case it matches his email address.
2. knowing the account login. What are the chances that the UPN is the same as the SIP and Email address? Knowing the domain and the samaccountname might be more tricky, but it’s possible.

As soon as someone knows a valid SIP address I can use a SfB client (windows of for mobile) and can now try to login with credentials 5 or more times.

You can try to login on Skype for Business with different credentials

A successful ‘DDoS’ will lock that user AD account (for some minutes or forever depending on the AD policy).

Not big security issue? Well let’s think:

1. a locked user is a person that cannot work until support unlock him
   (and gets locked again by an active DDoS).
   This can be harmful from but can a possible attempt to sabotage a competitor to reply on time to a last minute RFC. And what happens
2. It will a significant issue if the affected user is a VIP or the CISO
3. by default there is no direct way to prevent these, since even the remote access policies will only take effect after the user successfully logs-in
   remember the traditional message: ‘your account is not allowed to sign-in from external networks’?
4. It can get worse: imagine that you have several UCC 3rd party applications that uses SIP ‘service’ accounts? What happens if they get locked-out?
   What if there are still some companies that uses the default ‘administrator’ account? It can also get locked out the same way as many other critical domain ‘service’ accounts

Getting more concerned now? Using SfB clients is not the only way to cause account lockouts…. (Whaaaaatt?)

A simple SfB installation exposes some Webservices that actually ask for user credentials. How? Less experient SfB administrators configure topology with ‘easy-to- guess’ short URLs.

Here an example of a dialin.company.ch. If you click on the Sign In you will get the chance to enter a username and password (to change a PIN).

But there are more services: (1) the join a meeting URL allows you to try to authenticate, (2) the webscheduler (if installed) request the same, (3) if NTLM is enabled on the IIS (SfB webservices), any browse access attempt on known URLs will request authentication (ex: for the address book service: https://lsweb.company.ch/abs)

1.2 Denial of Services or attacks on published external Web Services

As stated before, a very basic SfB deployment will expose the web services to the internet throw a basic reverse proxy role that forwards the traffic throw the DMZ.

There are at least some potential challenges:

1. Simple DDoS against the IIS services until the servers CPU/memory is exhausted
2. Direct exploit of SfB vulnerabilities that on worst case can run unwanted command on the Front-end servers
3. ‘Espionage’: someone can try to guess and find some running meeting with ‘guest’ permissions, since the formats are usually https://meet.company.ch/<username>/<meetingID>

 

1.3. Service exposures

The default (at least for mobile clients) discovery service is https://lyncdiscover.company.com. Just by sniffing this URL you might get some more information. And if you provide a SIP address you will get even more because of the authentication.

In the above example you will get the front-end server FQDN that reply to the request. Might be harmless, but this server internal FQDN is an additional clue to try to use it for user UPN DDoS attacks and as you dig throw other responses you might get to know a little bit more about the internal topology.

 

A normal lyncdiscover and authentication process of a SfB mobile client

Ready to proceed to part 2?

Skype for Business 2015 Server CU8a or CU9?

UPDATE 13/March 21:48 – Microsoft is updating now the info. It’s March2019 update to address a security vulnerability (CVE-2019-0798).Specific details here:
Microsoft Lync Server/Skype for Business spoofing cross site scripting
Better start planning to rollout March/2019 CU9 then! (and Lync 2013 Server if you still use)

I downloaded all the cumulative updates as soon as they are released. I like to keep an history and peek on the changes. Today I need to get the January/2019 CU8, but my repository was unavailable. So I went to official CU download site, but I noticed that the date published was from yesterday, but pointing the the KB3061064 (?!). When I got back the access to my repository, I noticed that this file also has a different version:

Now I have two January/2019 CU with different versions (6.0.9319.537 and 6.0.9319.544) and different file sizes.
Time to dig and spot the differences: there are two msp files that changed:

OcsCore.msp

Two noticeable changes:
– non-US dll language files: they were compiled in different dates, but still have the same version number
– The Tracing files (used by CLS/OcsLogger tracing tool). These one have some significant changes:

EnterpriseWebApp.msp

The files on both packages have the same size, but a ‘look inside’ reveals one particular difference: the ‘Lync.Client.Common.Consolidated.js’ is different.

A closer look reveals 5 lines of codes changes (one seems an additional protection)

So… since MS didn’t update any documentation so far:

• Is this CU8 republished?
  If so, MS will now have customers with different files for the same CU
• Is this a CU9 (or a Cumulative Security update -SU-)?
  It could be, since the date matches the usually releases cycles.

Running the cumulative update installer on a Front-end server with January2019 CU8, confirms the patches changes on the identified components:

The point the a KB4492303 and KB4492302 that don’t exist.

UPDATE 3/April/2019: Microsoft update the ‘Get updates’ section of KB3061064 section with an additional line:
4494279 Fix for Skype for Business 2015 and Lync Server 2013 spoofing vulnerability

That document mentions a ‘March 20’19 security update’:

The March 2019 security update contains a security fix for the spoofing vulnerability that is described in the following security advisory:

• CVE-2019-0798
• CVE-2019-0624

My official guess is now is that this is a SU9 and MS just decided to update this ‘silently’

But some IT engineers might believe that they are downloading and installing CU8 today.

 

The importance of knowing about certificates

Deploying and managing a Lync/Skype for Business environment demands you to know a lot more about technologies and protocols. Their communications are encryption which means that you need to deploy certificates and specially to maintain them over time. One wrong, forgotten or misplaced certificate can give you lot of headaches.

The following issue I faced recently, on a Lync 2013 environment is a good example of how a simple misplaced CA certificate can cause unexpected behaviours.

ISSUE and symptoms

After restarting the servers, the Lync services start reporting connection errors and denials due to certificate validation.

As a consequence, the users experience several issues:

• Contacts presence status unknown
• Address book unavailable
• Unable to schedule, start or join meetings
• External users unable to join/dial-in meetings

They are still able to sign-in, send IM’s and perform peer-to-peer calls (including video and desktop sharing) and PSTN inbound/outbound calls.

On the servers you will find from several others, eventID 32042 errors ‘Invalid incoming HTTPS certificate’ and eventID 30998 ‘Sending HTTP request failed’

Cause

The clue came from some informational events of services receiving invalid client certificates.

The last description line ‘Certificate error: 21482049809.’  translates to error code 0x800b0109, which is defined as CERT_E_UNTRUSTEDROOT. Lync server could not trust the subordinate CA that was installed on the local machine store ?!

Turns out that a new PKI has been deployed, and I found a subordinate CA incorrectly installed on the ‘Trusted Root Certificate Authorities’ of the servers.
(a subordinate CA can be easily identified because it’s not self-signed (‘Issue To’ name doesn’t match the ‘Issued By’)

Windows Server 2012 (and higher) implements checks for a higher level of trust for certificate authentication. By finding the invalid certificate, doesn’t provide any Trust Root CA list and therefore the services cannot to validate the certificates presented to them.

SOLUTION

1. Delete the subCA from the Trusted Root CA store of the server
2. Reboot the server so it can load correctly the Trusted Root CA list.

Final notes

The issue will only start after a server reboot, so it can take quite some time to associate the cause/effect…. especially if you have just install a OS update !! (and blame it, uninstall, …)

Only after knowing exactly the issue, I manage to ‘google-fu’ a 4 years-old KB2795828 with a similar situation.
From that one I got a very usefull powershell script that help us to find any non self-signed CA on the Trusted Root CA store of the machine

Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Format-List *

Skype for Business 2016 client critical exploit public available

The code mentions the Skype for Business 2016 client, but the base vulnerability affects also the Lync/Skype4b 2015 client:

• Risk: severe – exposes the user data
• Exploit codes available here or here
  No user-interaction is required for the XSS to execute on the target machine. It will run regardless of whether or not they accept the message. The target only needs to be online.

Solution: apply asap the June 2017 update on the Skype4b 2016 or 2015 clients

UCaaS: Part #1 – planning to provide Skype for Business as a service?

Over the years of consulting services, managing and deploying UCC solutions, virtualization,… you might have thought several times about: why not provide them to all my customers from as a packed service?.
It’s nothing new nowadays – you called it ‘cloud’, hosting services, …

I’ve been planning and designing it for quite some years. So why not just share some memories, experience and concepts?
I decided to call it, since the 0 day, UCaaS – Unified Communications as a Service. Looks a cliché now, but it’s short and easy to catch 🙂 but also much broader than just providing Skype for Business

In this first part, I will just do a global overview that applies to any XaaS. I will focus on Lync/Skype4B on later posts.
Look at this one as a cooking lesson, starting by the main topic: the Kitchen 🙂

#1 It’s a service, so you need to see more then installing a couple of servers and connect the users! Before getting to that stage think, discuss and question about everything that comes to your mind.
Why? because we are about to run a business, any resource costs something and you need to count them on your selling price !

#2 We can go this way:
* buy a server, put on your basement or garage, connect to the internet, rent/create a webstore site and ready! (it can actually work)… or,
* everything below this line (more or less complex and as a existing company you might already have)

I like to group things to be easy to read and explain . Be aware that you might not need to own or have all on your side (you can just rent datacentre space, VM’s, backups).

Base infrastructure

The ‘hardware’, where you place it and how you reach it: Servers, Storage, Backup robots, Switching, Routing,  Firewalls, Load Balancers, Rack/Datacentre space, energy, internal and external connectivity (cabling, telecommunications, internet), …
By the way: all these also have something called ‘yearly maintenance costs’ if purchased and you will need to allocate some earnings  to replacement them when it’s time.

if planning big, consider consulting Hardware providers that support ‘pay as you grow model’. Many of them have cloud-ready solutions from small footprint up to large scale (and you can find some nice surprises on less-known brands)

Support infrastructure

Some invisible, but ‘must have’ systems:

• Virtualization- of course you will use it 🙂 and on this one we can even have mixed scenarios as the group above (virtual firewall, load balancers,…)
• Monitoring – Ever heard of an SLA ? if you are providing a service you will have to agree on an uptime. How can you measure and show to the customer? How can you detect failures or when you need more resources?
• Backup and DR – Are you ready to loose your data? what about the customers data?
• Security – you will need to manage patching and upgrades, antivirus, IDS, IPS… it’s a dangerous world outside, waiting to steal your customer data or take down you business.

 

Customer interfaces

This is your front porch and you should not hide it. If the prospective customers don’t like it, will they trust the inside of the housing?

• Customer infrastructure connectivity – the way that the customer systems and users will connect to your services. The simplest way is the internet, but it would require for some services, WAN, private networks and interoperability/integration solutions.
• The customer portal/tools – This can be from a simple status/account/billing view up to a self-provisioning, self-management
• Ticket / support handling – five customers might be easy to deal with phone calls and emails, but what about 30,100,…?

Software

I include on this section separately, because not everybody is aware of some legal aspects. Let’s take a look at Microsoft products: you cannot just buy a Windows license, install on your server and charge it to one (or more) customer(s) for a running service there.
Microsoft is clear on this: if you are a Hosting/Service Provider, you need to buy licensing throw a SPLA .
Like Microsoft, VMware and many other vendors provide (or enforce) this model and is not a bad option:

• The advantage is that it allows you to pay monthly for what you really use –
  this is the pay-as-you-use model
• The inconvenient is that you need to report the usage periodically and allow auditing to your business

My advice is:

• as a Service Provider, contact the vendor and explain your intentions. They will help you to find the most profitable solution… most of the times.
•  Don’t try to find ‘loopholes’…trust me: it will cost you much more later!

Shared services

Here you put all the platforms and services that can be shared between multiple.
Great examples are web servers hosting multiple websites. But pushing your skills to the limit, you can have a lot more. Multitenant solutions also would fit on this group.

Dedicated your best resources planning them! This is where your cost savings make a difference.

Dedicated services

This is what your best customers are looking and willing to pay for.
It’s your cash cow – the more you have here, the quicker your revenue increases. These business models have more opportunities, specially combined with standardized offers.

It has every group of systems serving unique customers:

• It should be a ‘block model’ – same deployment and standardized procedures, automation and self service tools, will keep operational costs low and predictable.
• But you can also include very ‘$pecific $olutions’ – these one gives you the opportunity to upsell consulting and managed services.

Skype for Business services model would fit on this group… but it might also fit for the group above? 😉

Not there yet

If you already own a company then this is known to you: Work office place, furniture, energy, personal computers, HR, billing / account management, mobile devices, transportation/gas… you also pay for that, right

All done! What now?

After joining and calculating all the pieces, test yourself doing at least these questions:

• What is the cost per service/per user? How much a VM with a specific size costs?
  This is the lowest value will charge for your service.
  => Capitalism rule: to earn money you need to sell it for more than you pay for
• What is the break even point?
  simulate, over and over the time! Be ready to answer: how many customers (or users) do you need to have to cover all the costs made so far?
  Whomever is going to put money on this will make do that question before writing the checks (even yourself if you got the money)
• What is the ROI?
  Investors, banks or stake holders will look for this (and many other tools).
  If you need financing you need to promise when and how much will you pay back.
• Psychological one: are you an entrepreneur?
  It needs investment analysis, financial control, technical know-how, purchasing/selling/negotiation skills, HR management, and so on. As the business creator you need deal with of them.
  This will not be a one-man-show for long. Sooner or later you need help trust and delegate to others. Starting with someone it will look less difficult.

This might seem directed to start-ups guys, but it applies to existing companies. Product and/or Business developers need also to assume the above challenges, the problems, assume risks and responsibilities.

You are putting you neck and reputation on it and get ready to accept failure.
But of you planed all (not just a suicidal gambler), showed you the balls, and if successfully, the personal reward is priceless.

It’s still not over! there are some more strategic decisions for a go-to-market, on part 2 (writing in progress).

 

The minimal amount of servers for a full HA Enterprise Pool: less than 4

By the end of 2015, I started enumerating some challenges that the Lync/Skype topology presents when you just need minimal resources and provided some answers. Since the beginning of Microsoft UC and the more I dig inside the product versions, I came with some out-of-the-box personal challenges:
– 4/Feb/2012 – Installed a Standard Edition Lync 2010 in a Domain controller (DNS, CA) – LyncIn1Box
– 11/Dec/2016 – Installed Skype for Business 2015 in a Windows Server Core – The smallest Skype for Business front-end server

As strange as it sounds, there are some small companies around the world that depend on Lync/Skype for critical business mission but don’t have enough resources. Companies with small amount of users can go for Office365, but it might no cover all the features of the on-premises. Pool pairing (1 STD Edition + 1 STD edition) might also not be possible if there is no feasible secondary location.

A short technical review of what roles you need to cover for High Availability (HA):
* Front End Lync/Skype server roles
* Back-end databases  – you need a SQL mirror or a Cluster/Always-On
* File Share – can be provided by a DFS or a File Share cluster

Now we can start asking: how many resources do I need to deploy the above HA ?
Here some possible answers. Note that ‘something’ means that a SQL HA needs 2 Instances and a sort of a witness (either a SQL server or a File Share/disk quorum)

Microsoft recommended: 5.x (5 servers + something)

Following Microsoft recommended practices, you will need 3 front-end servers and a redundant back-end database. File Share can be easily deployed on the Back-end servers.

Microsoft minimum: 4.x

As Microsoft allow the usage of two Front end server, but: “This small pool will not provide a robust high-availability solution like a larger pool would, and needs extra care in managing“.

My recommended PoC: 3.0

By joining all the roles across 3 windows servers, you can have one server down and the built-in automatic failover mechanisms will take care of that.

My minimum PoC : 2.x (2 servers + something)

Redundant means at least two systems, so this is the absolute minimum :). It has the same risks as Microsoft states for two servers.

The main reason is the fabric quorum: If you unexpectedly loose a server, the fabric cannot elect the owner of the several roles… even if there is only that server. There will be the need of a ‘soft’ quorum reset (manually or a triggered task)
But if you shutdown a server/services gracefully all services will be transferred to the working node.

About the 2.x equation:
* If you use a SQL mirror, a SQL witness is required (which actually means an additional/existing Windows server)
* If you use SQL clustering/Always On, you will need: (1) a file share witness -doesn’t have to be a Windows but can count as a server :)- (2) a quorum disk (share storage) – and this is the ‘true’ 2.x–

Final thoughts

Deploying this ‘minimalist’ solution is more about ‘out-of-the-box-thinking’ than tweaking (actually it’s only one). The normal Skype for Business setup wizard will install normally all the roles defined on the topology.

Notes about this ‘minimal counting’:
* It’s about adding Skype services: it assumes that you have a Windows Domain, a CA and a sort of Load Balancer.
* SQL Clustering/AlwaysOn is still on test stage, but the service failover should behave similar as the Mirroring.
* If you already have a SQL HA in place at the datacentre, you can reuse it. This will make MY and MS minimal requirements equal to 2.
* At this time I excluded the Edge, Office Web App roles or Persistent Chat. The first one will required more resources, the second role… is for an updated post 😉

Logging scenarios that crash and cripple the RTCCLSAGT

Another quest with an issue that appears to be there since some time of Skype for Business 2015 server (if not even Lync server. I decided to find out and document it.

ISSUE

Consider the following scenario:
1) You start the Skype for Business logging tool
2) Pick one of the following builtin scenarios (CmdletDebug or IISLog)
3) As soon as you start this scenarios on the selected servers you will notice the following error message on the logging tool output window: “ResponseMessage: Error code – 20000, Message – Unknown error – Error calling agent <FE-FQDN>; Could not connect to net.tcp://<FE-FQDN>:50002/. The connection attempt lasted for a time span of 00:00:02.0071248. TCP error code 10061: No connection could be made because the target machine actively refused it 10.101.128.20:50002. . Please refer CLS logs for details.”

By this time you will notice that the (Skype for Business Server Centralized Logging Service Agent) RTCCLSAGT service has crashed on all the servers you initiated the trace, with the following event error id 33040:
“Centralized Logging Service Agent Error starting background thread to process traces.
Log Type – IISLogManager, Error – Object reference not set to an instance of an object.
Cause: Internal error
Resolution: Examine error details to determine resolution.“

Worst, the service will crash every time you try to start it with the same error. You find yourself without any logging capability on your servers!

CAUSE

The reason is that the two scenario contain invalid trace components called ‘Internal’ and ‘External’. Somehow this triggers an internal failure on the IISLogManager component and damaged it.
When you edit those scenarios, you will notice a warning about the components

SOLUTION(s)

This is how you can fix the two problems:

• Repair the failed service start (no server downtime required)
  Run the repair of the Skype for Business 2015, Core Components.
  
  Pay attention that two services need to be set to Automatic (Delayed Start) or it may fail on Windows 2012 Servers (as the above picture). Just noticed the instructions on my previous post.
• Fix the scenarios to prevent from happening again:
  1) notice the valid components and flags.
  2) delete the scenario
  3) create a scenario with the same name and all the components (excluding the ‘External’ and ‘Internal’)

Cannot repair/reinstall Skype for Business core components on Windows 2012 Server

There are some specific situations where you need to repair or install an a new component of Skype for Business 2015 server. Because of a corruption, I only faced this very recently, but it’s there since November 2016.

ISSUE

You have the following conditions on your Skype for Business server:
– Windows 2012 Server with .Net Framework 4.5.1 installed (with or without the most recent updates)
– Skype for Business 2015 Nov/2016 cumulative update (or later) installed

You are unable to run the repair option of Skype for Business 2015, Core Components with the error message: ‘…installation requires Microsoft .Net Framework 4.5. Installation cannot continue.’

CAUSE

The Nov/2016 cumulative update added a new ‘Launch Condition’ to the .msi installation:
– before it validates the existence of .Net Framework 4.5
PATCH Or MSIPATCHREMOVE Or REMOVE Or NETFRAMEWORK45 OR SKIP_NETFRAMEWORK_CHECK
– now, it checks for the existence of .Net Framework 4.5.2 (releasecode 379893)
PATCH Or MSIPATCHREMOVE Or REMOVE Or (NETFRAMEWORK45 >= “#379893”) OR SKIP_NETFRAMEWORK_CHECK

SOLUTIONS

I’ve identified 3 possible solutions:

A) The ‘SysAdmin Pro’
1. Manually run the following command line (with the skip .Net check):
msiexec /i {DE39F60A-D57F-48F5-A2BD-8BA3FE794E1F} SKIP_NETFRAMEWORK_CHECK=1
2. Follow the setup GUI and choose the repair option
Note: the msiexec repair option ignores the custom parameters and will give the error

B) The ‘sneaky sysadmin’
1. Take ownership (and grant write permissions) of the registry key
HKLM\SOFTWARE\Wow6432Node\Microsoft\NET Framework Setup\NDP\v4\Full
2. Change the REG_DWORD key ‘Release’ to a value equal or greater than 379893

3. Run the repair option
4. Change the REG_DWORD key ‘Release’ to the original value

C) The ‘let’s upgrade and see’ sysadmin’
1. Update .Net Framework to version 4.5.2
2. Run the repair option
NOTE: This was not tested, but it’s recommended at least to rerun repair option on all Skype Web-related components. The main advantage is that is a permanent solution… pr at least until MS release another CU requiring 4.6 😉

Final notes and caveats

I could not find the reason for this change documented:
* Skype for Business per-requisites still points to .Net Framework 4.5
  (Skype for Business runs fine on a Windows 2012 server with .Net Framework 4.5)
* The recent cumulative updates don’t push/required an update to .Net Framework 4.5.2

Important: After you repair the Skype for Business 2015 Core components, it disables the two associated windows services:
* RTCCLSAGT (Skype for Business Server Centralized Logging Service Agent)
* REPLICA (Skype for Business Server Replica Replicator Agent)
Just make sure that you set the services back to Automatic (Delayed Start)

Warning: May2017 windows update break Lync 2010,2013 and Skype for Business 2015 services

This is a quick post notice since I’m still trying to analyse the cause, and more Engineers are reporting the same issue.

ISSUE

 After you install the latest Windows May2017 updates, Skype for Business 2015 will start reporting the following errors:

• External Users reported that couldn’t use WhiteBoard, Polls, Q&A or present PowerPoint with the following errors messages:
  We can’t connect to the server for sharing right now.
  Network issues are keeping you from sharing notes and presenting whiteboards, polls and uploaded PowerPoint files.
  
  

  

   

• Front-end servers: (Event ID 41026/41025) “No connectivity with any of Web Conferencing Edge Server, External Skype for Business clients cannot use Web Conferencing modality”
• Edge servers : “Web Conferencing Server connection failed to establishOver the past 3 minutes Skype for Business Server has experienced incoming TLS connection failures 1 times(s). The error code of the last failure is 0x80072746”
  or event id 41026
  

Cause

The issue seems to be some secure hardening on the TLS negotiation.
According to MS Support, it looks like an ‘update blooper’ on the May2017 patch:
“This update adds an additional check on Enhanced Key Usage (EKU), since all Lync/ SfB Server usually use the Web Server template they will only have the Server Authentication in the EKU.“

Impact and affected systems

Impact level: low. Web conferencing service (whiteboard, powerpoint presentations ).

 According to MS: “The issue has been reproduced on Lync 2010, Lync 2013 and Skype for Business 2015 on all supported server versions (2008r2, 2012, 2012r2).”

Workarounds

– uninstall the May 2017 Security and Quality rollup for the .Net Framework 4.5.2
– Request new Edge Internal certificate with the Client and Server Authentication
– On the Front Ends disable the check for the Web Conferencing Service (follow this post for details on how to do it)

Update 18th May 9:30 –  Update content based on this post (thank you Erdal for pointing me to that blog)

Update 24th May 6:30 – Updated with end-user error messages and picture. Microsoft published KB4023993 describing the issue and the workarounds described above

Skype4B interoperability: Cisco Expressway

Cisco Expressway “offer users outside your firewall simple, highly secure access to all collaboration workloads, including video, voice, content, IM, and presence. Collaborate with people who are on third-party systems and endpoints or in other companies. Help teleworkers and Cisco Jabber mobile users work more effectively on their device of choice. Cisco Expressway allows you to do all this and more.” (source)

Using Skype for Business terminology, this is the “Edge Server” of Cisco. But it besides the described functions, it has one that allows you the integrate Cisco Video endpoints (desktop client, video conference rooms up to telepresence system) with Lync/Skype for Business clients that allows SIP dialling from both sides with audio and HD video: search for the contact and press the call button.

Skype user view

Cisco meeting room view

 

The infrastructure configuration is based on a ‘dedicated expressway’ configured as a Skype for Business ‘trusted application’. The rest is all about SIP domain ‘static routing’.

Expressway topology for on-premises Skype for Business and Cisco video endpoints 

Great features of this integration:

• IM & presence is possible between systems;
• Desktop sharing is supported;
• Both Cisco and Microsoft can share a single SIP domain;
• Skype for Business remote clients can use the Expressway on the DMZ (TURN server) for audio traffic with Cisco endpoints;
• Skype for Business clients can make and receive audio/video calls with external parties whom the Cisco has federation with.

There are also some limitations:

• Presence is based on SIP simple. Only can have the status ‘Available’ and ‘Busy’;
• Skype for Business clients can join a multipoint conference in Cisco, but Cisco endpoints cannot join Skype meetings.

 

In conclusion, although it’s not a better solution than Polycom, the Expressway leverages the existing investment  already done on Cisco infrastructure and can be extended to a complete 3rd interoperable federation solution: