Skype for Business 2015 Server CU8a or CU9?

UPDATE 13/March 21:48 – Microsoft is updating now the info. It’s March2019 update to address a security vulnerability (CVE-2019-0798).Specific details here:
Microsoft Lync Server/Skype for Business spoofing cross site scripting
Better start planning to rollout March/2019 CU9 then! (and Lync 2013 Server if you still use)

I downloaded all the cumulative updates as soon as they are released. I like to keep an history and peek on the changes. Today I need to get the January/2019 CU8, but my repository was unavailable. So I went to official CU download site, but I noticed that the date published was from yesterday, but pointing the the KB3061064 (?!). When I got back the access to my repository, I noticed that this file also has a different version:
SfBupdate-list

Now I have two January/2019 CU with different versions (6.0.9319.537 and 6.0.9319.544) and different file sizes.
Time to dig and spot the differences: there are two msp files that changed:

OcsCore.msp

diff-ocscore
Two noticeable changes:
– non-US dll language files: they were compiled in different dates, but still have the same version number
– The Tracing files (used by CLS/OcsLogger tracing tool). These one have some significant changes:
diff-trancing

EnterpriseWebApp.msp

The files on both packages have the same size, but a ‘look inside’ reveals one particular difference: the ‘Lync.Client.Common.Consolidated.js’ is different.
diff-enterpriseWeb.png

A closer look reveals 5 lines of codes changes (one seems an additional protection)
diff-javascriptupdate

So… since MS didn’t update any documentation so far:

  • Is this CU8 republished?
    If so, MS will now have customers with different files for the same CU
  • Is this a CU9 (or a Cumulative Security update -SU-)?
    It could be, since the date matches the usually releases cycles.

Running the cumulative update installer on a Front-end server with January2019 CU8, confirms the patches changes on the identified components:

unNamedCU

The point the a KB4492303 and KB4492302 that don’t exist.

My guess now is that this is a CU9 or SUx and the documentation team is on vacations this week ūüėČ

But some IT engineers might believe that they are downloading and installing CU8 today.

Advertisements

The importance of knowing about certificates

Deploying and managing a Lync/Skype for Business environment demands you to know a lot more about technologies and protocols. Their communications are encryption which means that you need to deploy certificates and specially to maintain them over time. One wrong, forgotten or misplaced certificate can give you lot of headaches.

The following issue I faced recently, on a Lync 2013 environment is a good example of how a simple misplaced CA certificate can cause unexpected behaviours.

ISSUE and symptoms

After restarting the servers, the Lync services start reporting connection errors and denials due to certificate validation.

As a consequence, the users experience several issues:

  • Contacts presence status unknown
  • Address book unavailable
  • Unable to schedule, start or join meetings
  • External users unable to join/dial-in meetings

They are still able to sign-in, send IM’s and perform peer-to-peer calls (including video and desktop sharing) and PSTN inbound/outbound calls.

On the servers you will find from several others, eventID 32042 errors ‘Invalid incoming HTTPS certificate’ and eventID 30998 ‘Sending HTTP request failed’

Cause

The clue came from some informational events of services receiving invalid client certificates.EventID-61029

The last description line ‚ÄėCertificate error: 21482049809.‚Äô¬†¬†translates to error code 0x800b0109, which is defined as CERT_E_UNTRUSTEDROOT. Lync server could not trust the¬†subordinate CA that was installed on the local machine store ?!

Turns out that a new PKI has been deployed, and I found a subordinate CA¬†incorrectly installed on the ‘Trusted Root Certificate Authorities’ of the servers.
(a subordinate CA can be easily identified because it‚Äôs not self-signed (‚ÄėIssue To‚Äô name doesn‚Äôt match the ‚ÄėIssued By‚Äô)

subCA-on-TrustedCAstore

Windows Server 2012 (and higher)¬†implements checks for a higher level of trust for certificate authentication.¬†By finding¬†the invalid certificate, doesn’t provide any¬†Trust Root CA list and therefore the services cannot to validate¬†the certificates presented to them.

SOLUTION

  1. Delete the subCA from the Trusted Root CA store of the server
  2. Reboot the server so it can load correctly the Trusted Root CA list.

Final notes

The issue will only start after a server reboot, so it can take quite some time to associate the¬†cause/effect…. especially if you have just install a OS update !! (and blame it, uninstall, …)

Only after knowing exactly the issue, I manage to¬†‘google-fu’ a 4 years-old¬†KB2795828 with a similar situation.
From that one I got a very usefull powershell script that help us to find any non self-signed CA on the Trusted Root CA store of the machine

Get-Childitem cert:\LocalMachine\root -Recurse | Where-Object {$_.Issuer -ne $_.Subject} | Format-List *

Skype for Business 2016 client critical exploit public available

The code mentions the Skype for Business 2016 client, but the base vulnerability affects also the Lync/Skype4b 2015 client:

  • Risk:¬†severe – exposes the user data
  • Exploit codes available here¬†or here
    No user-interaction is required for the XSS to execute on the target machine. It will run regardless of whether or not they accept the message. The target only needs to be online.

Solution: apply asap the June 2017 update on the Skype4b 2016 or 2015 clients

UCaaS: Part #1 – planning to provide Skype for Business as a service?

Over the¬†years of consulting services,¬†managing and deploying UCC solutions,¬†virtualization,… you¬†might have thought several times¬†about: why not provide them to all my customers from¬†as a packed service?.
It’s nothing new nowadays – you called it ‘cloud’, hosting services, …

cmn_en_fig_services_vvc_sip_02

I’ve been planning and designing it¬†for quite some years. So why not just share some memories, experience¬†and concepts?
I decided to call it, since the 0 day, UCaaS – Unified Communications as a Service. Looks a clich√© now, but it’s short and easy to catch ūüôā but also much broader than just providing Skype for Business

In this first part, I will just do a global overview that applies to any XaaS. I will focus on Lync/Skype4B on later posts.
Look at¬†this one¬†as a cooking lesson, starting by the main topic: the Kitchen ūüôā

#1 It’s a service, so you need to see more then installing a couple of servers and connect the users! Before getting to that stage think, discuss and question¬†about everything that comes to your mind.
Why? because we are about to run a business, any resource costs something and you need to count them on your selling price !

#2 We can go this way:
* buy a server, put on your basement or garage,¬†connect to the internet, rent/create a webstore site and ready! (it can actually work)… or,
* everything below this line (more or less complex and as a existing company you might already have)

I like to group things to be easy to read and explain . Be aware that you might not need to own or have all on your side (you can just rent datacentre space,¬†VM’s, backups).
UCaaS-model

datacenterBase infrastructure

The ‘hardware’,¬†where you place it and how you reach it: Servers, Storage, Backup robots,¬†Switching, Routing, ¬†Firewalls, Load Balancers, Rack/Datacentre space, energy, internal and external connectivity (cabling, telecommunications, internet), …
By the way:¬†all these¬†also have something called ‘yearly maintenance¬†costs’ if purchased and you will need to allocate some earnings¬† to replacement them when it’s time.

if planning big, consider consulting¬†Hardware¬†providers¬†that support ‘pay as you grow model’. Many of them¬†have¬†cloud-ready solutions from small footprint up to large scale (and you can find some nice surprises on less-known brands)

virtualization-and-nos-150x150Support infrastructure

Some invisible, but¬†‘must have’¬†systems:

  • Virtualization- of course¬†you will use it ūüôā and on this one we can even have mixed scenarios as the¬†group above (virtual firewall, load balancers,…)
  • grafana-150x150Monitoring – Ever heard of an SLA ? if you are providing a service you will have to agree on an uptime. How can you measure and show to the customer? How can you¬†detect failures or¬†when you need more resources?
  • Backup and DR – Are you ready to loose your data? what about the customers data?
  • cybersecurity-590x393-150x150Security – you will need to manage patching and upgrades, antivirus, IDS, IPS… it’s a dangerous world outside, waiting to steal your customer data or take down you business.

 

management-dashboard-500x311Customer interfaces

This¬†is¬†your front porch and you should not hide it. If the prospective customers don’t like it, will¬†they trust the inside of the housing?

  • Customer infrastructure connectivity – the way that the customer systems and users will connect to your services. The simplest way is the internet, but it would¬†require for¬†some services, WAN, private¬†networks and¬†interoperability/integration solutions.
  • The customer portal/tools – This can be from a simple status/account/billing view¬†up to a self-provisioning, self-management
  • Ticket / support handling –¬†five customers might be easy to deal with phone calls and emails, but what about 30,100,…?

Softwaresoftware-icon-150x150

I include on this section separately, because not everybody is aware of some legal aspects. Let’s take a look at Microsoft products: you cannot just buy a Windows license, install on your server and charge¬†it to one (or more)¬†customer(s) for a running service there.
Microsoft is clear on this: if you are a Hosting/Service Provider, you need to buy licensing throw a SPLA .
Like Microsoft, VMware and many other vendors provide (or enforce) this model and is not a bad option:

  • The advantage is that it allows you to pay monthly for what you really use –
    this is the pay-as-you-use model
  • The inconvenient is that you need to report the usage periodically¬†and¬†allow auditing to your business

My advice is:

  • as a Service Provider, contact¬†the vendor and explain your intentions. They will help you to find the most profitable solution… most of the times.
  • ¬†Don’t try to find ‘loopholes’…trust me: it will cost you much more later!

xml-formShared services

Here you put all the platforms and services that can be shared between multiple.
Great examples are web servers hosting multiple websites. But pushing your skills to the limit, you can have a lot more. Multitenant solutions also would fit on this group.

Dedicated your best resources planning them! This is where your cost savings make a difference.

140912_cloud_phoneDedicated services

This is what your best customers are looking and willing to pay for.
It’s your cash cow – the more you have here, the quicker your revenue increases. These business models have more opportunities, specially¬†combined with standardized offers.

It has every group of systems serving unique customers:

  • It should be a¬†‘block model’¬†– same deployment and standardized procedures, automation and self service tools,¬†will keep operational costs low and predictable.
  • But you can also include very ‘$pecific $olutions’¬†– these one¬†gives you the¬†opportunity to upsell¬†consulting and managed services.

Skype for Business services model¬†would¬†fit¬†on this group… but it might also¬†fit for the group above? ūüėČ

desperate-business-broker-face-bad-investment-concept-48317142Not there yet

If you already own a company then this is known to you:¬†Work office place, furniture, energy, personal computers, HR, billing / account management, mobile devices, transportation/gas… you also pay for that, right

All done! What now?

presentation-screen-with-business-activities-download-royalty-free-vector-file-eps-14696After joining and calculating all the pieces, test yourself doing at least these questions:

  • What is the cost per service/per user? How much a VM with a specific size costs?
    This is the lowest value will charge for your service.
    => Capitalism rule: to earn money you need to sell it for more than you pay for
  • What is¬†the break even point?
    simulate, over and over the time! Be ready to answer: how many customers (or users) do you need to have to cover all the costs made so far?
    Whomever is going to put money on this will make do that question before writing the checks (even yourself if you got the money)
  • What is the ROI?
    Investors, banks or stake holders will look for this (and many other tools).
    If you need financing you need to promise when and how much will you pay back.
  • Psychological one: are you an entrepreneur?
    It needs investment analysis, financial control, technical know-how, purchasing/selling/negotiation skills, HR management, and so on. As the business creator you need deal with of them.
    This will not be a one-man-show for long. Sooner or later you need help trust and delegate to others. Starting with someone it will look less difficult.

This might seem directed to start-ups guys, but it applies to existing companies. Product and/or Business developers need also to assume the above challenges, the problems, assume risks and responsibilities.

You are putting you neck and reputation on it and get ready to accept failure.
But of you planed all (not just a suicidal gambler), showed you the balls, and if successfully, the personal reward is priceless.

It’s still not over! there are some more strategic decisions for¬†a go-to-market, on part 2 (writing in progress).

 

The minimal amount of servers for a full HA Enterprise Pool: less than 4

By the end of 2015, I started enumerating some challenges that the Lync/Skype topology presents when you just need minimal resources and provided some answers. Since the beginning of Microsoft UC and the more I dig inside the product versions, I came with some out-of-the-box personal challenges:
–¬†4/Feb/2012 – Installed¬†a Standard Edition Lync 2010¬†in a Domain controller (DNS, CA) – LyncIn1Box
–¬†11/Dec/2016 – Installed Skype for Business 2015¬†in a Windows Server Core – The smallest Skype for Business front-end¬†server

As strange as it sounds, there are some small companies around the world¬†that depend¬†on¬†Lync/Skype¬†for critical business mission but don’t have enough resources.¬†Companies with small amount of users can go for¬†Office365, but¬†it might no¬†cover all the features of the on-premises. Pool pairing (1 STD Edition + 1 STD edition)¬†might also not be possible if there is no feasible secondary location.

A short technical review of what roles you need to cover for High Availability (HA):
* Front End Lync/Skype server roles
* Back-end databases  Рyou need a SQL mirror or a Cluster/Always-On
* File Share Рcan be provided by a DFS or a File Share cluster

Now we can start asking: how many resources do I need to deploy the above HA ?
Here some possible answers. Note that ‘something’ means that a SQL HA needs¬†2 Instances and a sort of a witness (either a SQL server or a File Share/disk quorum)

Microsoft recommended: 5.x (5 servers + something)

Following Microsoft recommended practices, you will need 3 front-end servers and a redundant back-end database. File Share can be easily deployed on the Back-end servers.

MS-recommend-EE

Microsoft minimum: 4.x

As Microsoft allow the usage of two Front end server, but: “This small pool will not provide a robust high-availability solution like a larger pool would, and needs extra care in managing“.

MS-minimal-EE
My recommended PoC: 3.0

By joining all the roles across 3 windows servers, you can have one server down and the built-in automatic failover mechanisms will take care of that.

MY-recomended-EE

My minimum PoC : 2.x (2 servers + something)

Redundant means at least two systems, so this is the absolute minimum :). It has the same risks as Microsoft states for two servers.

MY-minimal-EE
The main reason is the fabric quorum: If you unexpectedly loose a server, the fabric cannot elect the owner of the several roles… even if there is only that server. There will be the need of a ‘soft’ quorum reset (manually or a triggered task)
But if you shutdown a server/services gracefully all services will be transferred to the working node.

About the 2.x equation:
* If you use a SQL mirror, a SQL witness is required (which actually means an additional/existing Windows server)
* If you use SQL clustering/Always On,¬†you¬†will need: (1)¬†a file share witness¬†-doesn’t have to be a Windows but can count as a server :)- (2) a quorum disk (share storage) – and this is the ‘true’ 2.x

Final thoughts

Deploying this ‘minimalist’ solution¬†is more about ‘out-of-the-box-thinking’ than tweaking (actually it’s only one). The normal Skype for Business setup wizard will install normally all the roles defined on the topology.

Notes about this ‘minimal counting’:
* It’s about adding Skype services:¬†it assumes that you have a Windows Domain, a CA and a sort of Load Balancer.
* SQL Clustering/AlwaysOn is still on test stage, but the service failover should behave similar as the Mirroring.
* If you already have a SQL HA in place at the datacentre, you can reuse it. This will make MY and MS minimal requirements equal to 2.
* At this time I excluded the Edge, Office Web App roles or Persistent Chat. The first one will required more resources, the second role… is for an updated post ūüėČ

Logging scenarios that crash and cripple the RTCCLSAGT

Another quest with an issue that appears to be there since some time of Skype for Business 2015 server (if not even Lync server. I decided to find out and document it.

ISSUE

Consider the following scenario:
1) You start the Skype for Business logging tool
2) Pick one of the following builtin scenarios (CmdletDebug or IISLog)
LoggingIIS-crash3) As soon as you start this scenarios on the selected servers you will notice the following error message on the logging tool output window: “ResponseMessage: Error code – 20000, Message – Unknown error – Error calling agent <FE-FQDN>; Could not connect to net.tcp://<FE-FQDN>:50002/. The connection attempt lasted for a time span of 00:00:02.0071248. TCP error code 10061: No connection could be made because the target machine actively refused it 10.101.128.20:50002. . Please refer CLS logs for details.”

RTCCLSAGT-eventerrorBy this time you will notice that the (Skype for Business Server Centralized Logging Service Agent) RTCCLSAGT service has crashed on all the servers you initiated the trace, with the following event error id 33040:
Centralized Logging Service Agent Error starting background thread to process traces.
Log Type – IISLogManager, Error – Object reference not set to an instance of an object.
Cause: Internal error
Resolution: Examine error details to determine resolution.

Worst, the service will crash every time you try to start it with the same error. You find yourself without any logging capability on your servers!

CAUSE

The reason is that the two scenario contain invalid trace components called ‘Internal’ and ‘External’. Somehow this triggers an internal failure on the IISLogManager component and damaged it.
When you edit those scenarios, you will notice a warning about the components
Unknown-traces

SOLUTION(s)

This is how you can fix the two problems:

  • Repair the failed service start (no server downtime required)
    Run the repair of the Skype for Business 2015, Core Components.
    S4B-needNetFramework
    Pay attention that two services need to be set to Automatic (Delayed Start) or it may fail on Windows 2012 Servers (as the above picture). Just noticed the instructions on my previous post.
  • Fix the scenarios to prevent from happening again:
    1) notice the valid components and flags.
    2) delete the scenario
    3) create a scenario with the same name and all the components (excluding the ‘External’ and ‘Internal’)

Cannot repair/reinstall Skype for Business core components on Windows 2012 Server

There are some specific situations where you need to repair or install an a new component of Skype for Business 2015 server. Because of a corruption, I only faced this very recently, but it’s there since November 2016.

ISSUE

You have the following conditions on your Skype for Business server:
– Windows 2012 Server with .Net Framework 4.5.1 installed (with or without the most recent updates)
– Skype for Business 2015 Nov/2016 cumulative update (or later) installed

You are unable to run the repair option of Skype for Business 2015, Core Components with the error message: ‘…installation requires Microsoft .Net Framework 4.5. Installation cannot continue.’
S4B-needNetFramework.png

CAUSE

The Nov/2016 cumulative update added a new ‘Launch Condition’ to the .msi installation:
– before it validates the existence of .Net Framework 4.5
PATCH Or MSIPATCHREMOVE Or REMOVE Or NETFRAMEWORK45 OR SKIP_NETFRAMEWORK_CHECK
– now, it checks for the existence of .Net Framework 4.5.2 (releasecode 379893)
PATCH Or MSIPATCHREMOVE Or REMOVE Or (NETFRAMEWORK45 >= “#379893”) OR SKIP_NETFRAMEWORK_CHECK

ocscore-dotnet-msi.png

SOLUTIONS

I’ve identified 3 possible solutions:

A) The ‘SysAdmin Pro’
1. Manually run the following command line (with the skip .Net check):
msiexec /i {DE39F60A-D57F-48F5-A2BD-8BA3FE794E1F} SKIP_NETFRAMEWORK_CHECK=1
2. Follow the setup GUI and choose the repair option
Note: the msiexec repair option ignores the custom parameters and will give the error

B) The ‘sneaky sysadmin’
1. Take ownership (and grant write permissions) of the registry key
HKLM\SOFTWARE\Wow6432Node\Microsoft\NET Framework Setup\NDP\v4\Full
2. Change the REG_DWORD key ‘Release’ to a value equal or greater than 379893
netFrameworkSetupReg
3. Run the repair option
4. Change the REG_DWORD key ‘Release’ to the original value

C) The ‘let’s upgrade and see’ sysadmin’
1. Update .Net Framework to version 4.5.2
2. Run the repair option
NOTE: This was not tested, but it’s recommended at least to rerun repair option on all Skype Web-related components. The main advantage is that is a permanent solution… pr at least until MS release another CU requiring 4.6 ūüėČ

Final notes and caveats

I could not find the reason for this change documented:
* Skype for Business per-requisites still points to .Net Framework 4.5
  (Skype for Business runs fine on a Windows 2012 server with .Net Framework 4.5)
* The recent cumulative updates don’t push/required an update to .Net Framework 4.5.2

Important: After you repair the Skype for Business 2015 Core components, it disables the two associated windows services:
* RTCCLSAGT (Skype for Business Server Centralized Logging Service Agent)
* REPLICA (Skype for Business Server Replica Replicator Agent)
Just make sure that you set the services back to Automatic (Delayed Start)