Logging scenarios that crash and cripple the RTCCLSAGT

Another quest with an issue that appears to be there since some time of Skype for Business 2015 server (if not even Lync server. I decided to find out and document it.

ISSUE

Consider the following scenario:
1) You start the Skype for Business logging tool
2) Pick one of the following builtin scenarios (CmdletDebug or IISLog)
3) As soon as you start this scenarios on the selected servers you will notice the following error message on the logging tool output window: “ResponseMessage: Error code – 20000, Message – Unknown error – Error calling agent <FE-FQDN>; Could not connect to net.tcp://<FE-FQDN>:50002/. The connection attempt lasted for a time span of 00:00:02.0071248. TCP error code 10061: No connection could be made because the target machine actively refused it 10.101.128.20:50002. . Please refer CLS logs for details.”

By this time you will notice that the (Skype for Business Server Centralized Logging Service Agent) RTCCLSAGT service has crashed on all the servers you initiated the trace, with the following event error id 33040:
“Centralized Logging Service Agent Error starting background thread to process traces.
Log Type – IISLogManager, Error – Object reference not set to an instance of an object.
Cause: Internal error
Resolution: Examine error details to determine resolution.“

Worst, the service will crash every time you try to start it with the same error. You find yourself without any logging capability on your servers!

CAUSE

The reason is that the two scenario contain invalid trace components called ‘Internal’ and ‘External’. Somehow this triggers an internal failure on the IISLogManager component and damaged it.
When you edit those scenarios, you will notice a warning about the components

SOLUTION(s)

This is how you can fix the two problems:

• Repair the failed service start (no server downtime required)
  Run the repair of the Skype for Business 2015, Core Components.
  
  Pay attention that two services need to be set to Automatic (Delayed Start) or it may fail on Windows 2012 Servers (as the above picture). Just noticed the instructions on my previous post.
• Fix the scenarios to prevent from happening again:
  1) notice the valid components and flags.
  2) delete the scenario
  3) create a scenario with the same name and all the components (excluding the ‘External’ and ‘Internal’)

Cannot repair/reinstall Skype for Business core components on Windows 2012 Server

There are some specific situations where you need to repair or install an a new component of Skype for Business 2015 server. Because of a corruption, I only faced this very recently, but it’s there since November 2016.

ISSUE

You have the following conditions on your Skype for Business server:
– Windows 2012 Server with .Net Framework 4.5.1 installed (with or without the most recent updates)
– Skype for Business 2015 Nov/2016 cumulative update (or later) installed

You are unable to run the repair option of Skype for Business 2015, Core Components with the error message: ‘…installation requires Microsoft .Net Framework 4.5. Installation cannot continue.’

CAUSE

The Nov/2016 cumulative update added a new ‘Launch Condition’ to the .msi installation:
– before it validates the existence of .Net Framework 4.5
PATCH Or MSIPATCHREMOVE Or REMOVE Or NETFRAMEWORK45 OR SKIP_NETFRAMEWORK_CHECK
– now, it checks for the existence of .Net Framework 4.5.2 (releasecode 379893)
PATCH Or MSIPATCHREMOVE Or REMOVE Or (NETFRAMEWORK45 >= “#379893”) OR SKIP_NETFRAMEWORK_CHECK

SOLUTIONS

I’ve identified 3 possible solutions:

A) The ‘SysAdmin Pro’
1. Manually run the following command line (with the skip .Net check):
msiexec /i {DE39F60A-D57F-48F5-A2BD-8BA3FE794E1F} SKIP_NETFRAMEWORK_CHECK=1
2. Follow the setup GUI and choose the repair option
Note: the msiexec repair option ignores the custom parameters and will give the error

B) The ‘sneaky sysadmin’
1. Take ownership (and grant write permissions) of the registry key
HKLM\SOFTWARE\Wow6432Node\Microsoft\NET Framework Setup\NDP\v4\Full
2. Change the REG_DWORD key ‘Release’ to a value equal or greater than 379893

3. Run the repair option
4. Change the REG_DWORD key ‘Release’ to the original value

C) The ‘let’s upgrade and see’ sysadmin’
1. Update .Net Framework to version 4.5.2
2. Run the repair option
NOTE: This was not tested, but it’s recommended at least to rerun repair option on all Skype Web-related components. The main advantage is that is a permanent solution… pr at least until MS release another CU requiring 4.6 😉

Final notes and caveats

I could not find the reason for this change documented:
* Skype for Business per-requisites still points to .Net Framework 4.5
  (Skype for Business runs fine on a Windows 2012 server with .Net Framework 4.5)
* The recent cumulative updates don’t push/required an update to .Net Framework 4.5.2

Important: After you repair the Skype for Business 2015 Core components, it disables the two associated windows services:
* RTCCLSAGT (Skype for Business Server Centralized Logging Service Agent)
* REPLICA (Skype for Business Server Replica Replicator Agent)
Just make sure that you set the services back to Automatic (Delayed Start)

Warning: May2017 windows update break Lync 2010,2013 and Skype for Business 2015 services

This is a quick post notice since I’m still trying to analyse the cause, and more Engineers are reporting the same issue.

ISSUE

 After you install the latest Windows May2017 updates, Skype for Business 2015 will start reporting the following errors:

• External Users reported that couldn’t use WhiteBoard, Polls, Q&A or present PowerPoint with the following errors messages:
  We can’t connect to the server for sharing right now.
  Network issues are keeping you from sharing notes and presenting whiteboards, polls and uploaded PowerPoint files.
  
  

  

   

• Front-end servers: (Event ID 41026/41025) “No connectivity with any of Web Conferencing Edge Server, External Skype for Business clients cannot use Web Conferencing modality”
• Edge servers : “Web Conferencing Server connection failed to establishOver the past 3 minutes Skype for Business Server has experienced incoming TLS connection failures 1 times(s). The error code of the last failure is 0x80072746”
  or event id 41026
  

Cause

The issue seems to be some secure hardening on the TLS negotiation.
According to MS Support, it looks like an ‘update blooper’ on the May2017 patch:
“This update adds an additional check on Enhanced Key Usage (EKU), since all Lync/ SfB Server usually use the Web Server template they will only have the Server Authentication in the EKU.“

Impact and affected systems

Impact level: low. Web conferencing service (whiteboard, powerpoint presentations ).

 According to MS: “The issue has been reproduced on Lync 2010, Lync 2013 and Skype for Business 2015 on all supported server versions (2008r2, 2012, 2012r2).”

Workarounds

– uninstall the May 2017 Security and Quality rollup for the .Net Framework 4.5.2
– Request new Edge Internal certificate with the Client and Server Authentication
– On the Front Ends disable the check for the Web Conferencing Service (follow this post for details on how to do it)

Update 18th May 9:30 –  Update content based on this post (thank you Erdal for pointing me to that blog)

Update 24th May 6:30 – Updated with end-user error messages and picture. Microsoft published KB4023993 describing the issue and the workarounds described above

Skype4B interoperability: Cisco Expressway

Cisco Expressway “offer users outside your firewall simple, highly secure access to all collaboration workloads, including video, voice, content, IM, and presence. Collaborate with people who are on third-party systems and endpoints or in other companies. Help teleworkers and Cisco Jabber mobile users work more effectively on their device of choice. Cisco Expressway allows you to do all this and more.” (source)

Using Skype for Business terminology, this is the “Edge Server” of Cisco. But it besides the described functions, it has one that allows you the integrate Cisco Video endpoints (desktop client, video conference rooms up to telepresence system) with Lync/Skype for Business clients that allows SIP dialling from both sides with audio and HD video: search for the contact and press the call button.

Skype user view

Cisco meeting room view

 

The infrastructure configuration is based on a ‘dedicated expressway’ configured as a Skype for Business ‘trusted application’. The rest is all about SIP domain ‘static routing’.

Expressway topology for on-premises Skype for Business and Cisco video endpoints 

Great features of this integration:

• IM & presence is possible between systems;
• Desktop sharing is supported;
• Both Cisco and Microsoft can share a single SIP domain;
• Skype for Business remote clients can use the Expressway on the DMZ (TURN server) for audio traffic with Cisco endpoints;
• Skype for Business clients can make and receive audio/video calls with external parties whom the Cisco has federation with.

There are also some limitations:

• Presence is based on SIP simple. Only can have the status ‘Available’ and ‘Busy’;
• Skype for Business clients can join a multipoint conference in Cisco, but Cisco endpoints cannot join Skype meetings.

 

In conclusion, although it’s not a better solution than Polycom, the Expressway leverages the existing investment  already done on Cisco infrastructure and can be extended to a complete 3rd interoperable federation solution:

Lync (Skype for Business) November 2016 update – duplicate IM’s

ISSUE

On November 2016, Lync 2013 (Skype for Business 2015) customers start reporting cases of  duplicate of IM messages.

CAUSE

The issue started appearing after the installation of November 2016 client update (KB3127934) and it affects the receiver only.

Steps to recreate the issue (credits go to Alex) :

1. Send an IM to an user
2. Do not open the IM toast on the receiving user
3. Send another (or more) IM the that user
4. Open the notification toast of the receiving user and you will noticed that only the first IM line sent is not repeated

The issue does not occur when the IM conversion window is open.

Applying the December 6, 2016, update for Skype for Business 2015 (Lync 2013) (KB3127976) also causes the same issue.

Skype for Business 2016 clients are also affected on the same way by their corresponding monthly cumulative updates. There is an oldest thread on Microsoft community that was reopened by another 2 persons that report the same behaviour and uninstalling the KB3127934 would solve the issue.

SOLUTION

Apply the January 3, 2017 update for Skype for Business 2015 (Lync 2013) (KB3141468) or Skype for Business 2016 (KB3128049).
It is mentioned on the resolved issue list (no root cause provided):
“Assume that you send continuous instant messages (IMs) to a user in Microsoft Skype for Business 2015 (Lync 2013). Then you allow the toast notification window to be auto accepted. In the conversation window, you find every item but first gets duplicated. Also, in the Conversation History in Microsoft Outlook, you find that the conversation window shows duplicated IMs. “

I successfully tested and confirmed that it solves.

Skype4B 2015 quick tip: keep debugging tools automatically updated

As you know the Debugging Tools is a separate installation product published a few weeks/months from the initial Lync/Skype4B. This tool is fundamental on troubleshooting the platform.

It contains the information required to decode the debug traces of every component on two files: ‘default.tmx’ and ‘default.xml’, included and installed by the debugging tools package.
But they are also on the Lync/Skype4B package installation and all cumulative updates. Every new update/feature might require the ‘decoder’ to support the updated component. So, if you install the debug tools they are outdated and you might not be able to decode new features, partial logs line or even none.

The information on how to update them it’s referenced on the main cumulative update page  KB2809243 for Lync 2013 (doesn’t exist for Skype for Business 2015, but it’s the same principle):
“Debugging tools require the latest version of the Default.TMX file that is included in each Cumulative Update to properly decrypt logs files. In order to keep … Debugging Tools updated, you will need to browse to the “C:\Program Files\…\Tracing” folder, and copy the default.tmx and default.xml files to the install location of Lync Debugging Tools. The default location is C:\Program Files\…\Debugging Tools\.“

But there’s a much efficient and automated way to do this. Instead of copying the 2 files on every cumulative update, just replace them with an ‘symbolic links’ to the main Lync/Skype4B location:

• Delete the default.tmx, default.xml file on the debug tools installation folder;
• Create a symbolic link for each file (command line), ex:
  MKLINK “<Debugging Tools install folder>\default.tmx” “%CommonProgramFiles%\Skype for Business Server 2015\Tracing\default.tmx”
  MKLINK “<Debugging Tools install folder>\default.xml” “%CommonProgramFiles%\Skype for Business Server 2015\Tracing\default.xml”

Note: do not mix ‘symbolic link’ with a ‘shortcut’. The debug tools (and any other application) will not support the second option.

Every time the you run cumulative update package, the debugging tools will be pointing to the most up-to-date (for sure).

Hope you can find this simple trick useful. 😉

No Lync 2013 server updates available

If you are looking for a cumulative update for Lync 2013 servers you will not find any available since 6 December 2016.

I will find the download page KB2809243 with the following message:
“An issue was discovered in the Lync Server 2013 November 2016 Update (build 8308.974) that causes contact searches on mobile clients to return no results. Because of this issue, the November 2016 Update is no longer available for public download. The Skype for Business team is working on a fix that is scheduled to be delivered soon in a new update.“

Which complicates your Sysadmin life, since there were some the important fixes/updates:

• KB3204553 – Lync Server 2013 adds support for Skype for Business for Mac
• KB 3204552 – Skype for Business mobile clients don’t show telephone numbers for some users on contact card
• KB 3204547 – You can’t join a meeting from Safari or Firefox through a Lync Server 2013 Mac app
• KB 3204546  – (again this one) You can’t join a meeting from outside Skype for Business or Lync on iOS 10.0 and later versions

Update (13/12/2016): The cumulative update is available again for download. It has an updated published date, but no information that it replaces the one released two weeks ago.

Update (14/12/2016): Microsoft release KB3212869 identifying the November 2016 CU (8308.974) issue whose solution is to download and deploy this undocumented CU (8308.977)
I guess that if you ran it, will update any previous installed version… and one more ‘extra patching planning’ to perform.