Lync (Skype for Business) November 2016 update – duplicate IM’s

ISSUE

On November 2016, Lync 2013 (Skype for Business 2015) customers start reporting cases of  duplicate of IM messages.

CAUSE

The issue started appearing after the installation of November 2016 client update (KB3127934) and it affects the receiver only.

skyp4b-kb3127934-issue3

duplicate-imsSteps to recreate the issue (credits go to Alex) :

  1. Send an IM to an user
  2. Do not open the IM toast on the receiving user
  3. Send another (or more) IM the that user
  4. Open the notification toast of the receiving user and you will noticed that only the first IM line sent is not repeated

The issue does not occur when the IM conversion window is open.

Applying the December 6, 2016, update for Skype for Business 2015 (Lync 2013) (KB3127976) also causes the same issue.

Skype for Business 2016 clients are also affected on the same way by their corresponding monthly cumulative updates. There is an oldest thread on Microsoft community that was reopened by another 2 persons that report the same behaviour and uninstalling the KB3127934 would solve the issue.

SOLUTION

Apply the January 3, 2017 update for Skype for Business 2015 (Lync 2013) (KB3141468) or Skype for Business 2016 (KB3128049).
It is mentioned on the resolved issue list (no root cause provided):
“Assume that you send continuous instant messages (IMs) to a user in Microsoft Skype for Business 2015 (Lync 2013). Then you allow the toast notification window to be auto accepted. In the conversation window, you find every item but first gets duplicated. Also, in the Conversation History in Microsoft Outlook, you find that the conversation window shows duplicated IMs. “

I successfully tested and confirmed that it solves.

Skype4B 2015 quick tip: keep debugging tools automatically updated

As you know the Debugging Tools is a separate installation product published a few weeks/months from the initial Lync/Skype4B. This tool is fundamental on troubleshooting the platform.

debugging-tools

It contains the information required to decode the debug traces of every component on two files: ‘default.tmx’ and ‘default.xml’, included and installed by the debugging tools package.
But they are also on the Lync/Skype4B package installation and all cumulative updates. Every new update/feature might require the ‘decoder’ to support the updated component. So, if you install the debug tools they are outdated and you might not be able to decode new features, partial logs line or even none.

The information on how to update them it’s referenced on the main cumulative update page  KB2809243 for Lync 2013 (doesn’t exist for Skype for Business 2015, but it’s the same principle):
Debugging tools require the latest version of the Default.TMX file that is included in each Cumulative Update to properly decrypt logs files. In order to keep … Debugging Tools updated, you will need to browse to the “C:\Program Files\…\Tracing” folder, and copy the default.tmx and default.xml files to the install location of Lync Debugging Tools. The default location is C:\Program Files\…\Debugging Tools\.

But there’s a much efficient and automated way to do this. Instead of copying the 2 files on every cumulative update, just replace them with an ‘symbolic links’ to the main Lync/Skype4B location:

  • Delete the default.tmx, default.xml file on the debug tools installation folder;
  • Create a symbolic link for each file (command line), ex:
    MKLINK “<Debugging Tools install folder>\default.tmx” “%CommonProgramFiles%\Skype for Business Server 2015\Tracing\default.tmx”
    MKLINK “<Debugging Tools install folder>\default.xml” “%CommonProgramFiles%\Skype for Business Server 2015\Tracing\default.xml”

debugtools-tip

Note: do not mix ‘symbolic link’ with a ‘shortcut’. The debug tools (and any other application) will not support the second option.

Every time the you run cumulative update package, the debugging tools will be pointing to the most up-to-date (for sure).

Hope you can find this simple trick useful. 😉

No Lync 2013 server updates available

If you are looking for a cumulative update for Lync 2013 servers you will not find any available since 6 December 2016.

I will find the download page KB2809243 with the following message:
An issue was discovered in the Lync Server 2013 November 2016 Update (build 8308.974) that causes contact searches on mobile clients to return no results. Because of this issue, the November 2016 Update is no longer available for public download. The Skype for Business team is working on a fix that is scheduled to be delivered soon in a new update.

Which complicates your Sysadmin life, since there were some the important fixes/updates:

  • KB3204553 – Lync Server 2013 adds support for Skype for Business for Mac
  • KB 3204552 – Skype for Business mobile clients don’t show telephone numbers for some users on contact card
  • KB 3204547 – You can’t join a meeting from Safari or Firefox through a Lync Server 2013 Mac app
  • KB 3204546  – (again this one) You can’t join a meeting from outside Skype for Business or Lync on iOS 10.0 and later versions

Update (13/12/2016): The cumulative update is available again for download. It has an updated published date, but no information that it replaces the one released two weeks ago.

Update (14/12/2016): Microsoft release KB3212869 identifying the November 2016 CU (8308.974) issue whose solution is to download and deploy this undocumented CU (8308.977)
I guess that if you ran it, will update any previous installed version… and one more ‘extra patching planning’ to perform.

The smallest Skype for Business front-end server

There were some reasons that took me a week on this project:
– I have few resources on my personal lab (specially storage);
– take my knowledge on OS, Skype4B deployment to the limit.

 If I already ‘get on the nerves’ of customers and colleagues when I request and deploy servers with less then 100GB of HDD,  I can imagine Microsoft with the minimum requirements of 72GB of free disk space (not including the OS?).

My current standard uses a Windows 2012 server with a total of 55GB split between 3 HDD. The Operating System (Drive C) and Skype for Business Front-End (Drive D) take around 36 GB.
skype4b-win2012-size

How much smaller can you have the same Front-end server?  around 18GB (*)
skype4b-win2012r2core-size
(*)not counting the space for IIS logs, Windows Fabric traces and the Page file

The answer for this the same for the question: Can I deploy Skype for Business on a Windows 2012 R2 server core?
Here’s some good reasons to use the Windows server core edition as Microsoft describes:
– less disk space and ram consumption;
– Reduced attack surface (no GUI and less OS vulnerabilities).

In fact the core edition has 98% of the installation prerequisites for Skype for Business Server 2015. On this post I will enumerate the challenges you face if trying to do these. Some are real challenges, others are just glitches of the main Skype4B setup.

Windows Identity Foundation 3.5 (WIF)

This is one prerequisite that you will get an error, and Microsoft KB clarifies that you will not be able to install without installing 4GB of the minimal server interface. All this to get 7 small outdated files that are supposed to be included on the .Net framework 4.5 (included natively on Windows 2012 R2 Server).
In fact that is even described on the OS package:
Microsoft-Windows-Identity-Foundation-Package~31bf3856ad364e35~amd64~~6.3.9600.16384.mum: “Windows Identity Foundation (WIF) 3.5 is a set of .NET Framework classes that can be used for implementing claims-based identity in your .NET 3.5 and 4.0 applications. WIF 3.5 has been superseded by WIF classes that are provided as part of .NET 4.5. It is recommended that you use .NET 4.5 for supporting claims-based identity in your applications.

NOTE: bootstrapper.exe doesn’t validate if WIF is installed on the prerequisites stage. You will only get an installation failure at the package MicrosoftIdentityExtensions.msi.

The workaround is about being able to ‘add-package’  above 😉

IIS Management console

This is a ‘strange prerequisite’. Why do you need the IIS management console snap-in (MMC) to install/run a Skype4B ?
missing-iis-mmc
MMC support is only available installing the  minimal server interface or you will get an error when trying to install it: Add-WindowsFeature Web-Mgmt-Console.
Workaround: just provide the key that bootstrapper looks for by adding the REG_DWORD  value ‘ManagementConsole‘ to the ‘HKLM\SOFTWARE\Microsoft\InetStp\Components‘ key. You can even set to zero (not installed) since it only checks for its existence.

Media Foundation

This is a little more ‘ridiculous’. You can install media foundation on windows core:
dism.exe /online /enable-feature /featurename:ServerMediaFoundation /all
but even it appears as installed on the get-windowsfeatures, the bootstrapper will report missing
missing-mediafoundataion

The reason is that it’s checking for a different installed component: ‘Server-Gui-Shell‘ which is another additional extra to the  minimal server interface

Workaround: add the REG_DWORD value ‘Server-Gui-Shell’ (must be 1) to the ‘HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Server\ServerLevels‘ key.

The last ‘twist’

By this moment you managed to install all the Skype4B front-end components. You managed to start the main service (RTCSRV) but the ones who rely on audio (ex: RTCAVMCU, RTCCAA) and remote data access (ex: RTCDATAMCU). The reason is that 7 required dll’s are only included on the Windows server standard edition:
– DirectX11 and real-time media handlers;
– Remote Access handling.

Workaround: as soon as you get a copy of the missing 7 dll files, you manage to start the remaining Skype4B services and you now have a fully operational Front-end server!

Conclusion

From the description above, the big reason that Microsoft doesn’t support Skype4B on a Windows server core is 7 dll files that are not able to be separated from the install edition.

Off course by now, you can see that this is an option for functional testing in LAB or demos. Microsoft will never support this, even if there is a way to install all the missing parts using the several windows setup command lines available.

The other no-go would be the administration/operations team: There will be a ‘revolution’ if people find out that there was no Windows GUI to manage a server (although you can manage servers remotely with a full GUI ‘management server’.

As a last comment: using the ‘MS-approved’ Windows server, I will let you know that it’s possible to run using a Windows 2012 R2 with a little less than 30GB of HDD.
skype4b-win2012r2-size
…but there’s still room to squeeze a little more 😉

Lync/Skype4B embedded links exploit

I decided to share this MSitPros blog post to show how can you exploit a Lync/Skype4B rich IM, using embedded links with SMB shares.

careful
As stated by the author, exploiting for the NTLM hash might be less successful from an external attacker (SMB traffic blocking), but a rogue LAN user or a deceiving ‘hotspot provider’/’internet cafe’ might would try this one.

Rich text IM (rich fonts, embedded pictures and links) is a very nice feature of Lync/Skype4B but it is also where the common MS Office security issues are found:

  • MS16-039: Security update for Microsoft Graphics Component: April 12, 2016
  • MS16-097: Security update for Microsoft Graphics Component: August 9, 2016
  • MS15-116: Security update for Microsoft Office to address remote code execution: November 10, 2015

Don’t panic right way if you have a full control/security policies of your LAN users, so that no one can just plug a rogue device (or install the required exploit software on his work PC).
The attacker must be able to reach the user – either he has an internal Lync/Skype4B account (which means he already might have hacked the network), or using Company or Skype federation.
Even if the attacker get the hash, the next step is to use against a server resource to access. An external attacker will have an additional challenge to reach your internal LAN.

Just like using Outlook, be careful when opening links or attachments. Better ways to prevent this:
– block links on IM (at least for federations)
– use only the NTLMv2 or Kerberos authentication protocols (although there are known ways to exploit them the same way)

My keynote is that security is an important topic when planning and deploying Lync/Skype for Business… don’t just go for a plain next>next>ready installation.

Call Quality Dashboard – Part 3: The Portal

After describing the Call Quality Dashboard (CQD) QoE Archiving Database and the QoE CUBE, I will show now how to install the Portal component and how it works on the solution.

The CQD Portal is “where users can easily query and visualize QoE data.” synchronized by the Archive and processed by the CUBE.
ic841926The CQD Portal is a IIS based web application that allows you not just visualized but create new reports, views and assign permissions to them. As the above picture shows, it relies on a SQL database to keep all the information.

Installing CQD – Portal

Before performing the installation, the following pre-requisites need to be in place:

  • You need a SQL Databases Services (dedicated or existing) for the setup to install the Portal support database.
  • On the server that will host the Portal you need to install IIS. The following powershell command will install all the required components:
    Add-WindowsFeature Web-Server, Web-Static-Content, Web-Default-Doc, Web-Asp-Net, Web-Asp-Net45, Web-Net-Ext, Web-Net-Ext45, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Http-Logging, Web-Url-Auth, Web-Windows-Auth, Web-Mgmt-Console  -verbose
  • A dedicated domain service account is recommended to can grant the least required privileges. If you installed all the components on the same server you can use the local built-in server account but, if have the SQL Database/Analysis services (CUBE) deployed on a different  servers, the account is required.
  • The QoE Archiving and the CUBE needs to be already deployed.

The installation package is the same for all CQD components so, if: (a) you are installing all components you can go to step 2; (b) if you already installed the QoE Archiving and/or the CUBE on the same server, go to ‘programs and features’ and ‘change’ the package and proceed to step 2:

  1. Proceed throw the welcome screen, licence agreement, and choose the binaries install location:
  2. For this part, I will select the Portal and proceed to the configurations screen:

    Configuration options:
    sqlname-vs-instance QoE Archive SQL Server: SQL Server instance name for where the QoE Archive database is located.
    Cube Analysis Server: SQL Server Analysis Service instance name for where the cube is located.
    Repository SQL Server: SQL Server instance name where the Repository database is to be created.
    IIS App Pool User – User Name & Password: The account that the IIS application pool should execute under and access the other components. You can choose one of the local server services account, otherwise choose ‘Other’ and provide a domain service account credentials (see pre-requisites above explanation).

  3. After the validations the installation will ask to proceed until completion, hopefully without any error 🙂

Behind the CQD Portal

What happened and was configured after the previous installation steps?
This component setup  installed some specific files, created support database and made some updates on the QoE CUBE Database:
• QoERepositoryDb database was created. This database holds the portal all the configurations, customized reports, …
• ‘IIS App Pool User’ login created and assigned db_owner on the QoERepositoryDb
• ‘IIS App Pool User’ login created and assigned db_datareader on the QoEArchive database
• ‘IIS App Pool User’ added to the QoERole on the CUBE database
• IIS default web site configured with 3 folders that matches the directories and files installed.

Known ‘caveats’ regarding the installation and architecture process:

  • In rare cases, the installer fails to create the correct settings in IIS. Manual change is required to allow users to log into the CQD. If users are having trouble logging in, please follow the steps described on ‘know issues’ section of the  TechNet article.
  • Cube Sync Fails – QoEMetrics may contain some invalid records based on end user clocks. If the time skew is greater than 60 yrs, the cube import will fail. Check the Min and Max StartTime/EndTime using the selections below. Look for and delete records in the far past and very distant future, they can be disregarded and they will break up the sync processes.
    Select MIN(StartTime) FROM CqdPartitionedStreamView
    Select MAX(StartTime) FROM CqdPartitionedStreamView
    Select MIN(EndTime) FROM CqdPartitionedStreamView
    Select MAX(EndTime) FROM CqdPartitionedStreamView
  • After deploying the CQD on a new server, you can run into a problem where the Portal was not showing any data and returned a problem saying:
    We couldn’t perform the query while running it on the Cube. Use the Query Editor to modify the query and fix any issues. Also make sure that the Cube is accessible
    In order to solve it, process the CUBE object and make sure it’s accessible as described here.

How to manage and monitor the CQD Portal process

The main portal page is accessible via http://<portalserverFQDN>/CQD.
CQD-Portal-main.png

You probably will not see any data because “when the installer is done, most likely the SQL Server Agent job will be in progress, doing the initial load of the QoE data and the cube processing. Depending on the amount of data in QoE, the portal will not have data available for viewing yet.” To check on the status of the data load and cube processing, go to http://<portalserverFQDN>/CQD/#/Health.
CQD-Portal-health
Or (like my LAB) you don’t have any monitoring data to display :). After that you should see the last successful and failed update status:
CQD-Portal-health-ok

Other configurations that you can perform on the Portal are described on the Deploy CQD TechNet article:

  • Post-install tasks required to have reporting data regarding locations (buildings, networks name, subnets, BSSID)
  • By default, any authenticated user has access. This can be changed by using IIS Authorization rules to restrict to a specific.
  • Detailed log messages will be shown if debug mode is enabled. To enable debug mode, go to [CQD installed Dir]\QoEDataService\web.config, and update the following line so the value is set to True:
    <add key=”QoEDataLib.DebugMode” value=”True” />

And that’s it! you now have CQD fully deployed!
You can now see how the Lync/Skype4b is performing, and even build you own reports. Creating them is tricky, but you can learn some basics here.

<Am I missing something? maybe some more posts about it. provide me some feedback suggestions/requests 😉 >

Call Quality Dashboard – Part 2: The CUBE

After describing the Call Quality Dashboard (CQD) QoE Archiving Database on part 1, I will show now how to install the CUBE component and how it works on the solution.

The CUBE is “where data from QoE Archive database is aggregated for optimized and fast access” by the Portal component: this is the ‘data crusher’
ic841926

The CUBE is a SQL Server Analysis Service (SSAS) or generically known as an online analytical processing (OLAP).

Installing CQD – QoE CUBE

Before performing the installation the following pre-requisites need to be in place:

  • You need a server with SQL Server Analysis Services (SSAS) installed. The following picture  (all-in-one example) shows the required SQL components for CQD installationsic797717
  • It’s recommend to create a dedicated domain service account to grant the least required privilege to it. This account is used to trigger the cube processing.
  • The QoE Archiving Database needs to be already deployed.
  • You need to run the installation on the SQL server where the QoE Archive Database was installed. This is because some files will be installed and used by the SQL Agent.

The installation package is the same for all CQD components so, if: (a) you are installing all components you can go to step 2; (b) if you already installed the QoEArchiving on the same server, go to ‘programs and features’ and ‘change’ the package and proceed to step 2:

  1. Proceed throw the welcome screen, licence agreement, and choose the binaries install location

  2. For this part I will select the QoE CUBE and proceed to the configurations screen

    Configurations options:
    sqlname-vs-instance• QoE Archive SQL Server Instance: SQL Server instance name for where the QoE Archive DB is located. To specify a default SQL Server instance, leave this field blank. To specify a named SQL Server instance, enter the instance name
    • Cube Analysis Server: SSAS server and instance name for where the cube is to be created. This can be a different machine but the installing user has to be a member of Server administrators of the target SSAS instance.
    • Use Multiple Partitions: ‘Multiple partitions’ requires Business Intelligence edition or Enterprise edition of SQL Server. ‘Single Partition’ only requires for a Standard edition, but cube processing performance may be impacted.
    • Cube User – User Name & Password: Domain service account that will trigger the cube processing.

  3. After the validations the installation will ask to proceed until completion, hopefully without any error:)

    Behind the CQD QoE CUBE

    What happened and was configured after the previous installation steps?
    This component setup  installed some specific files, created a SSAS database and made some updates on the QoE Archiving Database:
    • QoECube database was created;
    • ‘Cube User’ login created and assigned db_datareader and db_datawriter on the QoEArchive
    • a credential created with the ‘Cube User’. This will be used to impersonate the connection to the QoECube to the source SSAS server.
    • A linked server source, mapping all the databases on the source SQL server
    • A 2nd step on the SQL Agent Job (created by the QoE Archive) and a proxy. This is the ‘brain’ that will trigger the cube.
    • The files used by the agent to trigger the cube

Known ‘caveats’ regarding the installation and architecture process:

  • The script command ‘process.bat’ to trigger the cube process overwrites the error log ‘process.log’ at every execution. Since the Agent execution is ran every 15 minutes you might not catch a cause/history of past errors.
    As quick workaround, you can change the script command to pipe and add (>>) the output to the existing log file:
    “%~1QoECubeService.exe” “%~1cubeModel.xml” >> “%~1process.log”
  • Don’t use a domain user account password starting with ‘+’. The setup SQL procedure will ignore it and then you will get the following on the SQL job and the cube trigger will not start:
    “Unable to start execution of step 1 (reason: Error authenticating proxy LAB\service.cube, system error: The user name or password is incorrect.).  The step failed.”

How to manage and monitor the CQD QoE CUBE process ?

The main CUBE processing is triggered using the same SQL Agent job created by the QoE Archiving. A second step is added to the job and whenever there is new data synchronized from the QoEMetrics to the QoeArchive, the job will launch a command script:
CQD-CUBE-SQLAgentExecution errors will be logged on the SQL agent log and details can be found on the file ‘process.log’ generated on the same folder as the command script.

Now you have a replica of your QoE data, a tool to process analyse it. You now need an interface to visualize and modulate described on part 3.

And finally…

There is a way to script the previous installation in one single command line (you just need to replace the orange text with your settings):

Msiexec /i “CallQualityDashboard.msi” ADDLOCAL=QoECube REBOOT=ReallySuppress CQD_INSTALLDIR=”D:\Skype4B\CQD” CUBE_ARCHIVE_SERVER=”LYNC-CQD.my.lab\CUBE” DISABLE_CUBE_MULTIPLE_PARTITION=”true” CUBE_ANALYSIS_SERVER=”LYNC-CQD.my.lab\CUBE” CUBE_USER=”LAB\service.cube” CUBE_PASSWORD=”WhoKnows?/qb!

  • You still need to run this it on the server holding the QoE Archiving database (it needs to install the agent script files)
  • Be sure to use lowercase ‘true’ or ‘false’ on the parameter.
    It will write ‘as is’ this value on the cubeModel.xml file, and the Agent job will fail and you will see an error on the ‘process.log’:
    Error while Processing: There was an error deserializing the object of type Microsoft.Rtc.Qoe.Cqd.QoECubeService.CubeProcessModel. The value ‘True’ cannot be parsed as the type ‘Boolean’.
    You can fix this by ‘lowercasing’ the value of the parameter <DisablePartitioning> on the cubeModel.xml