While I was preparing for a customer a PoC of a full redundant solution, I came again across a well-know limitation of attempting to consolidate multiple roles usingn the less amount of servers (less management,less licensing costs,…).
NOTE: the next text also applies to Skype for Business
So for this full redundant and automated solution, the plain idea would be:
- Pool1 with 3 front-end servers
- Pool2 with 3 front-end servers
- 2 Back-end servers, each with:
- 1 SQL instance for Pool1 (mirror with other server instance)
- 1 SQL instance for Pool2 (mirror with other server instance)
- 1 Lync file share storage (DFS between servers)
- 1 Office WebApp Server (Farm with other server instance)
- Use existing SQL express on a front-end server of each pool as witness for automatic mirror failover.
You can actually start creating SQL server instances
But when you try to assign one of the instances to a mirror, it will not appear:
The limitation is more visible if you try to create from the ‘New…’ button. Topology builder validates the existence of any existent server FQDN and also prevent the usage of instances named LYNCLOCAL or RTCLOCAL
You will also be prevented to add any other components (Office WebApp, gateways, trunks, edge servers,…)
With this limitations you might think of some Q&A:
- Question: There’s no technical limitation on the previous design. Is it possible to deploy this solution? Answer: Yes
- Question: Is it possible to use any existing SQL instance for mirror witness? Answer: Yes
- Question: Can I deploy Office WebApp components on an existing server in the topology? Answer: Yes
- Question: In a lab could be possible use only the Front-End servers for a Pool deployment? Answer: It would be radical !, but Y…
If you are not one of those guys that just follow the deployment wizards…
…. you just need to think ‘outside of the box/window’ (and more easy than installing Lync on a Domain controller) ;)
To be continued…
After applying the KB3101496 “security update for Lync 2013 (Skype for Business): November 10, 2015”, you might start noticing some back-rebranding of the “Skype for Business” skin :)
Here’s some samples:
And if you continue to use the ‘old-fashioned’ skin, they make noticed that is the Lync (Lync) one
An end user found this issue and I was able to replicate on other environments.
If you have Outlook (2013 version tested) installed you are able to search on Lync for Users or Distribution Lists (DL) and add them to your personal contacts address list.
But for DL, the Skype for Business (Lync) client will crash:
- Search for an DL and right-click on the ‘see contact card’
- Click on the ‘add’ option
- The Outlook contact card management window will open, but S4B/Lync client will crash and restart
- The Outlook window will still be open, so you can still add the DL to your personal contacts
– It will also happens by right clicking on any existing DL that you have on the Lync contacts;
– The issue occurs with the Lync or S4B skins. In either cases the error windows will display ‘Microsoft Lync has stopped working…’
– If you add the DL on Outlook, the contact card option changes to ‘Edit’ and you can successfully change the DL using the Lync interface without any issues;
– Adding an user doesn’t causing any problem (maybe because only the Lync client interface is used?).
(UPDATE 11.11.2015 1:05am) CAUSE
After dump analysis and a lot of KB add/remove sequence I found the reason(s).
Turns out that Outlook updates and Lync/SfB updates are ‘not tested together’:
- if you install KB3085495 (Outlook Set2015) with KB3085581 (Lync/SfB Oct2015) already installed you will get the issue;
- (Not fully tested) you will also have the issue if you install previous Outlook updates and/or with KB3085500 (Lync/SfB Set2015) installed
SOLUTION / WORKAROUND
I haven’t found yet a decent solution or workaround and several KB repairs/remove, really change the behaviour of the issue:
- sometime it doesn’t crash but the add button doesn’t work anymore.
- Warning: uninstalling the outlook update can make Outlook client unable to connect to Exchange :(
– A Lync enterprise deployment (with one or more servers), with persistent chat;
– Web Services behind a Hardware Load Balancer (HLB);
– the HLB is configured for cookie affinity (persistent);
– you try to upload a file greater than ~50kb, but it fails with an ‘Error in transfer’ message (as the picture on the right).
File upload to group chats was introduced February 2014 Update for the Lync Desktop Client (CU5/SP1). Mastering Lync has a great description on how the process works.
What I have found is that the Lync client (or just the group chat component) doesn’t honour/supports the set-cookie sent by the HLB. Since no cookie is sent the HTTP Post requests, the HLB might send some file chunk packets to the next available front-end server (depends on the balancer algorithm) and the server will drop with an 500 error.
Change the HLB internal Web Service VIP address to ‘IP source’ affinity. This way the HLB will send all client traffic to the same Front-end server.
Considerations and notes
There is some confusing documentation about Load Balancing internal services:
- The main MS reference identifies the Ports if Using Only Hardware Load Balancing, but not the affinity type;
- The Hardware load balancer requirements for Lync Server 2013 is not quite clear about External and Internal settings and how it supports cookie affinity:
> forcing the cookie to MS-WSMAN will not change the observed client behaviour;
> but then we find a line stating: ‘for internal Web Services VIPs, set Source_addr persistence (internal port 80, 443) on the hardware load balancer‘
> Barracuda ADC doesn’t specify this particular MS requirement. F5 overcomes HTTP persistence with a technique called OneConnect.
Since most of the companies provide private IP for each user client, there’s no issue on load distribution and this configuration is acceptable.
But if a lot of your user clients are behind NATed subnetworks, this configuration is not so much preferable (and that’s the reason to user cookie-based affinity).
As a final thought: Could the MS ‘requirement’ regarding HLB IPsrc persistence be a workaround to a client limitation? It would not be the first cookie handling issue found…
If you are thinking about updating your iOS to the newest Skype for Business client here a list of the features that are (still) not currently available from the previous Lync 2013 version.
If you have auto update enabled, then you might find out a little too late.
• Search your contacts from the Contacts screen.
• Search for Distribution Groups.
• Send an IM to all members of a Group.
• Use Group contact cards.
• Use Call Transfer.
• View Microsoft PowerPoint presentations.
• Set Call Forwarding and Simultaneous Ring options.
• Send the chat transcript as an email message.
• Access a participant’s Contact Card from the Participant List.
Meetings List screen
• Visually identify meetings that are currently in the Tentative acceptance state.
• Access the meeting organizer’s Contact Card from the Meeting detail view.
Voice Mail screen
• Delete a voice mail message.
• Use quick options to return a call and navigate to the Contact Card from a Voice Mail message.
• Use Accessibility features.
• Use Speed Controls for faster and slower Voice Mail playback.
According to Microsoft KB3102247: “We are working to address this difference in feature sets and to update the capabilities of Skype for Business for iOS as quickly as possible“.
If you don’t really have problems with this limitations, here’s a good look of the client.
Microsoft just added the following known issue regarding the latest September 2015 Security Update (SU):
- “You install security update 3085500 that is documented in security bulletin MS15-097 Security Update for Lync 2013 (Skype for Business) that was released on September 8, 2015.
- You join a meeting by using the Skype for Business client (version: 15.0.4753.1000).
- You click the More Options button, and then you click Start Recording.
In this scenario, nothing happens. You don’t see any notification or visual indication that recording has started. Additionally, when you open the recording manager, you don’t see any file or progress status.
This problem is scheduled to be fixed in the October 2015 update for Lync 2013 (Skype for Business).”
I try to check for this, but on the several clients, in different update levels (ex: update directly from April 2014 CU), I can still successfully record meetings.
There is no more explanation on the KB3099414, but there might be some pre-conditions to be meet before you to experience this issue. You might still want to apply the Set2015 SU (there are some of important fixes) and test if you’re enterprise users are affected.
The only reference so far is on a post in the Office365 community, where Microsoft Support confirms the issue.
These Microsoft KB documents where updated on the last 24 hours, got me puzzled:
KB3080353 (25/09/2015): Description of the security update for Microsoft Lync Server 2013 (Web Components Server): September 8, 2015
“To install this security update, you have to install the latest July 2015 update Lync Server 2013.”
KB3098577 (24/09/2015): Sign-in and query errors after you install MS15-104 Security update for Microsoft Lync Server 2013 (Web Components Server)
“After you install security update 3080353 …, you experience one or more of the following symptoms in Microsoft Lync Server 2013:
- Users can’t sign in to your dial-in page.
- Lync Mobile clients can’t sign in.
- External clients can’t sign in.
- Address book web queries fail.
- Users are prompted for credentials for some web services after they sign in internally to Lync desktop clients.”
But below on the KB text:
“If you install Lync security updates by using the Cumulative Server Update Installer (LyncServerUpdateInstaller.exe), the installer should prompt you to install the correct prerequisites
Security update 3080503 (MS15-104: Security update for Microsoft Lync Server 2013 (Web Components Server): September 8) can be installed without previously installing the required July 2015 cumulative update 5.0.8308.920 for Lync Server 2013.”. A new version of LyncServerUpdateInstaller.exe is shipped together with the security update.
My concerns are:
- The issue is applying the September “WebComponents.msp” (you can find it separately on the download details) on the download patch without updating the one from July (ex: using windows update)
- The “LyncServerUpdateInstaller.exe” is basically a package of the most recent released .msp files for each components
- So, what WebComponents.msp version is inside the September LyncServerUpdateInstaller ?
Just for the safe side (since I have a large deployment with 4 pools and 28 Lync Servers), I will apply the July 2013 LyncServerUpdateInstaller and after that the September one.
Time to compare to the July and September 2015 updates. It all comes to the WebComponents.msp content. Beyond the ~1500 files with some date differences, we have this:
||July 2015 CU
||September 2015 SU
||1511 Files total
||1507 Files total
||included in .msp
||Not included in .msp
||included in .msp
||Not included in .msp
||included in .msp
||Not included in .msp
|[Lync_Install_Dir]\Web Components\Web Scheduler\Ext\Scripts
||Not included in .msp
||included in .msp
These file are part of the puzzle, especially if they are not part of the first release of Lync. They ‘swf’ were introduced on the December 2014 CU, and update on May 2015 CU and July 2015 CU (check the file information section on each KB). This means that applying the Lync servers with September 2015 SU without the May 2015 CU, you will have any old version or even none (if you didn’t applied nothing before December 2014)
I noticed recently on several client troubleshooting (using the Snooper tool) on a particular “400 Bad Request” message that appears on the end calls.
After several cumulative updates (CU) history regression testing, I found that this behaviour started appearing with the latest two CU (or correctly, the security updates):
- KB3055014 – security update for Microsoft Lync 2013 (Skype for Business): August 11, 2015
- KB3085500 – security update for Microsoft Lync 2013 (Skype for Business): September 8, 2015
If you have any of those updates, you will find the 400 error message on every call (PSTN or Skype) when the called party disconnects. Here’s an example of the same call I made to a PSTN number:
- Until the July 2015 update, the client will send an “error” report in xml format for the front-end server
- After the August/September 2015 updates the same report is now different (similar response as a Skype-to-Skype)
There’s a <progressReport/> ending without any initial <progressReport>
- The server detects this incorrect xml format and that’s why you see the “400 Bad request” and a “Client.BadCall” fault
For the end-user there is no visible or functional impact.
I still don’t know if there are any impacts for the backend services. The call diagnostics for CDR and QoE diagnostics are correctly sent on another SIP message.
There is one more issue when you upgrade your iPhone or iPad to iOS9: Looks like when try to open a meeting url on Safari (ex: received through a webmail server), the Lync client is not launched and you stay on the meeting webpage (like the picture on the right).
- workaround: install/use the Opera mini web browser on the iPhone/iPad
- Microsoft confirms the issue on KB3097592 and has another workaround.
- 25.09.2015 – iOS 9.0.1 update doesn’t resolve the issue
- 01.10.2015 – Microsoft just released Set2015 CU13 (don’t get confused with Set2015 SU) that contains a web component update to address the issue (web pages and scripts) were updated and safari can now launch the Lync client.
- 29.10.2015 – After some testings, I noticed that it’s not working for iPad devices with iOS 9.1, released 3 days before which includes 24 security updates (details here)
- 13.11.2015 – Microsoft confirmed that the iPad 9.x is still not working. Those users still need to use the workaround described on KB3097592 to join meetings urls or my workaround :).
You might not face this particular issue, but it could be helpful to troubleshoot similar on how the mobile clients communicate with the external web services.
Everything was running normal on this customer, until the day we plan to implement a two factor authentication (TFA) engine. To understand the nature of the issue here’s the background design:
- In a standard deployment the ‘reverse proxy role’ will forward the external web services [public IP]:443 to the [front end pool]:4443
- The TFA solution replaces this roles and ‘intercepts’ the web services data (the components are installed on the Edge servers, work separately for mobile client devices). Besides DoS protection it validates user/device pairing
- For a high availability solution, hardware load balancers are used. Since these are web services, cookie passive affinity is used (a HLB cookie is inserted if the server response adds is own cookie). We could use MS recommendation of source IP affinity, but since there’s a lot of locations (with internet access through NAT) with thousands of users we want to ensure an equal load distribution for the TFA services.
Everything looks good and ready to go, but then this interesting behaviour start occurring:
- Windows and Android clients worked fine;
- but iOS (iPhone and iPad) start fail to sign-in;
- bypassing the ‘TFA filter’ will made all devices connect okM
- configuring the HLB to send all traffic to one node only, will also made iOS clients sign-inM
- With HLB Source IP affinity, all worked fine;
- Cookie insert affinity will still fails for iPhones and iPads;
- The latest Lync server and HLB firmware updates would not solve the issue.
You (and at the beginning I) would assume that the problem is somewhere related to the HLB on the TFA, right?
Well… troubleshooting the iPhone client give an additional surprise:
- When the client authentication process starts, the TFA component returns the server response to the client adding his cookie. The HLB will then also includes is cookie so the remain authentication process and traffic will be sent to the same server.
- but then I just notice the next client POST request is ignoring the HLB cookie and is only sending the TFA cookie !
The reason of the failure is that, without the HLB cookie, all client traffic requests are distributed across the two TFA and the other Front-end servers that didn’t participate on the initial negotiation and… the sign-in process fails.Looking at the same process for the Windows (Skype for Business) and Android (Lync 2013), all the cookies are sent.
After several reconfigurations attempts and troubleshooting, the issue was solved by defining on the HLB the cookie path (‘/’ instead of empty) , and the iPhone client start acting as it should:
- This time the HLB cookie is appending the path
- And the iPhone client send all the cookies on the next POST request
So at the end, it looks like that there is some sort of undocumented behaviour on Lync 2013 client for iOS.
Update: the issue is the same on the new Skype for Business version
If you just upgrade your iPhone/iPad to the iOS 9 released on the 16.09.2015, and have a language different from a region where the language is not native, like me (left picture):
You might be unable now to login with your Lync 2013 client with a strange error message (right picture):
Doing some “sniffing”, looks like Lync is sending a Culture ID that the server doesn’t recognize and replies with a “401 Bad request” in my case en-CH doesn’t exist:One workaround do far seems to be: define a language that exists on the region with so Lync can send a valid culture ID.
Tkmikal found this Microsoft KB3096704 recognizing the issue:
“This problem is fixed in the Microsoft Skype for Business for iOS app that will replace Lync for iPhone and Lync for iPad when it’s released. No fix for this issue is scheduled for the current releases of Lync for iPhone and Lync for iPad”.
This should stay by here, but then I went “under” the iOS and found an additional cause. If you have the location services disabled during the iOS process update, it will not be able to set all the new information parameters about language and region for version 9.
So, here’s a stable solution:
- Turn on Location Services on the iOS device, and be sure to have an internet connection active;
- Switch the language to another one and back to the one you want – this is a fast way to make the iOS download all the iOS9 configuration format;
- Start Lync and you are now able to sign-in;
- You can now disable back the location services. Lync will still sign-in correctly because iOS9 language settings are now installed.
When you have a 12’000 users infrastructure, you are sure that the any Lync client issue will be detected.
On this case someone notice a glitch with the “Skype for Business skin” only. After some testing we pinpoint the issue:
- You receive an PSTN call;
- When you start using another desktop application the call monitor will switch to the miniature version and…it flashes from time to time!
The issue was reported to Microsoft 2 weeks ago.
Although is not documented or confirmed by technical support, it seems that the security update for Microsoft Lync 2013 (Skype for Business): September 8, 2015 solved the issue.
The recent MS15-034 security update addresses a vulnerability on how the Windows HTTP stack (http.sys) handles requests.
Although you affected Operating System component is related to IIS, there are many applications that can rely on Windows HTTP stack. And Lync is one of them !
How serious is this vulnerability and why you should patch immediately?
- It can be used for DoS attacks, but there’s a chance to be used to run code remotly
- Any user can run an exploit of some type without any special permissions and good knowledge;
Can be just a simple copy > paste code (see PoC)
- Your Lync front-end servers can be exploited by an internet attacker, if the reverse proxy role (and/or the firewall) cannot detect and intercept the exploit attempts.
What Lync ‘roles’ are affected?:
- Front-End – There’s a lot of applications/pools that can be exploited
- Edge server – not affected from the outside. But the internal DMZ replica service (typically 4443) can be exploited
- Persistent chat – not affected
- (SQL) Monitoring reports – affected
- Office Web Apps – affected
To show you how easy the exploit can be built and run, here’s a simple Proof of concept. I just needed 10 minutes to find a possible http request and run cURL on an internal PC without any admin rights:
The server running Lync will stop responding and (if you are fast enough) you will see the operation system generating a dump report, before restarting.
An exploited server will also display a MER message when you logon to it:
You might want to look carefully for Lync and other collocated applications that can also allow an exploit. This command can be used to determine what is relying on http:
netsh http show servicestate | find “://”
So it’s better to start patching all windows operating system on you network… fast
I don’t know about you, but I have some difficult to memorize powershell commands.
In Lync 2013 there are 754 cmdlets available: some that I know and use on a daily basis, a few are specific for Lync Online, several that I never used or need at all but from time to time I find out some that are useful for specific tasks and that… I forget them later.
Google is still my first tool to find out how to get some particular job done in Lync, but I decided to create a summary list Microsoft TechNet Lync 2013 cmdlets (see picture sample on the right) with:
- The cmdlet that includes the url to the specific TechNet documentation
- The short description of the command as is on TechNet
(took me 2 days to build a function to extract this part of text using the page url of the first column.)
- Add-on: default Lync RBAC groups and command assignment
It was also another need for me to allow me to plan RBAC permissions grant and create specific roles
If this could be also useful to you, the file can be downloaded here: LyncCmdlets or at Technet
On my next available time, I plan to add more features (like categories)
I just discovered an issue when installing the December 9, 2014 cumulative update (CU) for Lync client 2013 (KB2910927)
When you try send an url with the prefix www you get the error message “the web page … was not loaded in response to you clicking the link because it is either invalid or restricted for security reasons” (picture on the right). This will appear even if have no URL policies restrictions.
– prepend http:// to the address you want to send, or
– uninstall KB2910927. The previous CU doesn’t have the issue.
Googling on the Internet I also found that a similar behaviour has already been reported and confirmed to Microsoft, and published as KB3053114 issue.
UPDATE: The latest Lync update (KB3039779) fixes the issue.