After several installations and Skype for Business 2015 (S4B) Server upgrades, a colleague of mine pushed my attention to a large ammount of event id 4097 warnings on the Administrative Events view related to Windows Fabric. The rate if this flood could be 3-5 events every 5-15 minutes:
– “cert chain trust status is in error: 0x1000040”
– “ignore error 0x80092013:certificate revocation list offline”
You can check all those events on the specific Windows Fabric log shown on the below picture.
Question: Why and where do this event come from?
S4B installs Windows Fabric 3.0 and added, between others, additonal securitys setting in the fabric intracluster communications. You can find this settings on several configuration files:
The relevant settings related to the events can be found on the ‘Security’ section:
<Parameter Name=”SessionExpiration” Value=”28800″ />
<Parameter Name=”IgnoreCrlOfflineError” Value=”true” />
<Parameter Name=”CrlCheckingFlag” Value=”3221225476″ />
The IgnoreCrlOfflineError is self-explanatory: If Windows Fabric encounters this error, ignore it and continue operations. For the CrlCheckingFlag values we need to look a little ‘deeper’. The Windows Fabric configuration files are generated by the Front-End server (RTCSRV / RtcHost module) at service startup based on this template file:
<S4B installation folder>\Server\Core\ClusterManifest.Xml.Template
On this file you will find a little more about the flags value:
CrlCheckingFlag setting follows the rest of the Lync Server components (sipstack, web) which set the following flags:
CERT_CHAIN_CACHE_ONLY_URL_RETRIEVAL=0x00000004 | // do not go on the wire for cert retrieval
CERT_CHAIN_REVOCATION_CHECK_CACHE_ONLY=0x80000000 | // do not go on the wire for cert revocation check
0xC0000004=3221225476 (unsigned int)
<Parameter Name=”CrlCheckingFlag” Value=”%CRLCHECKINGFLAG%” />
More information regardind the flags can be found on MSDN CertGetCertificateChain function.
Basically, S4B services (including the Fabric) are configured to check certificates revocation (CRL) using local cache only, as the comments on the template says: do not go on the network to retrieve and check the CRL.
So… how does it check for a certificate revocation?
Running a trace on the Fabric process, we can see that it never tries to connect to a CRL distribution server but, a little before the events are logged, it reads the registry keys related to certificates.
Since there are no cached or local CRL available, it will report an CRL offline error, ignore it, but still write it on the event log.
Question: What can we do about this ?
I can see at least 3 options:
Option A (The ‘no worries’ one) – Ignore it!
Difficult level: noob
Advantages: Nothing to be done. It’s a normal S4B operation and nothing is affected
Disadvantages: Your administrative events view will be full of these events which makes difficult to find important warning
Option B (The ‘clean’ one) – provide the CRL to the local server
Difficult level: professional services
Advantages: You will not just stop seeing the Fabric events, but it will also optimize all the other S4B services
Disadvantages: Requires knowledge and script programming, since you want to schedule a task to get the CRL and put it on the machine certificate local store.
This option also proves how the CRL caching / ignoreofflineCRL works.
1. Find the certificate name used by Windows Fabric. You can get this from the Fabric configuration files (spoiler alert: it’s the same used by Lync internal services)
2. Find on the certificate the Distribution paths where to get the CRL
3. Download the CRL and install it on the Trust Root CA or Intermediate CA store depending if the certificate was issue by a Root or a Subordinate CA
… and no more 4097 events !
As you noticed, the CRL are updated regularly (daily or weekly) so they have an expiration and need to be retrieved. This means that doing this manually is a error-prune, time-comsuming task, which you can solve with a scheduled script (not included on this blog… yet!)
Option C (The ‘tweak’ one) – manipulate the CrlCheckingFlag
Difficult level: moderate: just need to know what you are doing
Advantages: It’s easiest way to stop seeing the Fabric events
Disadvantages: It’s a S4B undocumented parameter customization (altough I don’t believe MS will not refuse support if they found out). It can also be overwritten by an update.
This ‘quick fix’ option is about manipulating the file
‘<S4B installation folder>\Server\Core\ClusterManifest.Xml.Template’
1. Change the parameter of the ClusterManifests.Xml.Template
<Parameter Name=”CrlCheckingFlag” Value=”0″ />
2. Stop the RTCSRV and FabricHostSvc services
3. Start the RTCSRV service – it will create new Fabric configuration files based on the template file and also start the FabricHostSvc
Actually, according to this blog, this parameter value is required if you issue certificates withouth any CRL Distribution Point information…. or your S4B pool will not start at all (!)
– Newly installed Skype for Business Front-End Pool refuses to start
- If you want to know more about Windows Fabric and S4B we have a great and detailed explanation on this blog