How to setup load balanced SSRS Servers for CRM 2016

While not very common, there are times when a customer or client requests two servers for the CRM SRS data connector/SSRS to be installed and have load balancing configured for the servers. The idea here is failover; when one server hosting SSRS crashes, all incoming requests are routed to the secondary server. There are a couple of ways to do this, but the following method is the most straightforward and simplistic way.

1.    Create a Virtual IP(VIP) to route to each SSRS server.

In order for this method to work, a virtual IP will need to be created. The virtual IP will accept incoming data packets, then route the requests to each of the IP addresses of the physical servers that will have SSRS installed.

2.      Install and configure SSRS on each server.

Install and configure SSRS on each server as you typically would by pointing to the SQL server and/or instance where you would like the report server databases to be located.

3.    Install the CRM SRS data connector on each server.

Grab the installation files for your respective version of CRM and install the data connector on both servers. Install the data connector as you would for a typical CRM deployment.

4.      Configure the host file on each SSRS server.

On both SSRS servers, open the host file: C:\Windows\System32\Drivers\etc\hosts. Edit the host file by adding the IP and DNS name of the virtual IP created in step 1:










5.      Add the BackConnectionHostNames registry key with the server name and FQDN.

Open Registry Editor on one of the SSRS servers, and locate the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0.

Right click MSV1_0, point to New and then click Multi-String Value. Type BackConnectionHostNames, then press ENTER.

Right click BackConnectionHostNames, then click Modify.

In the Value data box, type the host name of the VIP and FQDN of the VIP, and click OK.








Repeat these steps for the other SSRS server.

6.      Add hostname and URL root values.

On one of the SSRS servers, make a backup of the reportserver.config file located here: C:\Program Files\Microsoft SQL Server\MSRS11.MSSQLSERVER\Reporting Services\ReportServer.

Right click the original reportserver.config file and choose edit.

Modify the hostname and URL root tags of the .xml file located under the service tag by adding the FQDN of the VIP as shown below:



Save the file after making the changes.

Repeat these steps for the other SSRS server.

7.   Restart both SSRS servers.

8.   During the import or new creation of an organization in CRM, specify the name of the VIP.

Use the VIP created in step 1 when prompted for the reporting services URL during a new org creation or an org import:

ADFS setup error when using CRM 2016 and ADFS and Windows 2012 R2

An error occurred. Contact your administrator for more information” error when accessing CRM with ADFS/IFD set up.
After completing ADFS/IFD setup where ADFS is installed on a Windows Server 2012 R2 machine, you receive the below error:

 To resolve this issue you must enable Forms Authentication:

1.       Connect to the ADFS server

2.       Open the ADFS management console and click Authentication Policies







3.       Under Primary Authentication, click Edit next to Global Settings






4.    Put a check mark in the Forms Authentication option on the Extranet and Intranet sections

















5.    Click OK

6.       You should now be able to log in to CRM successfully

Thanks to Ian Holton, Client Field Engineer at Tribridge for putting this together!

Special Tribridge Customer Invite and Offer to CRMUG Summit 2016!

Special Invite to CRMUG Summit 2016

Are you searching for best practices in your role? Do you need to train new team members on Microsoft Dynamics® CRM, but don’t have the time? We invite you to join us and your Dynamics CRM peers October 11-14 at CRMUG Summit 2016 in Tampa, Florida. At CRMUG Summit, you will discover the premier opportunity to affect positive change in your career, and Tribridge is excited to share in this journey with you.

As a valued customer of Tribridge, you can save 10% when you register with our special coupon code at the time of check out. Tribridge Discount Code: Summit16TRI. (Ends September 1st. Registrations completed without this code are not eligible for the discount. No post-registration adjustments will be given.) We encourage you to register for CRMUG Summit today and save on your registration.

Here’s what you need to know about CRMUG Summit 2016:
• Customize your experience by choosing which of the 150+ sessions to attend including role-based sessions for System Admin, End Users and more
• Arrive early and participate in two full-days of instructor-led Pre-Conference CRMUG Academy Training
• Exclusive access to the Summit 2016 Online Community
• Network with thousands of your Dynamics Peers, including dedicated networking receptions
• Collaborate with Microsoft Dynamics® CRM Staff, Microsoft  & Tribridge MVPs (@ccognetta and @edwardsdna), and CRMUG All-Stars
• And of course, a beautiful destination – Tampa!
• CRMUG Infographic HERE
Gather your team and join Tribridge in Tampa, Florida this October – REGISTER TODAY! I look forward to seeing you in Tampa. Please make sure to stop by and see us – we will be in booth 731.

© 2016 Tribridge. All rights reserved.
4830 West Kennedy Blvd., Suite 890, Tampa, FL 33609 | P (877) 744-1360 | F (813) 287-8688

Why You Should Have TCP Port 80 Open Outbound On Your ADFS Server?

Active Directory Federation Services (ADFS) performs a lot of tasks when it comes to authenticating users into CRM securely. One of those tasks in particular is a certification revocation check to validate that the certificates being used are still valid. ADFS completes this process by reaching out to certification revocation lists (CRLs) over TCP port 80 – basic HTTP communication.

What we’ve seen is that businesses will want to lock down their ADFS servers just to be on the “safe side” and that includes closing TCP Port 80 outbound (e.g. no internet access). If left in its default state, ADFS will break and cause authentication to fail as it knows that it is supposed to check the CLRs to validate the certificate before issuing a token to allow a user into CRM. If it cannot do this, it will not issue a token. You may see an error similar to the following in the ADFS event viewer logs after a failed authentication attempt:










Event ID: 364

Microsoft.IdentityServer.AuthenticationFailedException: MSIS3014: The encryption certificate of the relying party trust ‘’ identified by thumbprint ’01DEDF6E6F532BF7357457EBEC31DA82SFDA1234′ is not valid. It might indicate that the certificate has been revoked, has expired, or that the certificate chain is not trusted. —>

Microsoft.IdentityServer.Service.SecurityTokenService.RevocationValidationException: MSIS3014: The encryption certificate of the relying party trust ‘’ identified by thumbprint ’01DEDF6E6F532BF7357457EBEC31DA82SFDA1234′ is not valid. It might indicate that the certificate has been revoked, has expired, or that the certificate chain is not trusted.

So what are your options?

  1. Have your networking team open TCP 80 outbound on the ADFS server(s). This would also apply to all ADFS Proxies or WAP servers. While opening a port might seem less secure at face value it would actually be the opposite as ADFS is able to validate the certificates being used.


  1. The less preferred, but still acceptable, method would be turning off the Certificate Revocation Check of ADFS. The check is controlled individually for each relying party in ADFS so it would need to be turned off for all one by one. To do this open an admin PowerShell prompt and issue the following command:

                    Set-ADFSRelyingPartyTrust  -TargetName <relyingpartytrustName> -EncryptionCertificateRevocationCheck None


CRM Database Log Growth Issue

A customer of ours had come to us facing a rather interesting issue. Every night around 1am their CRM database log file would grow to 31GB and cause the log drive to fill up. When users would log into CRM in the morning, they would receive SQL errors stating that their transactions could not be completed.   Given that this issue occurred on a regular schedule, we determined that the issue had to be attributed to some sort of automated job.

Figuring the first place to check would be the out-of-the-box CRM Maintenance Jobs, we downloaded the Job Editor from Codeplex

( – Very useful tool that everyone should be using) and got to investigating.

Right off the bat, we saw our error on the Deletion Service:
Deletion Service encountered an internal error: System.Data.SqlClient.SqlException (0x80131904): Invalid object name ‘SubscriptionTrackingDeletedObject’

From here we moved to SQL and queried the SubscriptionTrackingDeletedObject table of our CRM database.

What we found was astounding – the table contained 137 MILLION records. Basically after seeing this we knew that the job had to just be timing out – we tested our theory by running the job manually and immediately saw the log file grow to the expected 31GB.

It was decided to clean this table up manually via truncating it. Before you say “Oh no! Don’t delete records via SQL directly!” let’s explain what this table actually is.

When records are deleted from CRM, there are also records that get inserted into this SubscriptionTrackingDeletedObject table. This table gives the Deletion Service Job ObjectIDs that have been removed so that further cleanup can be performed asynchronously. So essentially, it is just a table of deleted records which gives the Deletion Service knowledge to clean up some other areas of CRM (e.g. POA records, duplicate detection records, etc…) if necessary. Once cleaned up, the records from the table are removed. We understood this and decided the need to clean this table outweighed having the other areas of CRM cleaned up (as you will learn later, this wasn’t a concern for us because of how the records got in there).

Please note we cannot condone the practice of editing SQL manually without full knowledge of the possible repercussions. Always consult with Microsoft support if in doubt and remember what works in one scenario, may not work in all.

After the table was truncated, CRM was tested and the deletion service job was run manually – this time not failing with the error above and the log did not grow to 31GB. Before calling this case closed, we still needed to understand what caused this problem in the first place. What could have possibly created so many records in such a short period of time? Luckily, we had the right people involved on the customer’s end and were able to determine that there had been a malfunctioning Scribe job that was running for a while unnoticed. It was a job that was bulk creating and deleting 50,000 records at a time within CRM but the job had since been fixed. Case closed.

CRM Organization Import Issue and SSRS MaxRequestLength

When importing an organization to CRM 2011 we came across an error during the import wizard process which was causing the import to fail:

14:54:33| Info| PublishReportsFromDatabase: Creating report in Reporting Services. ReportId: 9f973403-bc84-e111-88bc-0050569e0001, Name: INVOICE PAYMENTS
14:54:34| Error| Error while updating organization information: System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.Crm.Reporting.RuntimeReportServer.UploadReport(String path, Byte[] reportDefinition, String name, String description, Boolean overwriteExistingReport)
at Microsoft.Crm.Reporting.RuntimeReportServer.UploadReport(SRSReport report, String reportNameOnSrs, String name, String description, Boolean isSharedReport, Boolean overwriteExistingReport)
at Microsoft.Crm.Setup.Server.Utility.ReportsUtility.OrganizationPublishReportsScaleGroup(IDbCommand command, Uri reportingUrl, String orgUniqueName, Boolean ignoreCustomReportsFailure, Boolean publishOnlyCustomReports)
at Microsoft.Crm.Tools.Admin.DBImportHelper.RePublishReports(IDbCommand command, Guid organizationId, String organizationUniqueName, Uri reportUrl)
at Microsoft.Crm.Tools.Admin.ImportOrganizationInstaller.UpdateOrganizationInfo(Guid organizationId, OrganizationGroupsInfo organizationInfo, String organizationFriendlyName, String organizationUniqueName, Uri reportServerUrl, Int32 PercentUpdateOrganization, ICollection`1 users)

This was not an upgrade of any sort – just simply a CRM migration to a new environment so there were no versioning differences from a CRM perspective. While the error above is lacking much detail, it did give us enough to begin troubleshooting. We could clearly see that this issue was occurring with a report titled “Invoice Payments” but knew nothing else. After obtaining a copy of the report’s RDL file, we didn’t notice anything wrong in particular with the way the report was written but did think it was rather large for an RDL file – nearly 5MB – there were quite a few embedded images.

We decided to attempt uploading the report directly into SSRS and were met with a much more helpful error message – “Maximum request length exceeded”. What this was telling us rang a bell with what we noticed earlier regarding the RDL file size. By default, SSRS has a limit on the report file size that it will allow to be imported. This limit is 4MB but can be increased by doing the following:

Open the web.config of the Report Manager (%\Program Files\Microsoft SQL Server\MSSQL.X\Reporting Services\ReportManager) and find the “executionTimeout” setting. It should look something like this:

On this line, add the maxRequestLength attribute with the value (in KB) needed to upload the report. This value is not in here by default. It should now look like this (this shows a 10MB limit):

Save the file and then repeat these steps in the web.config of the Report Server (%\Program Files\Microsoft SQL Server\MSSQL.X\Reporting Services\ReportServer). Once both files have been modified and saved, restart the SSRS service.

Following this change, we were able to upload the RDL to SSRS directly to verify that the change worked and then subsequently were able to complete the organization import for CRM without issue.

CRM 2016 Spring Release Best Feature : A little bit of Polymorphism

Several-out-of-the-box entities in Dynamics CRM (including Case, Lead, and Opportunity) contain a field that represents a customer, which can either be an Account or a Contact. In CRM, these fields have the Customer data type, which is a special kind of lookup field for these two specific entities. Previously, system customizers couldn’t add a Customer field to their own entities. As part of this feature, Customer is available as a custom field data type that can be added to any entity, system or custom.

One of our MVP’s Shan McArthur, now with Microsoft leading the the Microsoft ADX Portal development as been fighting for this feature for a long time. In memory of that fight, we will refer to these field type as a “ShanMC Field.”. Thanks for your support throughout the years Shan!

CRM 2016 available, but not ready for your Production Server just yet!

Over the last few weeks my colleagues and fellow MVP’s have been working on a few CRM 2016 systems. Our general thought is that a lot of this code is from the CRM Online 2015 code base, and should really be production ready. However, the Microsoft CRM team has been hard at work delivering new features, so looks like we introduced a few bugs.

I’m going to recommend that you wait for update rollup 1 before deploying to your production system. There’s still plenty of time to test and install in your dev and test systems as you wait for these fixes. Here’s the main areas:

  1. Overall Outlook 2016 Client Configuration and Usage Issues – Support Recommended to stay with 2015 client
  2. New Form Rending Engine slowing overall form performance – fix in the works for update roll up 1
  3. Tweaks to the 2016 Solution Engine, breaking some backwards capability while importing solutions – fix in the works for update roll up 1
  4. Mobile Client performance issues

So we will re-test about the update roll-up 1 and provide future recommendations when we feel this is ready for production. Remember that there are some best practices and one is two wait until the first update roll-up is available. Stay tuned.

Understanding the ADFS Token Signing and Decrypting Certificates Rollover Process

Active Directory Federation Services (ADFS) creates and manages the two certificates used for the tokens issued. These are the Token-signing and Token-decrypting certificates. By default, these certificates are valid for one year from their creation and around the one-year mark, they will renew themselves automatically via the Auto Certificate Rollover feature in ADFS. Once this happens, CRM can no longer properly authenticate users as it still holds the old certificates’ metadata in the database. This is easily resolved by rerunning the “Configure Claims Based Authentication” and “Configure Internet Facing Deployment” wizards from the CRM Deployment Manager and then issuing an IISRESET on the CRM server(s).

More details and resolution can be found in this KB:

While most CRM administrators understand that aspect, there are a number of settings and configurations that lead up to this issue that are less well-known to most. One of the biggest complexities is understanding EXACTLY when CRM will be going down because of the Auto Certificate Rollover and how to avoid it. We will be going through that today.

We will start with the process that ADFS goes through for certificate renewal:

  1. ADFS determines that its certificates will be expiring soon.
  2. ADFS creates new certificates and sets them as secondary certificates.
  3. ADFS updates the new certificates to primary certificates.

There are a number of settings for ADFS only accessible via PowerShell that control the Auto Certificate Rollover options and properties for the process above. To access these, open an administrative PowerShell prompt and execute the following (Note that if you are using ADFS 2.0, you will need to add the ADFS PowerShell Snap-In first by executing Add-PSSnapin Microsoft.ADFS.PowerShell):



This will display a listing of the deployment properties for ADFS, including the properties around the certificates and rollover. For our purposes, we will keep our focus on just a handful of these properties:

  • AutoCertificateRolloveradfs2
  • CertificateDuration
  • CertificateGenerationThreshold
  • CertificatePromotionThreshold
  • CertificateRolloverInterval



So what do all of these values mean? Below are the same steps provided earlier but now account for the values in the table above.

  1. ADFS determines that its certificates will be expiring within 20 days.
  2. ADFS creates new certificates valid for 365 days and sets them as secondary certificates.
  3. After 5 days’ time the Certificate Management Cycle kicks off and ADFS updates the new certificates to primary certificates.

As you can see, knowing these values can greatly help in planning for certificate rollover. Here is an example:

In the screenshot below, we can see our primary certificates expire on 2/12/2015 and ADFS has already created new certificates to rollover. The new (secondary) certificates were created 20 days prior to the expiration of the primary certificates (1/23/2015). On 1/28/2015, 5 days after the creation of the new certificates, ADFS will change them to primaries.



In the above example, you know your deadline is 1/28/2015. Rather than sitting back and waiting until CRM goes down, plan a short outage afterhours and roll the certificates over manually! You can force ADFS to generate new certificates and promote them to primaries immediately using the following command in PowerShell:



Once the new certificates are in place in ADFS, re-run the Claims and IFD Wizards in the CRM Deployment Manager to update the metadata and issue an IISReset on the CRM server(s). Voila! Happy CRM users!

Of course, given the properties we have at our disposal to modify there is much more you can do to create a better life for yourself. For example, set the CertificateDuration to 1095 days (three years) rather than just 365 (one year) so this is not as frequent of an issue. Another idea would be to set the CertificateGenerationThreshold lower so the actual rollover date is closer to the true expiration of the certificate. Or just turn off AutoCertificateRollover altogether, set a reminder, and take care of it all manually before expiration!

Another great post from my team at