Moving From AD FS to Pass-Through Authentication – Some Field Notes

As you may know I’m managing a pretty large “Office 365” tenant. “Large” as in 20.000 licensed user all over the world. And these users all originate from 3 different on-prem Active Directory forests.

Since we got started with our O365 journey all those years ago the only way to get good authentication user experience was using Active Directory Federation Service (AD FS) which, in a highly available configuration for this workload, had to be made up of 6 servers + 2 load balancers and alot of network configuration and different certificates.
This AD FS solution was later upgraded to Windows Server 2016 mode and one of the main drivers for that was to enable features like “extranet lockout“. However, because of some weird communication issues with one of the domains we could never really activate that without locking those users out. So when looking at alternatives, with the condition that authentication had to be in “real time” (i.e if a user was locked out in the AD it would instantly be locked out of Azure AD as well) and with SSO, then there was just one alternative for us and that was Pass-through Authentication (PTA). It was now in a state where we could get SSO and a good enough user experience to make the switch.

Even though Microsoft have done a good job of documenting the switch, it leaves a bit to be desired (for example they mix up AD Forest and AD Domains all the time) and there are also some twists along my specific scenario so I decided to write some field notes on this.

 

The Azure AD Connect conundrum

Now if you go through the Microsoft documentation and guidelines for this you’ll see that there is a difference in how to go about moving from AD FS to PTA depending on how AD FS was configured. If it was configured manually, then you’re in luck and you get to manage the entire switch yourself and testing will be really easy. If it was configured using Azure AD Connect you may end up having some issues. This is because when changing “how users authentication” in Azure AD Connect, it will kick off a job to automatically switch all those domains listed as “Federated” and make them “Managed” giving you very little control of how the switch occurs in your production environment. Luckily for us, our AD FS was set up manually so that gave me the option of switching every domain individually.

And this is incredibly important to keep in mind – this switch is entirely based on the “Authentication” setting for each domain in Azure AD!

 

Preparation

You should ofcourse read through Microsofts documentaion first to make sure you’ve ticked every box. One of the most important steps is for the SSO to work is to add the URL to the Intranet zones in MSIE, just like you would the AD FS URL.

You should also try to find a UPN suffix domain in your on-prem AD that isn’t used by any user in production and add that to your Azure AD and set it up for Federation in AD FS and verify that works fine. If you can’t find one, either create a new one or use one with as few users as possible.

Another important step is to determine where the PTA agents are going to reside. One will be on the Azure AD Connect server but it will want 2 more agents somewhere.

 

Changing Azure AD Connect

The first step on your journey is to enable this feature in Azure AD. This won’t change the user experience so this is safe to do whenever. It’s pretty straightforward, you just start the Azure AD Connect wizard and go the task “change how users authenticate“. In this step you simply chose “Pass-through authentication“. If you only have one AD forest you can go ahead and activate SSO for that forest, but I had to do it afterwards in Powershell instead since I had 3 Forests.

This enabled the feature in Azure AD and if you go to your Azure portal -> Azure Active Directory -> Azure AD Connect you’ll see “pass-through authentication” = “Enabled”. But you’ll probably have a warning symbol next to it because you only have 1 agent right now. This would be a good time to download the agent and save it somewhere. If you activated SSO for your forest you’ll also see SSO = Enabled for 1 domain (which is actually a forest!).

That was easy enough and didn’t change much. Because at this stage every domain (except the “onmicrosoft” one) that is set to “Managed” authentication will go through PTA but everyone set to “Federated” will still be using your AD FS. And your domains are still “Federated”!

 

Changing Your Test Domain

Now it’s time to change your test domain and this is one instance where Microsoft can’t make up their minds. The documentation states that you switch the domains using “set-msoldomainauthentication” while MSFT support says that you should use “convert-msoldomaintostandard“. I chose to go with the documentation and so far I haven’t seen an issues as a result of that choice.

You hit up powershell on your computer, connect to your msolservice and then execute:

And just like that, all the users in that domain will start using PTA to authentication. You should ofcourse test this by logging into “portal.office.com” using a test user with this domain suffix and you’ll see that it is no longer routing to AD FS, now the login stays in Azure and you’ll simply be prompted for the password. After entering the password, the login goes into a queue that your agent picks up and authenticates and sends the result back and you should be in.

Pretty cool, but even cooler with SSO!

 

Seamless Single Sign-on

One of the better things about AD FS was the SSO experience. When done properly users never even realised it happened! And with PTA you can get the same user experience.

The first step is to add the URL “https://autologon.microsoftazuread-sso.com” to the Intranet zone list in MSIE which is easily done through a AD group policy. You should also add the ability to “Update the status bar” with the setting found in User Configuration -> Administrative Templates Policy -> Windows Components -> Internet Explorer -> Internet Control Panel -> Security Page -> Intranet Zone. All this is explain on this MSFT page.

This was the easy part. The tricky part is adding the SSO functionality to your Azure AD and your on-prem forest(s). And this is all done with powershell commands. These powershell should be run on the Azure AD Connect server, as no other server should have the cmdlets used.

What is basically happening is that it creates a computer object named “AZUREADSSOACC” in the “Computers” OU in the forest that the user you specified belongs to and this is the computer object that will be used to handle SSO authentication so pretty important to NOT delete it.

By now you should be done and the forest you added should now be visible in the Azure portal -> Azure AD -> Azure AD Connect and clicking “Seamless single sign-on” but listed as a domain (come on Microsoft!).

As you can see in the screenshot above, I also have a warning. This is because the token for this computer object that was created should be refreshed every 30 days so somewhere you’re going to have to setup an automated task to perform this refresh monthly. Unfortunately to do this you need domain admin permission! This is detailed on this page at Microsoft and there is a user voice request to get this functionality integrated into Azure AD Connect which is coming soon(tm).

But the process is almost identical to enabling SSO for the forest, but instead of “enable-azureadssoforest” you “update-azureadssoforest” like this:

And that’s pretty much it, right? Well ofcourse you should test that the SSO does work and how it behaves in corporate environment vs. “on the internet” but yeah, you’re pretty much done to roll this one.

 

Rolling It Out

The roll out process is pretty simple – you just convert another domain to managed using “set-msoldomainauthentication” cmdlet. You can do one at a time or just do them all. Just make sure that you tell your firstline support that this is happening because some users may call, either because they are having some issues or maybe even that they notice the difference in how the login pages look.

But you should do this outside of office hours since the change can take anything from 10 minutes up to an hour to replicate through AAD (see notes below).

 

Rolling It Back

The process to roll it back if something failes is rather easy – you simply “convert-msoldomaintofederated” from the AD FS server. It’s simply as that (although like I said before, it may take 10 minutes up to an hour to replicate). This simple fallback had to be used for us for two domains since those users used a terminal service farm that didn’t like PTA (which has since been fixed).

 

Notes from the field

  • Microsoft still confuses AD forest and AD domain in both the documentation and the Azure AD Portal.
  • After you have switched all your domains to Managed then your Azure AD will think that your AD FS no longer does anything with O365 anymore. So if you roll back, it will completely re-initiate the party trust for O365, completely removing all claim rules you previously had configured! It also means you have to use “SupportMultipleDomains” when switching back domains, or your AD FS will only work for one O365 domain!
  • We had issues updating users UPN domain suffixes after this switch. It turns out there is a feature in your Azure tenant that controls this which may be set to false! So enable that feature with:
    Set-MsolDirSyncFeature -Feature SynchronizeUpnForManagedUsers-Enable $True
  • During investigation of this UPN domain issue, MSFT support stated that we should have converted the domains using “convert-msoldomaintostandard” instead of the “set-msoldomainauthentication“, since it also changes the users. But I haven’t done that and it can’t be done post change since the cmdlet will only return the error that the domain is already Managed and it will then proceed with something bonkers – going through all on-prem AD user objects that it can find using the logged in user and updating them in Azure???

 

 

Skype For Business Online Users Disabled ?

I recently came up against an issue that I eventually needed MSFT to investigate and come up with a solution for and in the hopes of saving someone else the trouble, I’m going to go ahead and write a small blog post about it.

Symptom: The issue was that users in Skype For Business Online were stuck in the “Disabled” state and with the Directory Status “On-premises (hybrid)”. Nothing I did changed that. But other users belonging to the same domain (“contoso.com”) were enabled without issues?
After analyzing alot of the users I found that the attribute “HostingProvider” was set to “SRV:” for the users that it didn’t work for, but “sipfed.online.lync.com” for the ones that it did work for.

Root cause: The root cause for this is that when a user is provisioned in Skype For Business Online, “O365” checks in real time for a DNS record “lyncdiscover” of the domain of the user, in this case “lyncdiscover.contoso.com”. If the DNS record is set to “webdir.online.lync.com” then the user will be provisioned as an “Online” user and enabled. But if the DNS record is something else, then it will assume the user exists in an on-premise environment and it will be provisioned as an “On-premises (hybrid)” user and disabled. And in our case, sometime during the adoption of O365 services this DNS entry lost the trailing period (“.”), so for O365 it looked like “webdir.online.lync.com.contoso.com” and that’s why it assumed they were on-prem!

The Fix: Fixing the DNS entry is easy enough so that should solve it, right? Unfortunately this check happens when a user is provisioned and then it’s set. And the only way to trigger an update is to delicense the user (that is “removing Skype For Business license”), wait a few hours and then license the user again! That will trigger a provisioning process again and O365 will see the correct DNS setting and the user will be “Enabled”!

Thank you MSFT for the details regarding this process!

Checking AD FS Federation & Certificate Status

SCENARIO
You’re managing a large O365 tenant with AD FS service or multiple AD FS services and those certificates are expiring and needs replacing.

PROBLEM
The main problem is that there is no good way of telling ADFS to do something on only the domains that it actually is federated with, it’ll just assume it has them all. This may lead to some complications.

SOLUTION
I wrote this little script because I wanted to know
a) the domains that were federated to this ADFS service
b) the domains that were NOT federated to this ADFS service
c) the domains that hadn’t refreshed the signing certificate.
This little script, which must be executed on the ADFS service in an admin powershell, will first check the URL of the local ADFS service and then go through every domain in your tenant to see which match, and if they match will check the certificate. That way you know exactly which domains to look at.

It spits it all out in the console but also in 3 files in the c:\temp directory. And if you feel brave enough, you can uncomment the “update-federation” command to run that command.

Also it assumes you are already connected to the MSOL Service.

Check If Connected to SPOService in Script

SCENARIO
When executing SharePoint Online scripts you need to be connected to your “admin” site or the script will just fail if you’re not.

PROBLEM
When writing a script you can’t assume that you’re already connected to your SPO tenant and unlike the “msolservice” connect call you need to specify your “admin” URL which can be quite long. But sometimes you’re already connected in the Powershell session.

SOLUTION
Writing this little thing in the start of your script will check if you’re connected to the admin site and if not will call the connect-sposervice command with the URL already set.

Issues changing ImmutableID with error FederatedUser.SourceAnchor

Recently ran into an issue where a user in the on-prem AD had been deleted unintentionally and in the next sync his user went along with his mailbox.
Googling around I found a helpful article how to best go about restoring this. It’s basically about creating a new on-prem users and setting the new GUID on the recovered AzureAD user so AzureAD Connect can tie them together.
However, when trying to set the new “ImmutableID” with “set-msoluser” I got this error:
Set-MsolUser : You must provide a required property: Parameter name: FederatedUser.SourceAnchor

Took alot of Googling to realise what was wrong! The issue here is that you can’t set a new ImmutableID on a user in a Federated domain! So the trick here was to change the user to an “onmicrosoft” user, change the ImmutableID and then changing it back to the federated domain!

Hope that saves someone some headache.

What MFA method is used by how many users

SCENARIO
You want to know how many users are using SMS for MFA or mobile app to change user behavior to drive adoption of the MFA app.

PROBLEM
By default when users enrol with MFA they click “Next” all the way and end up with SMS authentication, regardless of what information we provide them with. And the way Microsoft stores this information isn’t very friendly for us to see this easily.

SOLUTION
I wrote this to demonstrate to management that users indeed doesn’t read the e-mails sent out to them which detailed that they should use “Mobile app” verification and what actually happened was they just clicked “Next” all the way and ended up with SMS authentication. In our case we ended up with about 2% of users chosing the application!

SharePoint Upgrade From Hell

This is going to be a wall of text. And 99% of the people I know aren’t even interested. But I’m writing this on behalf of every other SharePoint admin out there who are unfortunate enough to discover just how easy SharePoint is to break!

Little background: I’ve been working with SharePoint since about 2005. Not that long for some but long enough to know that after a few years of use a SharePoint farm has a few quirks in it and it’s a good idea to upgrade it. And you never upgrade an existing farm, you always start with a new fresh one and import all data! Now one of my jobs (!) is managing a 30k user corporate SharePoint – a business critical solution since all documentation are in there. And not only that, our entire BI solution is in there as well, complete with “PowerPivot” and “Reporting Services for SharePoint”… No pressure!

So now it was time to upgrade it from SP2013/SQL2014 to SP2016/SQL2016, including all BI solutions. We’ve gone through a “dev” environment, a “test” environment and even a “preprod” environment and everything went surprisingly well. There was ofcourse the usual glitches getting the BI features to work (and the S2S cert trust that is required for Excel with data source connection files now that Excel service moved out of SP to OOS!). But anyway, the preprod farm was so great that the plan was to take it into production. Our BI team didn’t see a big problem doing that in an afternoon on a weekday, whereas for me the biggest problem was the 1.5TB of data that needed to be shuffled and upgraded. And “even the best laid plans”, you know. I also knew that one of the biggest issue was network infrastructure which for a global company is so complex that the best way forward was to swap IP addresses of the servers so we wouldn’t have to change DNS or static IP routes anywhere, we’d just solve it at the load balancer level. So I managed to get a whole Saturday from the business to have SharePoint offline, but no more. After all, all documentation is in there!

That Saturday was last Saturday April 14th. I got up at 4am to start shuffling the data. By 7 that was done and I started upgrading the database with the normal “mount-spcontentdatabase”. Here was my first mistake (in hindsight). I had already written a script to do this, but that’ll come later. By 10 everything was loaded, upgraded and I proceeded to change IP addresses around and change it in the load balancer, then go through my long list of checks that normal user SP functionality works while our BI team were updating all of their things.

After lunch we had a “go/no-go” meeting and everything looked good. I also noticed at this point I had a case to create a new SharePoint site for a project, something I actually hadn’t tested since that’s not a “normal user SP functionality”. And that’s when the shit hit the fan! What I had missed thanks to my scripting was that one of the content databases had failed to upgraded and was now corrupt and when I wanted to create a new site it did it in that database since it was the “least used” and hence the error. “No problem, plan a) I’ll just delete this database”, right? Nope, SharePoint wouldn’t have it because the database wasn’t attached since it was corrupt. Yet I could see the sites in that database listed with get-spsite?

Tried a few things but couldn’t recover so I decided plan b) remove the web app and create a new and re-import/re-mount this corrupt DB, all other DB’s were already upgraded successfully so not a big operation. Well, SharePoint wouldn’t have that either – it couldn’t dismount this database because it was corrupt so I couldn’t remove the webapp! I was completely stuck with a broken web app that I couldn’t remove because of a content database that wasn’t mounted?!

So plan C) rename that webapp with the corrupt database and give it a nonsense URL so I could create a new web app with the proper URL. That seemed to work but when I tried importing a new backup of this content DB it didn’t import any of the site collections! .. digging around I could see that the sites in the broken webapp, with the new nonsense URL, still had the original URL! It couldn’t update them because… there was no content DB attached to them! I dug around in SharePoint Manager (which was designed for 2013 I know) but it kept crashing when I clicked any of the sites in the broken webapp.

So there I was with a broken web app with a corrupt contentdb with sites occupying the URL I needed to create our proper web app. Came to the conclusion that the config db was pretty much fucked at this point at now we’re at 2pm. Best option available to me at this point was calling Microsoft premiere support case with a severity A case. I’m pretty sure if I had gone for that they would have looked at it, made the same determination as me and said “since this is a farm not yet in production, I’d say the best way forward is to recreate the farm”. During that time our BI would be in SharePoint 2016 but the “big” web app in 2013 on separate IP addresses! God knows how the network would handle that and getting the engineers in India to change firewall routes in less than a week wasn’t that likely. Because rebuilding a new farm in production would take at least a week, right?…


After clearing it with my supervisor that this was indeed the best way to solve it NOW! All other options led to some unknown hellhole – going back was always a possibility no matter what.

I got a green light and Red Bull at about 3pm …

 

  • SP Product Config Wizard to detach all server from the farm
  • delete all databases from SQL except all the (successfully) upgraded content DB’s
  • thank myself for having saved all of the “AutoSPInstaller” response files
  • create a dummy web app to upgrade the corrupt DB (no way I’m doing that in the proper web app again!)
  • eat the food my awesome wife brought me
  • recreate all webapps
  • restore all content DB at about 1.5TB
  • upgraded the service apps

Basically I had done at least a weeks work in 7 hours and all in production environment!

The “Done!” mail got out at 9pm! Now I’m not one to brag, but any SharePoint admin must be impressed by that! Hell, even Scotty would be proud! I spent a few hours on Sunday cleaning up the mess and sorting out the BI issues (since this was a new farm there were a lot of BI configuration that was lost) but by Sunday 6pm everything was fully operational and I promptly went to be and slept like a baby. And one of the first things to hit me on Monday morning was “why is Managed Metadata empty” because yeah, in my haste I forgot that little thing 🙂

How was your weekend?

NASA. Again.

K mår bättre än bra

Yeah, I don’t know why but NASA has a special place in my heart. Maybe cause I’m a tech nerd, a scifi geek or just like to have my shit together, or maybe it’s some visionary part of me I don’t know I have but I just love it. In everything from fictional NASA in “Contact” to proper NASA in “From the Earth to the Moon” it’s an inspiration. That’s one of the reasons I made a point at going to Kennedy Space Centre when I was in Florida and one of the reasons that trip was an awesome success to me. And one of the reasons I didn’t have a problem opening up my wallet in the gift shop!

And yesterday I saw the movie/documentary “Mission Control” – about those 20-something engineers that made up the mission control team. I really recommend catching it on Netflix or renting it on bluray cause it’s awesome. One of the things that surprised me was the interview with one of the engineers who was there in the trench for a lot of the Apollo missions, even the moon landing, that said he regretted doing it because of the toll it took on his family! I mean, it’s one of the things I can only dream of doing so hearing that makes you wonder what really is important – making a mark in history or being with your family.

Another NASA “merchandise” I can recommend is the book “View From Above” by austronaut and “photographer in space” Terry Virts. You can get it from Amazon or something but it’s well worth it. Not only because of the awesome pictures that makes you feel tiny and insignificant but also because of the stories he has to tell.

Manage External Sharing in SharePoint Online

SCENARIO
You’re managing a SharePoint Online environment and you want to know where external sharing is enabled.

PROBLEM
The problem is that Microsoft hasn’t fully launched a way of getting a good overview of this where you can change it. When you first enable sharing for example alot of sites will have it turned on by default etc. The new SharePoint Admin center has the ability to add the “external sharing on/off” column in the list of sites but that is very limited and you can’t enable or disabled it.

SOLUTION
Fortunately there is a very good attribute that you can retrive to get this and alter the external sharing setting called “SharingCapability”.
So using that you can get a list of all sites and what the status is of them, or you can filter for all that have it enabled or disabled:

Once you have that you can script it so you can disabled external sharing on all sites by doing this:

The reason you want to do this in a “foreach” is there will be a site or two you may get an error that you can’t change the setting, so that would exit the command on that error.

How many has what Exchange storage limit?

SCENARIO
You’re asked how many users has what storage quota/limit in Exchange

PROBLEM
The problem originates from MS saying that the standard Exchange Online mailbox is 100GB in size. But some of our users are reporting they “only” have 50. I thought this was a minority if people so not a big thing. My manager disagrees.

SOLUTION
I began writing quite a complicated powershell for this but when I looked at it after a coffee break I said to myself “there’s gotta be a better way”. And sure enough it is. I’ve simply never used the “group-objects” function before! But now I can clearly get a report that’s just 3 lines long!

SharePoint 2016 Installation Errors

SCENARIO
You’re trying to install SharePoint 2016 on a Windows 2016 server and thinks just aren’t going well.

PROBLEM
To be honest I don’t know how else to explain the problem in any other way than Microsoft’s Windows Server 2016 team was in a feud over lunchboxes with the SharePoint 2016 devs because there is no other way to describe the complete incompatibility between the two!

SOLUTION
I’d say “Google it!” but that’s probably what got you here in the first place!
The first problem is the prerequisite installer that can’t configure Windows IIS role or download things. Fret not for there is plenty of help to find. When first running the prereq you’ll probably get this error: “Web Server (IIS) Role: configuration error”. To configure the IIS use this Powershell :

Make sure to edit the source file to the Windows Server 2016 ISO!

The next place you should look at is this blog by the Microsoft Field Engineer Nik. Although be careful about some of his links as those are outdated and replaced with new versions, although downloading the version he’s linking will still work. He even provides a script that will run the Powershell to configure everything. Why this isn’t on the SharePoint 2016 ISO is beyond me!
But even when downloading all of that and installing it properly I was still faced with this error when trying to setup the farm: “New-SPConfigurationDatabase : One or more types failed to load. Please refer to the upgrade log for more details.“. Going through the install log I found this: “SharePoint Foundation Upgrade SPSiteWssSequence ajywy ERROR Exception: Could not load file or assembly ‘Microsoft.Data.OData, Version=5.6.0.0, Culture=neutral, PublicKeyToken=31bc3856cd365e35’ or one of its dependencies. The system cannot find the file specified.

It seems that the WCF prerequisite file when installed using the Powershell method of manually downloading and installing it! Fortunately the quick fix is to find the file “WcfDataServices.exe” in your profile directory (i.e NOT the one you downloaded!), running it and choosing “Repair”. Only then did SharePoint 2016 install properly!

Get Users With Multiple Licenses

SCENARIO
You’re managing a large O365 tenant and you want to make sure there are no users that have multiple licenses assigned.

PROBLEM
The original problem is that you actually can assign a user with a F1, E1 and E3 license and end up paying three times for a user! Next problem comes with how license information is stored and retrieved with Powershell.

SOLUTION
Here is little code that will read out all your users and go through each one to make sure they don’t have more than one of the licenses assigned. It should work as long as Microsoft doesn’t change the _actual_ names for licenses!

The script can ofcourse be enhanced to write a log or even mail a log to an admin if you want.

Check E-mail Addresses From File

SCENARIO
You’re handed a list of e-mail address for mass mailing from HR and they need to verify that all e-mail addresses are valid and won’t bounce “like last time”.

PROBLEM
There are a few problems with this. One is the fact that not all e-mail addresses are the primary e-mail address and won’t show up in a normal search.

SOLUTION
I put this little script together that will first connect to your MS Online tenant, then read all MSOL users into an array, import the CSV file containing the employees, go through each row and check that the e-mail address from the file in the column “employeeemailaddress” exists as a proxy address on at least one user. If not it writes out the e-mail to a log in c:\temp. Nothing too advanced, just a few things put together to achieve a very, VERY tedious task when you get a list of 10.000 e-mail addresses!

This can also be modified to check if any other attribute exists or not on users if you want to, it was just for this scenario that I had to check e-mail addresses! It can also be modified to read out the local AD and not the Azure AD, ofcourse.

Please comment out the first two lines if you run this more than once in a Powershell window since the list of users is already in the variable and reading out all MSOLUsers can take a very long time!

Get User With E-mail Address

SCENARIO
You’re getting some error that a specific e-mail address can’t be or send mails. But you have no clue about which user/mailbox is the owner of this specific e-mail address

PROBLEM
Most of the times this isn’t a problem, the Exchange Management Console or EOL Admin Center will do the trick. But sometimes it can be a bit tricky if the e-mail address is to say a public folder, which isn’t scoped in the search.

SOLUTION
This quick little powershell will do the trick for you to find it:

Credit goes to Fulgan @ ArsTechnia for the post here.

Enable Versioning On Entire SharePoint 2013 Application

SCENARIO
For some reason, probably money, you can’t use a proper backup solution for your farm. So you want to use versioning as a cheap mans backup.

PROBLEM
Going through every document library in every site in every site collection in every application to enable versioning isn’t possible. And there is no way to specify in Central Administration or declare a policy to enforce this.

SOLUTION
This powershell script will do the trick for you. It’s written to enabling versioning for an entire web application (with easy alteration it can be scoped to a specific site/site collection). What’s neat about this is that it will not change settings on the document libraries that already have it enabled! It will not enable minor versioning, but you can just enable that if you want.

As always, use on your own risk and test in a test environment first and then scope it to a test site collection in production farm!!

Credit goes to Amrita Talreja @ HCL for this post which is the basis for this Powershell script.

Get All MSOLRoleMembers

This is a small little script I wrote for going through all administrator roles in your O365 tenant and listing out the members of each. This can be handy if you feel like you’re losing control over who has what permission in the tenant or someone says the classic “I want what he has”.

Licensed Shared mailboxes

SCENARIO
You’re managing Office 365 for a company. You start seeing those licenses count down and you don’t know where they are going. Then it’s time to check if you have “Shared mailboxes” that are licensed!

PROBLEM
The problem here is that sometimes user mailboxes are converted to a shared mailbox. Maybe it’s an employee that left but you still want to access the mailbox, or maybe the mailbox was accidentally created as a user mailbox to begin with. And Microsoft even has a button in Exchange Online for converting to Shared mailboxes! But the problem is that button does indeed convert it – but it’s not deactivating the license! This is probably working as intended as deactivating the license has some other affects like legal hold and other services.

SOLUTION
This little command will list the mailboxes that are tagged as Shared mailboxes and lookup if their Azure AD object is licensed or not. It requires you to already have a remote PS session with Exchange Online as well as a connecting to MSOL service:

Then you may want to write something like this to automatically remove the licenses. Change “yourlicenseplan” to.. well, your license plan 🙂

Credit goes to Mohammed Wasay (https://www.mowasay.com/2016/03/office365-get-a-list-of-shared-mailboxes-that-are-accidentally-licensed/).

SharePoint 2013 error “Exception from HRESULT: 0x80131401”

SCENARIO
After patching a SharePoint server with normal OS patches and reboot you are no longer able to browse to the applications or Central Admin. When looking at the log you see this error from C2WTS:
“An exception occurred when trying to issue security token: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401).”
This was on a normal SharePoint Foundation 2013 server (build 15.0.4569.1506)

PROBLEM
If you check this TechNet forum post it seems to be related to “third party monitoring tools”. Unfortunately, SCOM is considered a third party tool in this case. And as it happens, we have just upgraded to SCOM2016!

SOLUTION
According to the TechNet forum post (and this official MS post) you should update or disable any third party monitoring tool. So, uninstalling the SCOM monitoring agent, rebooting, reinstalling it with “NOAPM=1” parameter will solve the issue (atleast it did for me).
However, if that it not an option (sometimes you can’t just reboot a critical server!), disabling Load Optimization does work, even if means your SharePoint is now unsupported. So I’m posting this for all us “it needs to be fixed now and I can’t find, update or disable whatever DLL is causing this!”-techies! So setting these registry keys works:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework, create a new ‘DWORD (32-bit) Value’ named “LoaderOptimization” with a value of “1”.
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework, create a new ‘DWORD (32-bit) Value’ named “LoaderOptimization” with a value of “1”.

But as those MS people will tell you, it really isn’t a recommended solution (which is weird since it originated from MS support!).

New-MSOLUserPrincipalName

SCENARIO
You’re changing the e-mail domain of a user or even a bunch of users. After that you also need to set their UPN’s to reflect the change.

PROBLEM
The problem is that Azure AD Connect service doesn’t currently support changing domain of a UPN of an object that is already synced! So you have to run a powershell command to change it. But it get’s even more complicated because you can’t change the UPN from one federated domain to another without making it “unfederated” first.

SOLUTION
Enter New-MSOLUserPrincipalName, which is a function that will take the user with the current UPN ($UserPrincipalName), change it to a temporary UPN with the domain extension “@[your tenant].onmicrosoft.com” and change it to the new UPN ($NewUserPrincipalName).

Thanx to Johan Dahlbom for this one!

Download PS1 from Dropbox

Download PS1 from Dropbox

Upgrading AD FS 2012R2 to 2016

SCENARIO
You have a working ADFS farm running version 3 on Windows 2012R2 and want to upgrade to ADFS 2016 delivered in Windows Server 2016.

PROBLEM
The problem is that this is, if you ask Microsoft, a very straight forward “next-next-finish” process to do as the only TechNet article I found about it makes it look pretty straight forward. But that article was written for Windows Internal Database (there is now also one for SQL cluster backend. Also you’ll notice at the bottom that it’s written for Technical Preview of Windows Server 2016 and also assumed you have no AD group policies that may break stuff! So there are still alot of things that can, and will, go wrong if you follow that procedure.

SOLUTION
There really isn’t one solution since there are so many issues you may run into but I managed to work through them all. But here are my comments to the TechNet article and where things went wrong for me:
2) It’s never showing in a screenshot but it is shown in the next – you have to chose to join an existing farm, the default option is creating a new farm which is a totally different thing!
But even after going through the setup process succesfully after patching and rebooting I got the error 1297 “A privilege that the service requires to function properly does not exist in the service account configuration. You may use the Services Microsoft Management Console (MMC) snap-in (services.msc) and the Local Security Settings MMC snap-in (secpol.msc) to view the service configuration and the account configuration“. As it turns out, this is a policy issue with the Windows Server 2016 baseline that limits who and what can “Log on as a service” and “Generate a security audit”. Creating an override policy for this and adding the service account running the AD FS service solved this issue for me! (thanks to https://blogs.technet.microsoft.com/pie/2015/09/04/adfs-refuses-to-start-error-1297/)
3) This is actually very important later on knowing which server is primary and not!
4) and 5) These are confirmed as not required if you’re running a SQL cluster backend. However, you still need to check later for which server is primary and not.
6) This entire Powershell is just wrong and not accepted at all, atleast in my environment! You’re much better off starting the Remote Access Manager and starting the Wizard from there. This will allow you to chose the certificate in a dropdown without knowing the thumbprint. But this is where I ran into problems and lot’s of them!

The first problem I had when configuring the WAP was connectivity resulting in the error “An error occurred when attempting to establish a trust relationship with the federation service. Error: Unable to connect to the remote server”. This was first due to physical firewall, then the local firewall policy settings and in the end that the service itself was down! So this was basically alot network issues, not the biggest thing in the world.

Now that that was done with I ran into the next problem that caused so much headache for me – “An error occurred when attempting to establish a trust relationship with the federation service. Error: Unauthorized. Verify that the service account has administrative access on the target Federation Server.”! The account that the WAP uses to connect to the internal AD FS server with that has to be a local user and local admin account on the internal AD FS server (since the WAP server shouldn’t be a member of the same domain as the internal AD FS servers). The problem is that there is a group policy baseline for Windows Server 2016 that denies logon from the network for all local users (“Deny access to this computer from the network“)! This resulted in the error since it wasn’t allowed to login with anything but the console! Setting that to only “Guest” should be enough for this.

So after getting that problem solved I got the next error – “An error occurred when attempting to establish a trust relationship with the federation service. Error: Internal Server Error“. Looking at all logs and events and I couldn’t figure out what tha hell was causing this issue. Well, as it turns out it was related to step 4 and 5 which you shouldn’t have done if you’re running SQL backend! When you point to the internal AD FS service address (the web address sts.xxxxxxx.com) you’re supposed to use a host file to control that and point it to the load balanced IP address. Well when I did that I always ended up on a server that was NOT the primary computer and therefor I couldn’t add the WAP! When I changed the host file to point directly to the IP of a server that was Primary computer for the farm it worked! Just remember to change this back since you don’t want the WAP servers point to one specifik AD FS server.

That is as far as I’ve gotten as the rest of the upgrade involves upgrading the forest and domain schema which I’m really not ready to do.