About the Author
Author

stoff

Election Time

So it’s time for a general election again here in Sweden as is traditional every 4 years. On September 11th we’re heading out to cast our votes and both me and my wife definitely will. My son still has to sit out this and next election.

Opinion Polls 2022-08-18But I really have a problem with this election. Europe is suffering it’s worst heatwave ever and is drying up fast. There’s water shortages, rivers are so dry that boats can hardly transport freight and fish are dying by the tons, people are losing their homes and livelihoods and it’s all kinds of chaos caused by global warming. And you’d think that even after ignoring Greta protesting outside of parliament for over 4 years that surely this would wake politicians up? Nope. And why would it since the media isn’t holding them accountable. And worse yet, the voters aren’t holding them accountable.

Jimmie put it best during the “Aftonbladet” debate last night when he ‘attacked’ our environmental party by saying – “you only want to make things worse for everyone!”. And that’s exactly the problem. Elections used to be about people asking themselves “what party align best with what I want?” and vote for that. And nobody wants to make things worse for themselves. Nobody wants to give up their summer vacations by the mediterranean, take the bike instead of their car, go vegetarian and generally stop buying new stuff and focus on recycling. But in my opinion the time for “what I want” is over – now it’s critical for everyone to think “what does the planet want?”.

And it’s clear over the past 3 years that it certainly doesn’t want us! First it gave us the wildfires in Australia, then covid, then monkeypox and now a heatwave from hell.

So I wish that for once the voters would think more long term and more globally. Because yes, in the short term not going on vacation in Greece may suck but at least there’s a chance there will be a habitable Greece in 50 years!

As for who I’m going to vote for? Well I also always ask myself “what party align best with what I want?”. And I want a positive future for my son and the planet. And again there’s only one party that is taking any kind of attempts at doing anything about it so again I’m voting for our environmental party. I wish there was at least one more party that would at least try so I can look for other things that align with what I want but … nope!CO2 emissions Under 2% matters Sorry but saying that “what Sweden does doesn’t matter and won’t affect anything” is a poor excuse for not doing anything (read “The Guardian” article here). And building nuclear powerplants won’t solve nothing since they won’t be operational for at least 12 years and by then the waters in the seas will be too hot to cool them (read analysis here, only in Swedish I’m afraid). And that’s about it. That’s all the options right there. So again I quote one of my all time favorite movies – “damn you for forcing me into this position!” (Ed Harris as General Hummel in “The Rock”).

But the way things are going we’re going to end up with team “do nothing since we’re so small what does it really matter” & “build nuclear powerplants”. So that’ll be interesting and I really do hope it doesn’t matter. But I wish I could say “we did our best” to our son but it really doesn’t feel like we did. Even going flexitarian, voting green every time, driving electric car and solar panels on the house, it’s not enough.

My Bout With Covid

As previously mentioned Covid did eventually get me but fortunately I had 2 doses of the vaccine so it was fortunately mild-ish.

But I remember back in January of 2020 when I and a colleague at work discussed it and started hypotheticals about how bad it would get. And in February I realised that it wasn’t going to just go away so I did a bit of prepping- and yes I was one of the reasons there wasn’t enough toilet paper at our local store. Also bought masks, blood oxygen checker and stuff like that just in case.

And then me, the wife and our son got knocked out by something that could have been Covid but in late February 2020 they were only testing people that hade travelled to the high risk areas since it wasn’t out in the wild in Sweden yet. They said. And then we went to the Swedish Eurovision finale with thousands of people crammed into an arena. Right after that the shit hit the fan and those events became unthinkable because by now it was officially out in the wild in Sweden too.

And even though we took precautions and we spent over a year doing our best to social distance and going out as little as possible, I was never really afraid that I wouldn’t make it if I did get it because.. statistics! My wife’s grandmother unfortunately died from it just a few weeks shy of her 90th birthday. But when Adam Alsing died from it I actually did get a bit nervous.

At work this was a disaster since we pretty much had to stop working. No one was allowed to travel so very little we could do. After a few months of furlough for most of the staff (except me that kept on working at 100%, but from home) they eventually had to cut down on the staff. A lot of people in my IT operations team, a few of which I considered friends, was let go but I made the cut. Unfortunately my wife didn’t make the cut and her job so she was unemployed for a few months before finding another job.

Fast forward to October and my sister got it. She works at a hospital so it was just a matter of time. And she got it bad – not bad enough for hospital but bad. I don’t think her sense of smell and taste is fully back yet! But I was mostly worried for my parents and for my other sister in Scotland with comorbidity.

The winter was uneventful as was spring of 2021. During the summer we had a “staycation” on Gotland during which I felt that my place of work had lost its appeal a bit after so many people had left and things were really slow and no one knew how long this would go on for, so I decide to look for another job and pretty quickly found one. And I also got the vaccine as fast as possible because I didn’t want to be worried about it all the time, I had lots of other medical things going on so one less to worry about.

During winter I started my new job and a few months into 2022 our son got a mild case of the flu but we kept him home just in case. And I was still working from home. And the week after I got symptoms so I ordered a test and yep, it was covid! I was knocked out and in bed for a few days but nothing bad. My wife got it after me but she had just taken her third dose so she was only knocked out for a day. So that was disappointingly uneventful but I guess we have the vaccine to thank for that. Go science!

Health Anxiety

As I mentioned in the last post I’ve been struggling with mental health during 2020 and 2021. After talking with my psychologist trying to explore exactly what it is it was pretty clear that the major cause behind it was me worrying about my health and lately also my son’s.

I don’t know if I watched too much “House” or if it’s the availability of information but I’ve been self diagnosing myself for years and most of the time it was harmless. But since summer of 2019 I’ve taken it up quite a few notches to the level that I got panic attacks over small things. It could be a pain in my arms, something with my breathing, it was always something terminal. Or something serious enough that I couldn’t work anymore or something like that. Unfortunately it didn’t stop there – I would keep playing out the scenario that would end with us losing the house and “what would my wife do if it really was terminal”, all those thoughts that I couldn’t help having and losing sleep over. It doesn’t help that I know on the intelligent level that it’s ridiculous, when my mind started racing away it was too late.

But it didn’t stop there, when I started doing the same every time my son complained about something being wrong that’s when it got totally out of control. In my defense – the doctors still haven’t diagnosed reasons behind some things. Then you add a pandemic on top of that and eventually I broke.

But seeing a professional and talking about it really helped. I got some tips how to try to control it, also some breathing exercises and I even tried happy pills for a while. And now I still find myself having these thoughts but I’m way better at controlling it, being able to distract my mind with something and it’s been quite a while since I lost sleep over it! And it helps that so far after some 15 examination (everything from MRI and a camera up my ass) no one has found anything serious wrong with me.

Slowing Things Down

It’s been almost a year since my last update and there are a lot of reasons for that. The primary being I almost burnt myself out last year. And I changed jobs. And got covid. And stuff in the family.

But first thing first – how I almost burnt myself out from stress last year.

Me having a vacation on the island of Gotland

I pride myself in being efficient and optimising everything I do. This is required at work but when you start optimising how to empty the dishwasher it takes on a whole new level. And a few years ago my lovely wife almost got burnt out at her work, so we both try to be observant for one another when it comes to stress levels. And she’s told me that ever since we moved to our new house my stress levels have gone up and my mood gone down. Stupid male as I am I didn’t listen to her. But after 2 years of uncertainty at work thanks to a bankruptcy and covid, taking on extra on call duty to make more money, a year of panic attacks over my and my sons health (we’re fine!) and the life during the pandemic which cause agoraphobia, as well as a member of the family being very, very ill, after last summer I found myself in a meeting at work that I had organised and 10 minutes in I forgot where I was and what I was doing. My body had had enough. I had a meltdown in my head and took the rest of the week off. After trying to work 2 days the following week my body said “stop it” and I took 5 weeks off to get myself together. I sought medical help and met a psychologist for a few sessions and got a lot of help from my wife and eventually I made it back to work. But during this time I was also transitioning to a new job I managed to get and fortunately for me that job doesn’t require 24/7 on call duty and I’m starting fresh with no baggage or expectations of being a miracle worker which I had put on myself at my old work. So I’m going to set expectations low for myself at the new place but fortunately I won’t have to worry about us going out of business because of covid, heatwaves, airport chaos or any of the other things you have very little control over!

Me taking my psychologist’s advice and going on a walk in the forest and stopping to smell the flowers.

So to sum it up – I almost hit a brick wall but fortunately I had the help and means to stop it in time and I’m very grateful for my family and friends helping out because it would have ended very badly. And I have a newfound respect for mental health!

And my advice it simply – listen to your body. It will tell you way before it gets this far. I didn’t. And take things slowly. Go for a walk in a nature. Put your fork down between bites. Don’t try to get as much done as possible. Put your mobile away.

Solar Powered

So we finally did it. We got solar panels installed on the roof.

I’ve been sceptical about it since our house is facing east/west and for the highest efficiency you want south facing roof. But it was explained to me that with a east/west roof you can put panels on both sides to get solar electricity from when the sun goes up until it goes down. It might not be as efficient per solar panel but when it comes to getting the most amount of hours out of your installation it’s actually pretty good even if you have to pay more for more panels.

And after this winter with very high electric bills, finding out some of that was imported “brown” electricity and also finding out we’d consume about 16,000kWh annually, we decided to give it a shot. So we invested about $20,000 for an installation that would generate about 8,500kWh per year and got a company to do the entire installation for us, from climbing on the roof to hooking it up, they handled it all.

And now, after less than three months it’s produced over 4,000kwh! Granted, those months have been June-July-August but still. Also the electric bill have so far been in the negative, meaning they actually owe us money!

So so far we’re pretty damn happy about it. Fortunately we don’t have to look at our own roof much 🙂

Between 60-70 kWh per day in July!

Finally Double Vaccinated

Yay – I ‘m finally fully vaccinated to combat covid19 should I get it!

During this entire pandemic I’ve been trying to be cautious – staying at home as much as possible, avoid crowded places, not going out to restaurants or anything like that, order things on the web, basically doing my best to limit the risk of getting covid. But at the same time you still want to live, not just survive right so there’s been a few exceptions. I’ve had to take my woman out to dinner a few times and we’ve had to go on a crowded boat to Gotland and so on. I haven’t been panicking about “oh my god I’ll die if I get it!”. I’ve followed the statistics and risk of dying and I’ve been countering them as best as I can. For example I’ve started to… exercise! Yeah I know, but when statistics show that the biggest contributing factor to sever Covid, besides age, is obesity then you bet I’m on that treadmill!

But now that I’m fully vaccinated, how much of this will change? Covid hasn’t gone away and the mask wearing is still a thing for me to protect others. So until the booster shot I think I’m still going to be somewhat conservative of being in crowded places.

As far as going back to work, the biggest issue there is the actual going to work. They still recommend people working from home and one of the reasons is to avoid crowded trains and buses and since I can’t take the car to work (way too expensive) that means I’ll be working from home until either told by my superior I can’t or the recommendation goes away in September. But I’ll miss it, even if I haven’t been as productive as I would have been at work it’s still been way better for us as a household with me being home more and not having to spend about 3 hours commuting.

Exporting Assets from Freshservice using Powershell

I recently got a request to write an automation to check the warranty status of all our computers. Simply request, huh?

I decided the best way was to write a script that would export all the assets from our Freshservice instance and get the serial number of them. The next stage is to write a script that actually checks the warranty status with HP, but currently their warranty API is offline so that will come later. Right now we’re only exporting the assets and getting the serials from Freshservice.

You will need to find your API key in your profile for Fresh, and your instance URL which you already should now. The next thing to check for is if your ID for serial number is the same, in our case it’s “serial_number_26000476017” but I guess that may vary.

#################################################
# Export all assets and get serial to check with HP warranty
#################################################

# API Key
$FDApiKey="XXXXXX"

#################################################
# Prep
$pair = "$($FDApiKey):$($FDApiKey)"
$bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
$base64 = [System.Convert]::ToBase64String($bytes)
$basicAuthValue = "Basic $base64"
$FDHeaders = @{ Authorization = $basicAuthValue }
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::TLS12
# The Doing part
$FDAssetData = ""
$Output = @()
$i = 1
do
	{
		$FDAssetDatatmp = ""
		$FDBaseEndpointSummary = "https://<your fresh URL>/cmdb/items.json?page=$i"
		# You may want to write this out to check how many pages it loads
		# write-host $FDBaseEndpointSummary
		$FDAssetData = Invoke-WebRequest -uri $FDBaseEndpointSummary -Headers $FDHeaders -Method GET -ContentType application/json | ConvertFrom-Json
		$Output = $Output + $FDAssetData
		$i++
	} while($FDAssetData)
Write-host -foregroundcolor Cyan "$i pages imported..."

#Now let's go through every row and only filter out the HP computers and get the serial number.
foreach($Outputrow in $Output)
{
    $name = ""
    $serial = ""
    $name = $Outputrow.name
    # This levelfield for serial number may not be the same for everyone, please check this up!
	$serial = $Outputrow.levelfield_values.serial_number_26000476017
    # Using the example of HP here
	if($Outputrow.product_name -like "HP*")
        {
            write-host -foregroundcolor Green "Here you can check warranty for $name with serial $serial..."
        }
    else
        {
            write-host -foregroundcolor Yellow "Asset $name is not HP computer."
        }
}

Credit goes to Mark Wilkinson for writing the basics of this in a forum post here.

Still Alive and Kicking

Hey. It’s me. I know, it’s been a long, long time since I updated this place. But the past months have been pretty boring. Which considering everything that is happening is a good thing!

It got a bit depressing after summer when my wife lost her job and was unemployed for a while but she got a new job so that’s over and done with. But other than that we’re pretty much in a holding pattern until we get vaccinated (or infected) so we can start living again and society starting to open up. Because I’m pretty tired of our house by now after spending almost 12 months here. I’m one of the lucky ones that can do 99% of my job from home but that also means I don’t get out much. Or at all…

But so far we’ve been able to avoid getting covid19, the closest it has gotten was my sister that got symptoms day after she was here for an evening of F1. She had pretty rough 2 weeks and is still suffering a bit unfortunately even after 5 months!

So here’s hoping we can get vaccinated and start living again and people can start travelling so our company doesn’t go down as a result of this considering how much we’ve survived. Tsunami, ash clouds, bankruptcy, we beat them all!

We managed to find a new ice cream joint in Stockholm that we didn’t know about so we won’t have to travel to Söderköping that much!

Removing Domain From E-mail Addresses

Here is a script I had to put together to remove all e-mail addresses containing a specific domain (“contoso.com” in this example) on all your objects in your on-prem AD.

Naturally this script can’t be altered to only include a specific subset of users, just change the “get-adobject” to match.

What sets this script apart from all the others you can find out there on the Internet is that this uses on-prem AD. Reason for this is that a lot of organizations out there are using their on-prem AD to sync their users and groups and not all of them have complete on-prem Exchange infrastructure to manage this so using this seemed a lot better. At least it did for us.

Of course this script writes to the console what it does every step of the way and it writes to date stamped transcript, making tracing and rollbacks easy-ish.

It won’t remove the e-mail address if it’s a primary e-mail since that may cause issues but it will flag it in red so pay attention to your console or search the transcript for “primary”.

$DateStamp = Get-Date -Format "yyyy-MM-dd-HH-mm-ss"
$Logfile = $LogFile = ("c:\temp\removing_mail_addresses-" + $DateStamp + ".txt")
start-transcript $logfile
$Mailboxes = Get-adobject -Filter {proxyAddresses -like "*contoso.com"} -Properties *
foreach($mailbox in $Mailboxes)
{
    $DN = ""
    $upn = ""
    $DN = $mailbox.DistinguishedName
    $upn = $mailbox.UserPrincipalName
    write-host "Processing User $upn ..."
    $i=0
    while($i -lt $mailbox.proxyAddresses.Count)
    {
        $address = $mailbox.proxyAddresses[$i]
        write-host -NoNewline "Processing mailadress $address..."
        if ($address -clike "smtp:*" -and $address -like "contoso.com" )
        {
            write-host -ForegroundColor Yellow " removing address"            
            Set-ADObject -identity $dn -remove @{ProxyAddresses=$address}
            $i--
        }
        elseif($address -clike "SMTP:*" -and $address -like "*contoso.com" ) { write-host -ForegroundColor Red " MAIL ADDRESS IS PRIMARY!" }
        else { write-host -ForegroundColor Green " no action taken." }
        $i++
    }
    write-host "---------------------------------------------------------------"
}
Stop-Transcript

Vacationing In Sweden

Since I’ve worked for a charter company for a while now we’ve gone abroad to Spain, Greece or Turkey for the past decade or so. But given the current situation that wasn’t possible so we looked at around at some place to go in Sweden together with some friends. And we ended up with Daftö Resort – a camping resort on the west coast of Sweden which we hardly ever go to. So we booked a cabin from Thursday to Sunday and packed the car for a 6 hour road trip and off we went!

We made a quick stop in Mariestad to eat lunch and were happily surprised by the local restaurant “Kingfisher”, that Google suggested. Then some ice cream and off we went again.

Me and the wife in StrömstadWhen we arrived and tried checking in they looked at me like I was the town idiot. Which of course I was because we had booked Friday -> Monday! So there we were, outside Strömstad on the west coast with nowhere to stay! 4 calls later we had 2 rooms booked at a local “spa resort” which was a pretty damn good hotel. But they clearly didn’t care for the covid restrictions like “no hanging around the bars” so I wasn’t too comfortable there.

After spending the night there we went into Strömstad to explore what this coastal city had to offer. And there were some nice vistas and atmosphere and we even started fishing for some crabs down in the harbor. Not to eat though, just for fun!

PoolThen back to Daftö resort and we were allowed to checking and get comfortable in our rather large 8 person cabin. Unfortunately they had some covid restrictions here that mostly applied to how many people could be in the pool area at any given time but we managed to get a spot to get some pool action but you can clearly see that people weren’t too concerned about covid here. Safe to say I wasn’t jumping in when it looked like this!

Another covid thing was that they had shut down their little ammusement park which was really sad for our kids. Even though we had told them it was closed so they didn’t mind much but we knew how much more fun they would have had with there. But I fully support them not having it open given the current situation.

Crazy golfWhat was open was their crazy golf course. And it was really, really good. Nice designs and themes, well maintained and if it wasn’t for the queuing I’d say it was perfect. I wait, we were at the front of the queue all the time! Unfortunately the restaurants at Daftö left a bit to be desired but I guess for a camping site it’s ok – we all got something to eat and it wasn’t horrible but it wasn’t up to our standards of “going out”.

MirrorWe also got some time bathing in the sea and just kicking back in the cabin enjoying a few beers and I have to say that cabin was awesome – it had everything we needed and more (didn’t expect wifi!). We also tried some more restaurants in Strömstad and I can recommend the tapas restaurant “Il Toro” – but stick to the tapas. Not only was the tapas really good but the waiter even made a movie reference based off my wifes T-shirt that even I didn’t get, so kudos for that!

Then on Sunday it was time to pack up the cars and go back home. 6 hour drive back then unpacking and settling in to try to enjoy the F1 race from Silverstone. Which is hard since the Mercs are 1 sec faster than anyone else.

So that sums up our vacation for this year. Which was fun but also sad we didn’t do it more because our son really had fun without a computer!

When It Rains It Pours

I’m unfortunately not talking about that episode of Star Trek DS9 – it’s life in general right now. Not only is there a global pandemic that is crippling society and tanking our company and already caused my wife to lose her grandmother and her job.. now we got cancer in the family as well.

So now I’m gonna be one of those people who say – FUCK CANCER! Fuck it hard!!! Statistically it’s kind of amazing my family has been spared this long and I always assumed it would hit me because… well, not only am I the most expendable of the lot (in my opinion), I also live my life the least healthiest with all the chemical stuff I put in my body not to mention the red meat and then some lack of physical activities on top of that and sitting in front of a computer all day long, I thought it was a given! This fear is why I’m going to our local doctor so much and why I’ve have had two MRI scans in the past year. Both came back negative. But our luck had to run out.

We’re thinking of you and hoping for the best and let me know if there’s anything I can do!

This Corona Thing

So how are you handling this Corona thing?

Back in January even I thought China would manage to contain it. Neither the previous outbreaks had any major impact on life as I knew it. Even in the middle of the SARS outbreak I flew to Singapore without much hesitation because I had a job to do. But when we saw them building a hospital in what can only be described as panic I realised this would get out of hand. And boy did it ever!

But it didn’t take long for our CDC (“Folkhälsomyndigheten”, or “FOHM” for short) to tell people to work from home if possible. Which I gladly did and I’ve been to the office maybe four times in as many months. And working from home definitely has its challenges that I’ll make a separate post about. But I was still hoping it’d blow over rather quickly so we could all go back to normal but “little did I know” right…

In the middle of March my wife’s grandmother fell victim to our society’s failure to protect the elderly and she passed away which was a big shock to everyone because she was quite the iron lady and we thought that after a week of not getting worse that she’d pull through but unfortunately not.

So we, as in our family unit of 3, started doing our best to minimize the exposure. Not seeing as many people, not going out on many trips or excursions and staying home as much as possible. But the cabin fever of being home this much, especially when working from home, is rather intense and I’m so thankful that FOHM hasn’t instituted a lockdown in Sweden!

But by early April everyone realised how bad this situation was and all companies that were effected by this were allowed to give their employees a paid furlough – working from 20, 40, 60 or even 80% while retaining most of the salary. I wasn’t effected by this as I’m apparently that vital to our IT ops but most of my colleagues went down to 40% or less. And it also effected my wife’s work since people aren’t going out for coffee that much and B2B sales got hurt by everyone working from home so she went from working 5 days per week to 2. Which meant that for 3 days she was home while I was working from home which made it even more interesting.

In early June we finally got tested for antibodies which was negative so as it turns out, we just had the normal flu back in February.

Unfortunately that wasn’t sustainable and eventually there had to be downsizing. I survived the first round of layoffs but my wife didn’t so she’s soon out of a job. That sucks, not only because she’s unemployed but because this was a job where she worked with a product she loves – coffee.

But so far we’re in good health. Well more or less anyway, we’re not suffering from Covid at least. But I know 6 people that have had it and 3 of them are still suffering from fatigue, shortness of breath and heart arrhythmia, scary stuff to say the least so I’d rather sit this one out.

So right now we’re still doing the best we can in little family unit, limiting our exposure, shopping as much online as possible and trying not to go out unless we have to.

The Big Tenant-to-Tenant Migration

As you may know, I worked for the Nordic part of the Thomas Cook Group. I was the O365 admin for a tenant with over 30,000 user accounts. And we ran the Azure AD Connect service for the entire group and had just moved to pass-through authentication with Seamless SSO. Although it was a royal pain sometimes to work in such a large company where even a minor change could take weeks to implement and get approval for from everywhere.

As you may also know, Thomas Cook Group filed for bankruptcy in October last year. And there was no advanced warning or anything about what was going to happen next. But for our part, we realised that we would (if the company survived) most likely be moving our Nordic business to a new O365 tenant so we began planning for that. Over the next few months a lot of stuff happened. The Nordic part of the group was sold off and started their own company NLTG and the old group was shutting down all parts of their business. Except the German part because they were backed by their government so they survived (more on that later).

When we got back after the Christmas break we were given a clear order to evacuate the tenant before end of February. Since we were a separate company and legal entity we were no longer entitled to share the old tenant which, even though it makes sense, pretty much lit a torch under our asses to get this done now. And we realised it wouldn’t be a pretty or a smooth operation, as I recall saying, “this is going to take a sledgehammer, not a scalpel!”. Fortunately I’m very used to sledgehammer my way to getting results. Yeap, thinking back to that SharePoint upgrade that was all over the place!

So there we were, less than 8 weeks to pull off a migration with 3,000 users, 5,500 mailboxes, 10TB of SharePoint data, 8TB of OneDrive data and 12TB of Exchange data. And this is how it went…

Identities : The building block of any good tenant is the identities. When we first planned for the migration our plan was to have a new on-prem AD that would be fed by.. well that’s irrelevant since there was no time for that. The only way forward was to use our existing on-prem AD. But the problem was that MSFT doesn’t support syncing your on-prem identities to two tenants. Why? I have no idea – I fully get how you wouldn’t want that in a production environment (since the UPN domain can only be valid in one tenant) but for a migration like this it would have solved a lot of headaches if we were allowed to do it like that. But nope, we really wanted to have Microsoft support for this. And we also had to retain our e-mail domains since we’re heavily dependant on the brand which is almost as Swedish as Ikea, at least in Sweden. So that presented us with the first big problem – pre-populating the new tenant with 3,000 user objects so we could start copying the data and when it was time to migrate and then play around with the UPN domains so the matching would work. But the first step was creating the 3,000 users as cloud only “onmicrosoft” accounts. This was done using powershell to export as much info on the users as possible (including “usagelocation” and “preferredlanguages” since we’re an org with offices from Thailand to Mexico!) and then powershell to recreate the users as closely as possible. Another step we had to take here was setting up a filter in Azure AD Connect that would only sync users to each tenant depending on the value of an extensionattribute. That way we could make sure no user was synced to both tenants at any time, although it did take alot of tinkering to get that logic working but fortunately Microsoft have documented how to do attribute filtering, so thanks for that.

Authentication: Remember how I said we’d just gone over to PTA for the old tenant? Well this little thing meant that as long as users were logging in to the old tenant (which we knew the Germany company would) we couldn’t use PTA for our users since it’s all based around a computer object in the AD forest with a Kerberos encryption key that’s tied to the tenant! So if we set up PTA for our new tenant that would change the key on the computer object and they wouldn’t be able to login anymore! So to solve this we did a “quick and dirty” setup of a temporary AD FS for our users to use based on domain. This was a surprisingly easy thing to do in Windows Server 2019 but it was an added “gotcha!” of this entire scenario!

SharePoint : The first problem with SharePoint was to determine which sites were relevant to keep and which weren’t. Our entire SharePoint was well over 20TB so we had to make sure to only copy over sites we knew were relevant to the Nordics business. But there’s no way of determining that without going through all the underlying permissions and groups to determine if “our” users are working on the site or not. It’s not like you can ask SharePoint to “give me all the sites that any user with the UPN domain @domain.se is working on”. Or maybe there is, I just didn’t have the knowledge to write that powershell at the time. Once that was done we used ShareGate to migrate all the SharePoint data. The biggest fear was that it wouldn’t be able to match the old identities with the new ones – which it did! I’m pretty sure it went by “DisplayName” to match them but we’re just very very thankful it did because that would have been a mess to sort out. The biggest issue I had with ShareGate was how unpredictable it was when it came to doing incremental copies, which was done through powershell. We split it up on 4 different servers with about 80 sites per server. Sometimes it could complete them all in 2 hours, sometimes it took 8 hours for one server, sometimes longer. During the weekend of the actual move it took well over 12 hours to complete which caused me a bit of unnecessary stress.

OneDrive : Since we already had a pretty nice “masterlist” of users that we would be migrating it was pretty easy to setup a CSV file to map “Old OneDrive -> New OneDrive” that we then used ShareGate to copy. That went pretty nicely although there were some instances of data not being copied over so we had to sort that our after the switch and people were missing a few files. Other than that the issue was the same as above – it was very unpredictable and I had to mess around with the queues on the weekend of the switch. We had one incident of a users OneDrive being almost empty but looking back at the old OneDrive is was empty too. So our theory there is that his OneDrive client must’ve been paused so we had to send that computer to the lab for data recovery – but that’s not ShareGates fault one bit!

Exchange : Oh joy! I was in charge of the Nordics business moving from on-prem to Online 3 years ago so I wasn’t looking forward to another move at all. After doing a quick check around for what tool to use (with our extremely limited budget – our company had gone bankrupt and we were still getting back on our feets!). It ended up being CodeTwo which was by far the cheapest alternative. But as the saying goes “you get what you paid for” and in this instance we paid for a software to move data from Mailbox A in Tenant X to Mailbox A in Tenant Y. And it did that job without much of an issue. There were still a lot of things to sort out around the move (like transport rules, conference rooms) but the big issue was just moving all the data. The biggest issue I had with that software was that they didn’t have a CSV import function when moving tenant->tenant! When moving on-prem -> tenant that wasn’t an issue, but tenant -> tenant, well the only way to enter a mailbox was to actually manually enter a mailbox! So we spent days entering 5,500 mailboxes and matching them with their new mailbox. A simply CSV import would’ve saved us days of work on this. My next issue with the software was when we were up to about 800 mailboxes per server on 7 servers, that really slowed the UI down. At the end it was so slow that when you started a queue for a incremental copy the UI would stop responding and you didn’t even know it was working until it was done and it just popped alive again.

Teams: Now Teams was the most interesting bit. Because Teams is based on so many technologies it was difficult to do a proper Teams migration. No matter how far we looked we just couldn’t find a tool that would migrate Teams with the channel/chats that also took the entire underlying SharePoint site! If you had other document libraries or data on the SharePoint site, then that was lost if you migrated the Team. But if you migrated the SharePoint site you will have lost the data in Teams that wasn’t in the default document library! So we made the choice of migrating the SharePoint sites since noone should have have posted anything business critical in a chat in a channel in Teams. Fortunately ShareGate comes with the ability to recreate O365 groups so all the groups got recreated and we only had to make the ones that were Teams into Teams manually, that was it. But it was a bit of a “unexpected behaviour” for ShareGate when it came to legacy sites (that were migrated from on-prem) that now had an O365 group, it simply wouldn’t recognise them as O365 groups or O365 Group sites and created them as legacy sites in the new tenant regardless. But that was easy enough to handle afterwards.

Licenses: This was another headache but fortunately not mine! Since our old license agreement with Microsoft was tied to our old company we couldn’t use that. And since our company was brand new we had no credit score anywhere so Microsoft couldn’t just hand us 3,000 licenses and hope we’d pay. After a lot of back and forth we managed to get the licenses in place well enough to start the migration and begin copying all that data. But there was still the matter of support contract with Microsoft. There was alot of options floating around to try different support alternatives but in the end we agreed on a premiere support deal with Microsoft. Even though the paperwork got sorted and we were told on Friday January 31st that everything was done and we now had premiere support with MSFT it turns out that like a lot of things in O365, sometimes it can take a day or two for the wheels to turn and you’ll see how critical this became for us.

Additional headache: One headache we had was that we’re not only running a normal business, we’re also running an airline. And the pilots must be able to check their e-mail for any notices and warnings from the aviation authorities before takeoff. This may include stuff like “this aircraft model isn’t flight worthy so don’t fly this aircraft model” and “Iran just shot down a civilian aircraft, avoid their airspace”. Things like that is absolutely critical for the pilots to check for, so saying “e-mail will be down for a day” is completely unacceptable from that perspective. And we were supposed to retain all the e-mail domains, and a domain can only exist in one tenant at a time. So we had to figure out a way to handle this and move their accounts and e-mail domain as quickly as possible to avoid any flight delays because their e-mails isn’t working. (spoiler – their email was down for 90 minutes)

The plan: So the best plan we came up with was to start an incremental copy of all the SharePoint/OneDrive data first thing on the morning of Saturday February 1st. Then at about 18:00 CET we’d set automatic forwarding on everyone’s mailbox in the old tenant that would forward every mail to their new mailbox with the “onmicrosoft” address. That way we were guaranteed no mail would go missing in case of bad timings. Then we did an incremental copy of all mailboxes. We had done this in plenty of tests and it would only take 2 hours so we planned to start with the first most important domain for our airline at 21:00 CET, then when that was done continue with the largest domain we had (with about 800 users) and work our way through our list of about 10 domains.
The switch consisted of alof of steps since we weren’t allowed to sync an on-prem object to two tenants.

  • The first step was to change the UPN domain of the users on-prem to newtenant.onmicrosoft.com and let that sync to the old tenant. Since that domain didn’t exist in the old tenant that resulted in the user being given a oldtenant.onmicrosoft.com UPN domain which was crucial since we knew we would end up having to restore users from recycle bin, which would be problematic if they still had their old UPN domain which no longer was in the tenant.
  • The second step was removing them from sync in the old Azure AD Connect sync and changing the extensionattribute so it would sync to the new tenant. This resulted in all users being put into the recycle bin in the old tenant, and in the new tenant it would match everyone properly as long as the UPN matched perfectly for on-prem and in the new tenant. They were then automatically converted to “synced from on-prem” users in Azure AD.
  • Thirdly we removed the domain from the old tenant and added it to the new tenant. Even though this is a straight forward process when you’ve made sure all objects for that domain are changed so the domains aren’t in use, I feared this step the most since I’ve previously had alot of issues removing a domain like this. Then ofcourse we’d have to tell that domain to be federated so it would use ADFS.
  • Lastly we would change the UPN of the user back to their original UPN on-prem and let that sync to the new tenant which now had the new domain and everything was set.
    When we did this with our test domains (of about 20 users each) this entire process took an hour so we felt pretty comfortable we’d be done at about 3-4 on Sunday morning and then we’d get some sleep before the users woke up to check their phones only to see the “error signing you in” and they’d start calling.

But… “no plan of operations extends with any certainty beyond the first contact with the main hostile force“.

How it played out: I woke up early on Saturday (at about 5) to start incremental copy of all the SharePoint / OneDrive data. Unfortunately Sharegate was a bit unpredictable in it’s behaviour so I had to move sites around in the queues to make it before 18:00 but make it I did. Then I ran the powershell to set the automatic forwarding and started the incremental copy of the mail. The team (4 engineers, 1 external SME/contractor and the project manager) met up at the office at about 20:00 in the evening for pizza and a last “go-no go” check for everything. And at 21:00 I started with our airline domain And by 22:30 it was all done, every user had the proper UPN, licens, login everything was good to go. And that’s when it started – the operations team in our airline said they couldn’t access their emails in the Outlook app on their phones or computers. We had ofcourse verified that it worked through the O365 portal so we knew everything worked. After troubleshooting this for about an hour we decided to log a Severity A case with Microsoft (at 23:30) and one of us would work on this case and the rest continue working with the other domains. That work with the other domains came to a halt for one of our largest domains which wasn’t removed from the ole tenant. No user had it in their UPN, no recipient used the domain, nothing. But the domain never got deleted, it was stuck in “pending”. So another severity A case to Microsoft (at about 00:30) and we proceeded with the next domain. At about 02 in the morning that domain did eventually go away by itself and we thought everything was good when our airline operations team (who’s responsibility it is to keep the planes flying, so I have the utmost respect for them and their challenges!) wanted us to do a rollback and try again at a later date. We spent about an hour with them arguing than a rollback wouldn’t solve this issue and we didn’t have time to try again next week since we had to evacuate the old tenant. Another argument was that this is a client issue and the mails are accessible through the web and we can get Microsoft to solve the client issue after. Fortunately we were able to convince them to proceed but now we’re at 03 int he morning and I had been working for 22 hours straight and I had no energy left in me so I tried sleeping for a bit. After 2 hours I woke up to cheers because now the Outlook clients in our airlines started to work so the biggest issue we had was solved and we could keep on with the remaining few domains.
At about lunch on Sunday morning we were all done with all domains and users and started to do the clean up job of on-prem systems no one knew about that had a EWS configured to the old tenant that no longer worked etc and that continued for days.

So where was Microsoft in this? As I mentioned our premier support deal with them got activated on the day before the switch. But that hadn’t replicated to all systems and instances of those systems in Microsoft so there was a big challenge even to get them to accept a SevA case from us. But we had two cases that managed to register as SevA cases with them during this switch and they weren’t helping us with either of them. The first case was regarding Outlook clients no longer being able to connect. Many blogs on many sites on the Internet says “when moving to a new tenant this may take a few hours”. In our case we were already up to hours and when creating new users we were able to connect to them immediately, but not the ones that had been switched and we didn’t see a reason why. This started to resolve itself after about 6 hours. And it wasn’t thanks to Microsoft doing anything on their side because they called me at about 5:30 on the sunday morning to say “sorry but we still haven’t been able to find any engineer to work on this case”. The other case we had with them was regarding the domain that wasn’t getting deleted. The called back on that issue also after it was resolved to ask us to verify the domain name because according to what they were seeing the domain was no longer in the old tenant so they obviously hadn’t done anything on their end in that case either.

Lessons learned:

  • Powershell and CSV files rule! If we didn’t have the proper master files for data information this would have been alot more difficult.
  • Switching over 1,000 mailboxes from on tenant to another actually does take up to 6 hours for all of Exchange Online to know what hit it so the clients can connect again.
  • Azure AD Connect is very powerful and “smart” in how it matches users.
  • Information and user communication and support is vital for this! In our case we started informing right away it was coming and we had staffed up extra support on the monday to get our business up and running after this big switch and that was really needed.
  • You can get away with buying cheaper “off the shelf” products rather than more expensive products but expect to have to work around their flaws and shortcomings. Do you want to pay twice the amount for a more expensive solution or sacrifice a few days work for your staff for manual work?
  • Test-test-test and test again just to be sure.

Skype For Business Online Users Disabled ?

I recently came up against an issue that I eventually needed MSFT to investigate and come up with a solution for and in the hopes of saving someone else the trouble, I’m going to go ahead and write a small blog post about it.

Symptom: The issue was that users in Skype For Business Online were stuck in the “Disabled” state and with the Directory Status “On-premises (hybrid)”. Nothing I did changed that. But other users belonging to the same domain (“contoso.com”) were enabled without issues?
After analyzing alot of the users I found that the attribute “HostingProvider” was set to “SRV:” for the users that it didn’t work for, but “sipfed.online.lync.com” for the ones that it did work for.

Root cause: The root cause for this is that when a user is provisioned in Skype For Business Online, “O365” checks in real time for a DNS record “lyncdiscover” of the domain of the user, in this case “lyncdiscover.contoso.com”. If the DNS record is set to “webdir.online.lync.com” then the user will be provisioned as an “Online” user and enabled. But if the DNS record is something else, then it will assume the user exists in an on-premise environment and it will be provisioned as an “On-premises (hybrid)” user and disabled. And in our case, sometime during the adoption of O365 services this DNS entry lost the trailing period (“.”), so for O365 it looked like “webdir.online.lync.com.contoso.com” and that’s why it assumed they were on-prem!

The Fix: Fixing the DNS entry is easy enough so that should solve it, right? Unfortunately this check happens when a user is provisioned and then it’s set. And the only way to trigger an update is to delicense the user (that is “removing Skype For Business license”), wait a few hours and then license the user again! That will trigger a provisioning process again and O365 will see the correct DNS setting and the user will be “Enabled”!

Thank you MSFT for the details regarding this process!

Checking AD FS Federation & Certificate Status

SCENARIO
You’re managing a large O365 tenant with AD FS service or multiple AD FS services and those certificates are expiring and needs replacing.

PROBLEM
The main problem is that there is no good way of telling ADFS to do something on only the domains that it actually is federated with, it’ll just assume it has them all. This may lead to some complications.

SOLUTION
I wrote this little script because I wanted to know
a) the domains that were federated to this ADFS service
b) the domains that were NOT federated to this ADFS service
c) the domains that hadn’t refreshed the signing certificate.
This little script, which must be executed on the ADFS service in an admin powershell, will first check the URL of the local ADFS service and then go through every domain in your tenant to see which match, and if they match will check the certificate. That way you know exactly which domains to look at.

It spits it all out in the console but also in 3 files in the c:\temp directory. And if you feel brave enough, you can uncomment the “update-federation” command to run that command.

Also it assumes you are already connected to the MSOL Service.

Start-Transcript c:\temp\msolfederation_check_log.txt
# Getting the local AD FS server address:
$stsaddress = ""
$stsaddress = (Get-AdfsEndpoint -AddressPath /adfs/ls/).FullUrl
$stsaddress = $stsaddress -replace "https://","" -replace "/adfs/ls/",""
write-host "The local AD FS address is $stsaddress"
$federateddomains = Get-MsolDomain | where{$_.authentication -eq "Federated"}
foreach($feddomain in $federateddomains)
{
# Clearing the variables
$certmatch = ""
$feddomainname=""
$fedinfo=""
$fedinfosts=""
# Setting the domainname of this domain
$feddomainname=$feddomain.name
if($feddomain.rootdomain)
	{
		write-host -ForegroundColor Yellow "$feddomainname is a subdomain, skipping check"
		$feddomainname >> "C:\temp\FedDomains - Subdomains.txt"
	}
else
	{
	write-host -NoNewline "Checking Federation for domain $feddomainname..."
	# Getting federation information for this domain
	$fedinfo = Get-MsolFederationProperty -domainname $feddomainname -ErrorAction SilentlyContinue
	if($fedinfo)
		{
			# Getting the STS info for this domain that can be in either two of the resulting array
			if($fedinfo.source[0] -eq "Microsoft Office 365") { $fedinfosts = $fedinfo.tokensigningcertificate[0].subject }
			if($fedinfo.source[1] -eq "Microsoft Office 365") { $fedinfosts = $fedinfo.tokensigningcertificate[1].subject }
			# Now we check if the thumbprints match
			if($fedinfo.tokensigningcertificate[0].Thumbprint -eq $fedinfo.tokensigningcertificate[1].Thumbprint) { $certmatch = "1" } else { $certmatch = "" }
			if($fedinfosts -like "*$stsaddress*")
				{
					write-host -NoNewLine " Federated to "
					write-host -NonewLine -foregroundcolor Green "this AD FS service"
					if($certmatch -eq "")
						{
							write-host -ForegroundColor Red " but certificates do not match!!!"
							# You could try to execute the below command to update the Federation information # if you feel safe in this.
							# Update-MsolFederatedDomain -DomainName $feddomainname -SupportMultipleDomain
							$feddomainname >> "C:\temp\FedDomains - ADFS Match - Cert Mismatch.txt"
						}
					else
						{
							write-host -ForegroundColor Green " and certificates do match."
							$feddomainname >> "C:\temp\FedDomains - ADFS Match - Cert Match.txt"
						}
					$feddomainname >> "C:\temp\FedDomains - domains federated to this ADFS.txt"
				}
			else
				{
					write-host -NoNewLine " Federated to "
					write-host -foregroundcolor Yellow "another AD FS instance"
					$feddomainname >> "C:\temp\FedDomains - ADFS Mismatch.txt"
				}
		}
	}
}
Stop-Transcript

I Finally Saw “Bohemian Rhapsody” – And This Is My Problem With It

I don’t consider myself a huge Queen or Freddie Mercury fan – I was unfortunately too young in the 80’s for that. But I do love their music! I don’t think there’s any other band that have more songs that I love than Queen, not even ABBA. And I love 80’s music in general. So when there was talks about a Freddie Mercury / Queen movie I loved the idea of it but I was always sceptical any actor would be able to do a good Freddie. And when Rami Malek, who is awesome in Mr. Robot, was cast I was even more sceptical. Then the movie came out with all kinds of great reviews. But it took me this long to actually take the time away from work and everything in life to go and see it.

And I liked it. I liked it a lot! I even liked Rami as Freddie, he obviously did a lot of research on the moves and Freddie’s style. And with the sunglasses it was actually a very good resemblance. And since I’m not that big a fan that I know all stories and band-moments it’s difficult to know how much of the movie is factually correct but since both Roger Taylor and Brian May were credited as producers (and yes I even spotted their cameos) I guess it can’t be too far off the truth. And I know they took a lot of liberties with the timeline so they could build up the Live Aid show as the movie’s crescendo and Freddie’s redemption, I’m all ok with that. I mean I love Braveheart even though it has more holes than Al Powell’s car at Christmas.

So what was my problem with it? Two words – “false” and “advertisement”.

In the press material for the movie and on the poster there is this image:

And that is indeed an iconic photograph of Freddie at Wembley. But it was taken at their concert in July 1986 – a concert I happen to think is one of the best concerts I’ve ever seen (even though I’ve only seen it on DVD), even including the concerts I’ve actually seen live this one takes the gold. But as I wrote – the movie’s crescendo and ending is the Live Aid concert in 1985, one year earlier. So this poster is for an event that happened after the movie’s ending and never appears in the film! That’s most definitely false advertisement.

But if you know me, if that is my only gripe with the movie, then it’s a pretty damn good movie 🙂

Check If Connected to SPOService in Script

SCENARIO
When executing SharePoint Online scripts you need to be connected to your “admin” site or the script will just fail if you’re not.

PROBLEM
When writing a script you can’t assume that you’re already connected to your SPO tenant and unlike the “msolservice” connect call you need to specify your “admin” URL which can be quite long. But sometimes you’re already connected in the Powershell session.

SOLUTION
Writing this little thing in the start of your script will check if you’re connected to the admin site and if not will call the connect-sposervice command with the URL already set.

# First we reset the sitecheck to avoid having an old result
$sitecheck=""
# This is the address of your SPO admin site
$adminurl = "https://[your tenant name]-admin.sharepoint.com"
# Now we try to get the SPOSITE info for the admin site
Try { $sitecheck = get-sposite $adminurl }
# If we get this server exception for any reason, the service isn't available and we need to take action, in this case
# write it to the console and then connect to the SPO service.
Catch [Microsoft.SharePoint.Client.ServerException]
{
Write-Host -foreground Yellow "You are not connected!"
connect-sposervice $adminurl
}

Issues changing ImmutableID with error FederatedUser.SourceAnchor

Recently ran into an issue where a user in the on-prem AD had been deleted unintentionally and in the next sync his user went along with his mailbox.
Googling around I found a helpful article how to best go about restoring this. It’s basically about creating a new on-prem users and setting the new GUID on the recovered AzureAD user so AzureAD Connect can tie them together.
However, when trying to set the new “ImmutableID” with “set-msoluser” I got this error:
Set-MsolUser : You must provide a required property: Parameter name: FederatedUser.SourceAnchor

Took alot of Googling to realise what was wrong! The issue here is that you can’t set a new ImmutableID on a user in a Federated domain! So the trick here was to change the user to an “onmicrosoft” user, change the ImmutableID and then changing it back to the federated domain!

# Checking the original ImmutableID
get-msoluser -UserPrincipalName [email protected] | select *immutableid*
# Changing it to a "onmicrosoft" UPN
set-MsolUserPrincipalName -UserPrincipalName [email protected] -NewUserPrincipalName [email protected]
# Setting a new Immutable ID from on-prem AD
set-MsolUser –UserPrincipalName [email protected] -ImmutableId "Z/-XGv2W4kWPM1mR/ddSdn!)"
# Check that the change was applied
get-msoluser -UserPrincipalName [email protected] | select *immutableid*
# Changing it back to the original UPN
set-MsolUserPrincipalName -UserPrincipalName [email protected] -NewUserPrincipalName [email protected]
# Checking that the UPN is now correct and the correct ImmutableID is applied
get-msoluser -UserPrincipalName [email protected] | select *immutableid*

Hope that saves someone some headache.

Heatwave – I wonder what might’ve caused it…

We’re having a major heatwave here in Sweden, so bad that even Pokemon Go are alerting me about it! I’ve heard it’s been the same a little all over the planet. And this reminds me of something that’s been on my mind ever since physics in college when we were discussing energy and how it can’t be created or destroyed, it can only be converted from one form to another. And this made me go around thinking how different things convert energy and it didn’t take too long to realise that so incredibly much of today’s society relies on stuff that converts energy to heat!

I mean, even if we disregard the entire greenhouse effect caused by gases, today’s society relies so much on things that create heat as a bi-product that we really can’t be too surprised when we get heatwaves like this. Nuclear power plants creates so much heat they need gazillion gallons of water to cool it, driving your car creates heat both in your car and the asphalt, using your computer creates heat, running your fridge and freezer creates heat, running the train creates heat, running your air conditioning unit to keep cool creates heat, most of the stuff we use today create heat – even your cellphone! And then there’s the fact that every single person is a walking radiator stuck at 37′ and now there are like 10 billion of us?

So yeah, I remember Trump’s tweet about how that cold day in Manhattan was proof there was no global warming – well this is most definitely proof that it’s a thing 🙂

Too Old For Concerts !!

Me and the wife celebrated 10 years a few months back. And one of the things I had planned was going to see Ed Sheeran in Stockholm since she likes his music and the timing was pretty good. So I made sure to be there when tickets went up for sale last year and got two tickets and last Saturday was the day of the concert. And the result was … we’re too old for this!!

First of all, entry was at 6:30. I made jokes “the guy probably isn’t gonna go on until 9!”. I was wrong – he went up about 8:40 so off by 20 minutes. So spending 2 hours listening to pre-show / opening stuff is “as intended” I guess? And whoever organised the event should have planned a bit better and planned for the roof to be open for the event as it’s July!! The temperature was about 35 degrees in there by the end of the night. And after the concert there was a traffic mayhem to get out of there because they had blocked off most streets which congested everything. So instead of the usual 20 minutes from MoS -> home it took us about 1 1/2 hours.
“Well, what about the show itself”? .. well I’m not a big fan of his, even though I enjoy some of his songs and lyrics. I compared it to going to the movies to see a romantic comedy – It’s wouldn’t be my first choice, but I’ll do it with the wife and I’m bound to get at least some enjoyment out of it! But this show was really different than what I expected. The guy even tried rapping a few times! And no sign of Supermarket Flowers either.

But my biggest issue was the volume. I know, I know, concerts are loud. But this was way well beyond “loud”. I already have tinnitus on my right ear thanks to me underestimating the volume at a Röyksopp / Moby concert back in 2002. So I made sure to buy earplugs before the concert to try to make sure that didn’t happen again. And my ear is still ringing!! I honestly don’t get where the enjoyment is when the music is so loud it hurts my ears even with earplugs !? Yes, you want to feel the bass in your chest but I get that from my home cinema that never gave me a ringing in my ear! It just makes no sense!! Even in between songs when he was talking it was so loud I couldn’t hear what he was saying half the time. And this is completely accepted and expected today!? I really don’t get it!!!

The one good thing I can say is I was impressed that he handled the stage alone in front of 50 000 people superbly and his way of creating musical loops with his pedals was quite funny and unexpected. And I liked it!