Exciting news! Last week we released a beta of our free directory analysis product!
This is the first small step towards a much larger product we have coming later in September at Microsoft’s Ignite conference for helping organizations leverage their investments in SharePoint and Office 365 better. It will ensure organizations have rock solid people profile data in their directories to power experiences like profiles, Delve, search, personalized intranets, automated business process decision making and more. But more on that in the coming weeks (stay tuned to www.hyperfish.com also).
So what does this directory analysis do?
Simply put, it scans your on-prem Active Directory or online Azure Active Directory and tells you interesting facts and figures about the completeness and accuracy of your users profile data. We send you a report on what we find.
We have been testing it with our beta users and now we are ready to have others try it out.
It’s a Beta, you might hit a bug and hopefully will have feedback for us! If you do, please reply to the email you get from us with your thoughts.
SharePoint and I go way back. I have a special place in my heart for SharePoint. I got started with the product back in 2002 and it’s taken me places. It brought me to the USA when I transferred with Microsoft to join the Windows SharePoint Services engineering team in 2007 and onto the Technical Product Management team in 2009. I owe it a great deal. Its for this reason that although my new day job isn’t directly tied to SharePoint any longer that I care about its future. I lived its glory days in 2009-2012. SharePoint fever was rampant. SharePoint Conferences sold out at 12,000+ people. There was a gold rush in the enterprise collaboration space orbiting SharePoint.
I want SharePoint to rise again as a fantastic platform for enterprise collaboration.
I make no bones about the fact that I haven’t been SharePoint’s greatest proponent recently. I feel that Microsoft was chasing shiny things at the expense of what made SharePoint great to begin with. Things felt 80% done in it’s Enterprise Content Management and Collaboration capabilities. Like a theme park that was great for a while but started to age and you couldn’t help but think what could have been if only it had been updated. The SharePoint team went off and started building other things vs. focusing on updating its core features like Team Sites and document collaboration. Competitors like Dropbox showed what a better experience could be like as SkyDrive (now OneDrive for Business) fell further behind.
All this flailing about drove me nuts. I got disheartened honestly at Microsoft rewarding engineering for shipping new things that no one wanted while letting their golden goose slowly die.
There is nothing quite like a leadership change to signal to everyone that the status quo isn’t working.
Over a number of years SharePoint fell under a number of leaders who really didn’t understand the enterprise software in my humble opinion. They came from the consumer side of Microsoft. SharePoint got little attention. (PS: who lets a multibillion dollar business stagnate the wither like that anyway?!!? nuts!)
Last year Jeff Teper returned to lead the SharePoint Product Group. Many are hailing this as a turning point. The return of the king in many ways. After all … he is the father of SharePoint.
Jeff round 2…
Today we saw the first step in Jeff’s new era of SharePoint. Jeff has put the first stake in the ground for his new vision for SharePoint and where he wants to take it.
… empower people, teams and organizations to intelligently discover, share and collaborate on content from anywhere and on any device.
It focusses on four key areas:
File sharing on any device
Intranet/Team sites on any device
New extensibility options for developers
Enterprise features around security, compliance and privacy.
What I LOVE about this that it focusses on core SharePoint capabilities. It’s radically updating what is already there vs. adding something new.
Someone finally got the memo that to stay relevant you need to focus on what made you great to begin with, don’t lose sight of your fans and incrementally build from a solid base.
SharePoint lost it’s way … the real question is … is all this too late?
I don’t think so.
Many think SharePoint’s game is over. There are a multitude of smaller more nimble productivity apps out there that do a lot of what SharePoint introduced to businesses back in 2001. But the one thing many forget is Microsoft’s ability to execute end to end when the stars align just right. Building the features is just the first star to align. Company vision is another, Marketing is another, Sales is another. When all these align at Microsoft it is an unstoppable force of nature. None of SharePoint’s competitors can touch it when these things all come together end to end. But it hinges on it being a core imperative at the company to make it happen. I feel there is a very good chance of that given Satya personally pushed asked Jeff Teper to lead the SharePoint group again. He understands how critical owning File sync and Share is, how crucial owning team collaboration is and how important personal productivity is.
Here’s Satya talking about SharePoint and OneDrive at today’s event
New product experiences…
Like I already mentioned, the first star needed for perfect alignment is product. It all starts with compelling product capabilities.
Team sites vNext
Today the SharePoint team showed new product capabilities primarily in the team and document collaboration space with fresh web and mobile experiences. This in my view is playing catchup for the last few years of rot in the SharePoint UX.
Honestly … it looks great and end users who are not used to existing SharePoint sites will love it. Users already used to SharePoint will likely be confused about where the ribbon interface went and have a hard time finding things in the place they were used to.
This new UI is not particularly innovative or over and above what competitors have already, but it is super important table stakes that were desperately needed.
There is a new SharePoint mobile app. To be honest, I am pretty ho hum about this. In my experience companies give their intranet a brand. Microsoft does with its intranet, MSW (MS Web). I think SharePoint missed a great opportunity to give companies the ability to white label this app and deploy “MSW” to peoples phones, brand it with their colors and customize it with additional apps and capabilities.
Almost since the dawn of time one of the issues with SharePoint was the rampant proliferation of SharePoint sites with little to no ability to find any of them easily. SharePoint are revamping the Sites experience in O365 to show you all the sites you belong to.
Flow and Power Apps
This demands a whole post on its own!
SharePoint forms and workflows are still left wondering why they are not getting the attention they deserve.
Flow is a new tool (in preview) to help you build easy lightweight workflows that connect to different systems and tools. It actually shows quite a bit of promise in my view, but it’s built by the Azure team and you can tell they are used to building things for developers, not end users. Of course this will likely get better over time. To really succeed it needs to be deeply rooted in the Office and SharePoint user experiences, a little bit of which is seen in the screenshot below with the “Add Flow” button in SharePoint.
This is one area where either Power Apps + Flow really has to grow up fast, or SharePoint needs to take the reigns of these tools.
Ahhh … a topic near and dear to my heart. Prior to leaving MS I ran the technical marketing team in Office 365 that focused on extensibility and the partner product ecosystem around Office 365 and it is still something I deeply care about.
Microsoft has always been a partner lead company. Steve Jobs famously said the thing he admired most about Microsoft was it’s partner lead attitude. Extensibility and the ability to build on and beside products is critical to a vibrant and healthy product ecosystem.
SharePoint didn’t get to a $2.5B+ business for Microsoft by being a great out of the box application. It got there by partner companies building amazing tools, add-ins and products that enhance SharePoint with new capabilities.
SharePoint has had an ongoing identity crisis with extensibility. Each new version introducing enough of a change to piss off developers. Although on the one hand these new capabilities are great … developers also hate to have to change development processes and relearn best practices.
They are going to need to do that again this time around. The fantastic part about it is of course the ability to surface UI directly in SharePoint using modern web development tools and frameworks.
I have some pretty bias views on this. For a very long time there have been people bending SharePoint UI to do unnatural things. They inject script and insert, tweak and change things to make the SharePoint UI do what they want. They were driven to this by customer demands and consultants inability to say no when it comes to $.
This new framework for SharePoint UI gives developers a new UI API. Documented hooks and points at which extending the UI is condoned and encouraged and something the team will support. This should be the UI nirvana developers are looking for. But the proof will be in the pudding and we will see if it goes far enough with its ability to deliver what customers are demanding. There will be gaps, I’m 100% sure of that … but the key will be SharePoint’s ability to address those, driven in a collaborative manner with the community.
I have some concerns about the new framework with regards to its suitability in building commercial products. I need to do more digging before I draw conclusions, but from what I know now it is very much targeted at enterprise developers, not ISVs. The packaging and deployment model for example is different from what we have with Add-Ins today and I feel many of the benefits of the Add-In model that ISVs had, for example end user deployment and code isolation, are not present in this new framework. I can only hope that a re-focus on these in the not too distant future.
Where to from here…
I feel we are seeing the first glimpse of the stars aligning again in SharePointlandia:
I am impressed with the teams renewed focus on the core of what are SharePoint’s strengths and that feels good!
Having this line in the sand is great … they need to do this twice a year and keep the pressure on catching back up to the pack. They need to do this so that the shiny new things make sense again because the core is strong. You can’t innovate the body of a car if you leave the engine to languish (ala DeLorean).
I am bullish that Jeff understands this and has rallied the team around this as a core priority.
I am upset that they have not yet decided this is a core priority for ISVs building products with SharePoint as this is a critical component of what made them great the first time around.
I am also extremely happy to see some focus, care and attention on SharePoint again.
One of the ways AC and I track how many people are listening to the Microsoft Cloud Show episodes we put out is by using the logs created for Azure Storage. These track the various requests for the mp3 files for each episode. You have to turn on this logging for your account and once this is done then log files are written into the /$logs/blob folder of your storage account. You can read more about Azure Storage Analytics here: https://msdn.microsoft.com/en-us/library/hh343270.aspx
However, the way the storage logs are filed does not exactly make analyzing them easy in something like Excel. They are logged in /year/month/day/hour/5min chunk folders/files in CSV format. (detail of what is logged is here).
Until recently our process for downloading the analyzing these was rather laborious and included:
Download all the logs files.
Use some PowerShell to squash all the thousands of CSV files into one big one.
Use SQL Import tool to take the CSV and import each line into a temp table in SQL Azure
Run a T-SQL script over the data to remove all the rows we were not interested in (we only wanted the logs for downloaded MP3s, not other assets etc.…)
Mutate the data in various columns for better reporting. e.g. pulling important parts from the User Agent string and pulling an episode number out of the filenames.
Finally inserting all the new rows into the final reporting Table
Using Excel to report over the data
It was error prone, painfully slow at times and required all these things to be done each time we wanted new statistics. At times we would go months without doing this and were flying blind on how many people were listening. Not really very good.
What we really needed was an automated process!
Azure Function Apps to the rescue!
In March 2016 Microsoft announced Azure Functions. They are lightweight bits of code that can be run in Azure when certain events occur, like on a timed basis, when an HTTP request is made and when a blob is added to storage. What is nice about them is that you are not paying for a Virtual Machine to sit there mostly idle. You only pay for what you use. You can write them in a variety of languages and trigger them in various ways. If you know how to write a console app in C# they are super easy to understand, same with a hello world app in Node.JS.
They looked perfect for what I wanted to do. I wanted to read in log files as they were created, parse the content out of them & them log that to an Azure SQL Database. From there we just attach Excel to the DB and go to town with graphs and pivot tables etc.…
Problem!Currently it looks like you can’t trigger a Function off blob creation of a log file in the $logs container which is where the files are dropped 🙁 I don’t know if this is something that will change, but ill try and find out.
This means that for now we would have to do one of two things:
Manually copy log files to another container periodically. Ok solution.
Make another function that is triggered every 15 mins or so to copy the log files out to another container which would trigger or other function. Better solution.
I opted for copying the blobs on a schedule. I wrote some pretty crude code that:
Enumerates all the blobs in the source $logs container
Checks if they exist in the destination container
Copies them over if they don’t
I say this is crude because it doesn’t keep track of the last blob that it copied and therefore each time it runs it enumerates all the logs each time. This isn’t ideal and I will probably need to make it a bit more sophisticated in the future. But for now running it once a day shouldn’t be a problem.
Update: Since I published the post I updated the copy code to check to see if it needs to sync files since the last time the function ran + 1 extra day (to be safe). This means it wont scan all your logs every time.
So here is what we have end to end …
Copy logs from $logs in one subscription to /showlogs in another subscription
Process each log file as it arrives and put the data in the Database
Show me the code!
Ok so how did I do this. Below is the two Azure Functions. I have tried to comment them so you can follow along.
Copying log files from /$logs to another Azure Blob container.
Next, Processing those log files when they arrive.
Like most parents of young kids I have a son who is deeply into Minecraft. I also had a raspberry pi 2 sitting around and a friend mentioned that I could run a Minecraft server on it! Woohoo I thought. My son will think it’s the most amazing thing ever and I will be super dad! (Note: kids just like Minecraft, putting it on a raspberry pi is secondary to them it turns out and isn’t as cool a I thought it was). Turns out its pretty simple, drop on an OS, in this case Raspbian, and then load up a minecraft server following one of the many guides available on the internet. I got that all up running with a great sense of satisfaction and looking like superman, batman and a skylander all in one to my Son. #BadAssDad.
I’m also a bit of a space and aeronautics nut. And it got me thinking, I wonder if I can get ADS-B signals (most planes broadcast these) feeding to flightradar24.com to help them build their coverage map. The raspberry pi doesn’t have a receiver that can do this out of the box, but you can buy USB sticks that do this on amazon pretty cheaply like this one from NooElec (the one I bought).
FlightRadar24.comprovides pretty simple instructions about how to load an app onto your pi that takes signals from the USB stick and feeds them to their site. So simple i managed to follow it! Mind you I was already getting intimate with the command line, PuTTY for the Minecraft install, so things felt a bit simpler at that point.
Once you get the whole thing up and running you get log into their site and see statistics about your feed, how many aircraft you have tracked … and generally feel good about helping to crowd source this data.
Oh … and it’s all kinda nerdy which gave me a sense of satisfaction too.
Back in November 2011 I said “Adios Microsoft” and left to start the US office for Provoke Solutions. I can’t believe it has been over two years since then. Time has flown by!
Well all things come to and end and on the 10th of March I will rejoin Microsoft.
I have thoroughly enjoyed my time with Provoke. On Nov 1 2011 I literally unloaded a car full of things and set up myself in an office in Bellevue as employee #1. We have achieved some amazing things since then and I am personally very proud what we accomplished. It’s been very personally rewarding, challenging and character building. Running a small business is hard, but a lot of fun all at the same time. I was fortunate enough of having the backing from our New Zealand based team so I never felt like it was just me.
But now I am ready for my next chapter.
I am heading back to a familiar but different team from my last stint at Microsoft, the Office 365 team. I will be leading a team to help to make Office 365 a great place for developers, software vendors (ISVs), development houses (“System Integrators”) and others to customize, extend and build software for Office 365 and constituent products on-premises.
This opportunity was too good to pass up. The chance to work on the worlds #1 leading productivity software and to help take developers and ISVs along for the ride! Couple that with a unique time in computing history as the paradigm shift of cloud computing takes hold and it makes for a very unique opportunity that I couldn’t resist.
They had me at “helping developers”.
The current story for developers building for the cloud in 365 is only in its infancy and I am very much looking forward to being involved and helping as much as I can.
I know lots of people out in the community, the SharePoint community especially, who are very passionate and vocal about the issues they are facing with development for Office 365 today. I know fully well what many of the issues are, however, as always, I look forward to talking with as many people as I can and working on these issues together. It needs to be a collaborative effort and it wont happen over night. But it will happen.
I am really looking forward to the SharePoint Conference in Vegas next week!
When I left Microsoft a couple of years back I also decided to move house on the internet. I started looselytyped.net and made it my new location on the tubez (aka the internet). I didn’t technical “move” house I guess as you can still find my old blog.
In fact that is the problem!
If you search for “Chris Johnson SharePoint” my new site is on the first page of Bing, but not at the top & it’s the 2nd result on Google. However, my old blog is in the #1 spot in both search engines. Grrrr.
I didn’t want to delete my old blog, that seems like overkill and old posts still get a lot of traffic. There are 3 or 4 old posts that are knocking past 150,000 views which must mean someone finds them helpful. But given I have moved house I want people to find me at my new location. This issue has been confirmed by a couple of others who tried to find me and didn’t have good success. Classic SEO problem some would say.
Unfortunately, I cant seem to “relocate” my old content to my new blog & put in a good redirection process that would be search engine friendly. My old blog is hosted with Microsoft and there are limited controls as far as I can tell. So I think leaving it in place is my only option.
I have been making a concerted effort to try and get my new location indexed and ranking higher so people can find me. It has been working slowly.
What does HitTail do?
In a nutshell HitTail shows you what keywords (from your existing organic search traffic) you should target when writing content for your site that will increase your ranking.
Now this seems a bit counter intuitive at first, at least it did to me. Why would I write more content about things people are already coming to my site for that they found from a search?
Well, the suggestions it makes are based on the fact that people are finding your site but that you could be doing a much better job of ranking higher for those terms. So it helps guide you on what to write about that will bump you up the ranking.
So I am trying it out. I am reviewing the suggestions it makes and matching that up with topics I a) want to write about, b) have something useful to say about, and c) that I know people will find interesting.
At this stage it’s a trial to see how useful I find the suggestions, but I thought it was a handy tool to help with my internet home relocation.
If any of my readers are SEO experts I would love any other suggestions you have around my predicament. Suggestions welcome! Get in touch on Twitter @LoungeFlyZ
(the list above is not exhaustive and are really the ones I noticed and felt were interesting)
What is of real note is what is missing from SharePoint Online all together. This isn’t news to anyone who has been playing around with the preview … but for a lot of people I speak to these are not well known.
The biggest thing (IMHO) missing are around WCM features. Things like Content By Search Web Part, Catalogs, Topic Pages, Faceted navigation and Image Renditions are all pretty critical new features in SharePoint 2013 that are not currently available online. (they might be turned on later on at some point no doubt). This is a bit of a disappointment for those looking to build out search driven web sites.
For on SharePoint 2013 on premises the biggest “ouch” worth mentioning in my view is again WCM.
Pretty much all the new search driven WCM features are only included in SharePoint 2013 Enterprise. WCM Catalog, Facted Navigation, Image Renditions, Multiple Domains, Topic Pages, Content by Search etc…
Another odd thing I noticed was that Site Definitions were marked as not being included in Standard. This is weird and I suspect is wrong … but I have not checked it for myself yet.
Side Note: External Lists anomaly. For those with a keen eye might have seen that External Lists are included in the Foundation edition. Where as a lot of the other BCS related features are not. Reason: This is because of where the plumbing for External Lists sits. When we implemented this (I was the program manager who owned the external lists feature for 2010) we changed the SPList object so that you could access data in an External List the with the same SPList object model calls as you would with a regular list. Because this is a fairly deeply rooted object in the OM it’s part of the Foundation edition … and therefore in the 2010 release of SharePoint a lot of the BCS engine and core plumbing was moved to SharePoint Foundation from SharePoint Server to enable this SPList OM over external lists scenario. Bit of background for those still reading 🙂
PS: the fact that MS have published this means General Avalability and announcements around SharePoint Online upgrades should be coming soon … hopefully
Just pre-ordered a new Surface and really looking forward to getting it in a week or so.
I am really trying to stay open minded about this device. As you may know I am an MS kool-aid drinker for WAY back … but I have spent plenty of time with other devices too. I used to have an iPhone 3G, 3GS and a 4. I have an iPad (v1) currently that is my couch surfing device of choice.
A lot of people are asking my why get the RT version (ARM) over the Pro (x86) in 3 months time.
I want to replace my iPad. I am not looking for a laptop replacement right now. That is why I don’t care about running existing Windows desktop apps etc…
On top of that here are a few things why I think I will like the Surface better than my iPad:
Physical keyboard built into the screen cover. I’m only one flip away from productively typing a blog post like this vs. feeling like a nonce trying to type it on the crappy on screen keyboard.
USB – plug in full size keyboard, mouse, printer etc…
Multi-Tasking apps, snapped view of twitter while doing something else.
But who really knows right?!?! The proof will really be in the pudding and my intention is to update this post with my thoughts and views after having used it for a while.
Also ill post my “Will it blend” or whatever other iPad disintegration video I come up with.