Chris is an avid developer, speaker and is the General Manager of Provoke Solutions Inc. a Microsoft Gold Partner in Seattle WA that is one of the world's most renowned and sought-after online experience consultancies. Provoke Solutions specialize in software solutions for SharePoint and the Microsoft technology stack (http://www.provokesolutions.com). In Nov 2011 Chris left Microsoft Corporation after nine and a half years where he most recently was a Senior Technical Product Manager for the SharePoint product group in Redmond Washington managing SharePoint’s professional developer audience technical marketing programs. Chris moved to Redmond in 2007 to work in the software engineering team on the SharePoint 2010 release after working for Microsoft New Zealand. In New Zealand he consulted to customers across the Asia Pacific region on designing and implementing Content Management Server and SharePoint deployments. Chris’ background is in Microsoft software development and enjoys all things technical. He is a speaker at numerous conferences around the world such as Tech.Ed, SharePoint Best Practices Conference, SharePoint Connections and the world wide SharePoint Conference. Chris holds a Bachelor of Computer Science & enjoys throwing himself out of perfectly good airplanes from time to time.
Chris blogs and can be contacted via www.looselytyped.net
When you use a flight tracking app on your phone to see where a flight is it’s very possible that location data has been crowd sourced. Pretty cool!
Sites like FlightAware.com and FlightRadar24.com use feeds of data from people around the world to help build their datasets. Participating in those feeds is open to anyone who has some basic equipment. This works by listening to the ADS-B and Mode-S signals transmitted by aircraft. These signals identify aircraft and in some cases include positional data. It’s very easy to listen for these signals using a 1090Mhz antenna and an ADS-B receiver. A couple of years ago I bought some equipment on Amazon, hooked up the software running on a Raspberry Pi and got started feeding it to FlightAware.com.
However, recently I stepped things up a notch with a new better antenna and dockerizing my setup. More on that in a moment, but first …
Getting started – for those who want to try this themselves
If you want to try this out yourself you will need some basic equipment:
You should be up and feeding the network in an hour or two:
The “good” with the PiAware guide is it’s the simplest build process, the “bad” is that it’s specific to flight aware and doesn’t set you up to feed other providers.
“Need more input!” – Short Circuit (1986)
Eventually you might find yourself wanting more “range”. The small indoor antenna might let you track aircraft 30mi/50kms away, depending on the terrain around you, line of sight and trees etc…
When you “need more input” you will need a better antenna. This may possibly require some WAF (“wife acceptance factor”) (or HAF, husband acceptance factor) … as it will likely require putting something outdoors and, for best results, on your roof.
My situation called for mounting it externally on the roof along a gutter line. Ideally I would have mounted it on a peak of the roof, but I didn’t feel comfortable drilling holes in my roof, so opted for a mount that let me hang it from under the eaves.
The new outdoor setup and better antenna really bumped up my coverage. Even without optimal mounting (as you can see there are trees on the south side of our house) range went from < 50mi to ~150-200mi in some directions.
Dockerizing all the things
I like Docker containers. They make my life simple for running different apps and services on one box and it seemed to make sense to me that you should be able to run the piaware software and dump1090 software in containers instead of on the Raspberry Pi directly.
I came across an article “Get eyes in the sky with your Raspberry Pi” by Alex Ellis who had done just that! In Alex’s setup however the configuration of the containers is baked into the docker images at build time which isn’t ideal. I made some improvements like moving all configuration to Environment variables and added Docker-Compose support.
With everything in Docker containers it was relatively simple to add a feeder to another tracking site FlightRadar24.com. They also provide software “fr24feed” that takes a feed from dump1090 and processes/uploads it. You can find optional instructions in the ReadMe file on how to add this pretty simply.
I love this stuff. It’s a fun project with hardware and software aspects to it. Hanging out of a second story window being held by my wife around the waist was a “hilarious” exercise that I suggest every marriage attempts at some point.
I still have some re-wiring to do in the attic to secure the wiring a bit more, and possibly add some more feeders to feed other tracking sites before I’m complete with the project too.
Going forward I’m not sure what is next for this project yet. I’m sure there is more to be done and that I’ll likely be mounting more hardware the roof at some point! LOL.
I hope you can enjoy the frivolity of a project like this as much as I do!
The issue with just firing up Fiddler and watching traffic is that to decrypt HTTPS traffic fiddler installs a root certificate so that SSL is terminated in Fiddler first so that it can show you the decrypted payloads back and forth etc…
That is normally fine with apps that use the standard WinINET libraries etc… to make HTTPS calls (like chrome). However, Azure Storage Explorer an Electron app using NodeJS and doesnt use these. Node also handles root CAs a bit differently and a long story short is that it doesn’t by default trust Fiddlers Root Cert that it installs. This means that HTTPS calls fail with a “unable to verify the first certificate” error.
Setting up Fiddler
First you need to set up Fiddler to decypt HTTPS traffic. You do this in Fiddlers options under Tools > Options > HTTPS.
This will prompt you to install a certificate that Fiddler uses to terminate SSL in Fiddler so it can show you the decrypted traffic.
One You have completed this you need to export the certificate Fiddler installed so that you can set up Storage Explorer with it.
File > Add Remove snap in
Pick Certificates, when prompted choose “Computer account” and “Local computer”
Navigate to Certificates > Trusted Root Certificates > Certificates
Find “DO_NOT_TRUST_FiddlerRoot” certificate
Right Click > All Tasks > Export
As you go through the wizard choose “Base-64 encoded X.509 (.CER)” for the file format
Save it your desktop or somewhere you will be able to find it later
Setting up Azure Storage Explorer
First up you need to configure Azure Storage Explorer to use Fiddler as a proxy. This is pretty straightforward.
In Storage Explorer go to the Edit -> Configure Proxy menu and add 127.0.0.1 and 8888 (fiddler defaults). Note: Not authentication should be used.
Now Storage Explorer will use Fiddler … however, you will start getting “unable to verify the first certificate” errors as Storage Explorer still doesnt trust the root certificate that fiddler is using for SSL termination.
To add the Fiddler certificate go to the Edit > SSL Certificates > Import Certificates. Pick the .cer file you saved out earlier. Storage Explorer will prompt you to restart in order for these to take effect.
Now when you start using Storage Explorer you should start seeing its traffic in Fiddler and in a readable decrypted state like below.
Now you can navigate around and do various operations and see what and how Azure Storage Explorer is doing it.
Call me weird … but i like clear ice in my drinks for some odd reason. So I went on a mission to learn how to make it at home.
Note: This is amost totally pointless. It tastes the same, is just as cold (i think)… but it looks sweet!
Here is how I do it:
-= Making the Ice =- 1. Boil lots of water
2. Let it cool to room temperature (dont skip this step)
3. Boil it again
4. Let it cool to room temperature (yes, again)
5. Fill a small cooler with the water. Leave about 10cm (4″) of room at the top
6. Take the lid off the cooler or leave the lid open
7. Put the cooler in your freezer
8. Leave it for 24 hours
9. Take the cooler out of the freezer and it will be frozen on the top 8cm (3″) or so
10. Slide a knife down the side of the block and ease it around the block. This will let in air and release the block
11. Take the block out carefully and use a knife to remove sharp bits
-= Cutting it/Shaping it=-
Note: I have not mastered this bit yet
1. get a saw or knife and score the block where you want to cut it.
2. Place knife on score line and knock/hit it with a rubber mallet
3. Hopefully it will break cleanly
4. repeat until you have the rough blocks you want
5. Use a hot fry pan to shape the blocks and make perfect sides of the block
I recently sat down with Richard Campbell on the RunAsRadio podcast to talk about the state of directories, why people profile data is a critical component of SharePoint and Office 365 deployments and how Hyperfish can help organizations with their profile and directory mess.
Today Microsoft announced the much anticipated competitor to Slack called Microsoft Teams.
At the core Teams is all about lightweight team to collaboration spaces that include in persistent chat, docs, notes, instant message, video calls and more. It really brings together many parts of Office 365 into one experience and is simple and easy for users.
Why is this so important for Office 365? Well, in short, Microsoft has had all these pieces of technology for many years but has not brought them together in a cohesive application experience for users …. Until now.
I’m excited about Microsoft Teams! For some time many have marveled at Slacks simplicity and therefore friction free collaborative experience, but winced at having to manage, purchase and integrate yet another application. Teams is baked into Office 365 and it comes for free. Is it as good as Slack? We don’t know yet. It’s only in preview currently and only time will tell. But it’s off to a good start.
Reliance on people profile data is more important than ever
Virtually all the new experiences going into Office 365 have people at the center of them. Microsoft Teams is no different.
People are at the center of Microsoft Teams. However, to get the best experience in Teams, and to make it as compelling and interesting as the demos suggested, you must have great people profile information.
Take for example the org chart they showed in the demo. Looks great and gives you all the information you need to know when looking where in the organization someone sits. It’s also interactive and you can navigate around etc…
But 75%+ of companies don’t have the org structure data needed in Office 365 to power this feature!
Or take the photo thumbnails that show everywhere to help you connect with the person you are IM’ing with and make it more personal.
Those photos come from Azure AD and Office 365. If you don’t have your users profile photos in AD then chances are you will be getting a subpar experience in Teams.
Bots and automation
WhoBot was another example of how people profile information is powering new experiences. WhoBot is a chat bot that lets you look up people in the organization based on their name or skills for example. Again, all this is driven from people profile information in Office 365.
So what if you don’t have great profile data?
Most organizations don’t have great data in Active Directory, Azure Active Directory or Office 365 profiles. So don’t worry, you are not alone 🙂
This is why we created Hyperfish to help organizations understand their profile data and then complete or fix the missing or incorrect information.
Our mission is to make experiences like Microsoft Teams, Delve, people search experiences everything they should be with complete and accurate profile data.
If you don’t know what how good you profile data is you can try our free analysis app! It support Active Directory on-premises as well as Azure Active Directory / Office 365.
If you want to fix your data you can have Hyperfish reach out to users through email or IM to gather it from the folks missing data. e.g. missing photo photos, incorrectly formatted phone numbers, non-standard addresses, non-standard job titles etc… Learn more here about the full Hyperfish product: https://hyperfish.com/
At Hyperfish we are pumped about Teams! The preview looks great and we cant wait to see what else comes as part of it before it is due for release in Q1 of 2017.
We hope you enjoy these new capabilities and get in touch if you need a hand with keeping your people profile data clean and up to date!
At Ignite 2016 I did a quick 20 min theatre session in the expo hall on driving better adoption with your Office 365 rollout by leveraging people centric features. A lot of organizations dont make it part of their plan to ensure they have the right people profile data set up to make the best use of the ever expanding range of experiences in Office 365 that rely on it.
They are letting their users down, IT down and the business down.
Successful adoption means focusing on users
In Office 365 most of the profile data comes from Azure Active Directory. Sadly, it’s all too common to see it poorly populated.
Missing this data impacts your Office 365 deployment. Some are simple yet important and others are business criticial.
Here are the Top 10 things I felt are the most common things people are missing out on due to poor profile data that drive better user adoption and use of these tools.
#1 – Delve People Profiles
Microsoft is investing in making the Delve people profile pages THE definative profile page in Office 365 for users. Search etc… will use these. Without great data these look bland and dont provide users the information they need to find and connect with people quickly.
Delve without good data
Delve with good profile data
#2 – Contact cards everywhere
Contact cards pop up in all sorts of places e.g. in Outlook (clicked 10s of millions of times a month no less!) Without great data they make it hard to connect and discover people.
Populated contact card
#3 – New SharePoint experiences
SharePoint is in the middle of a visual makeover in Office 365. More people data drive experiences like the new rich contact panels will be popping up all over the place.
New SharePoint contact popups
#4 – Mobile SharePoint Intranet
Along with better desktop web experiences, SharePoint have released new mobile apps to help you find and connect with people while you are on the move. Great for finding someone’s phone number while out and about.
Mobile people profile
#5 – Office apps
There are plenty of places in Office applciations like Word and Excel where people information pops up. For example while co-authoring a document with someone else. Good profile data makes it a much nicer experience for seeing who is in the document.
Co-Authoring in Word
#6 – Dynamic Distribution Lists in Exchange
Not many people know, but you can create dyanmic distribution lists that include people that have specific words in their profiles. Great way to ensure that all Sales people are in the right DLs!
Dynamic DLs in Exchange online.
#7 – Workflows
One of the most common people want to do in workflows is excalate a task to someone’s manager if they out of the office or take too long to respond. Without good organization heirachy data in profile this isnt possible! This means people have to write more code to look it up from somewhere else e.g. HR system, or replicate the data somewhere else (which will become stale the day after).
Nintex workflow in Office 365
#8 – People Search
Here is a test. Go to your search portal in Office 365, flip to the people search tab and type “Sales”. Did you get great results? Probably not. People search is driven by people profile data and without it search is hard to use and find the right people at the right time.
Rich people search results in Office 365
#9 – Groups
Groups are slowly becoming the new team sites for projects etc… They are rich with people data and conversations. Make sure they sing with decent profiles.
Office 365 Groups
#10 – Skype for Business
Skype for Business uses people profile data for its Search, dialing and in calls. If you dont have good profile data you cant find the right people in search, you cant dial people if they have poorly formatted phone numbers and you dont get to see who you are talking to without nice profile pictures. Help remote users connect with one another by ensuring their experience rocks.
Skype for Business
These are just a few of the experiences I think most people are missing out on in Office 365 due to poor profile data. I founded Hyperfish to help people whip their people profiles into shape and start making these experiences rock for users thus driving satisfaction, adoption and reuse of the tools you have already bought.
I hope you are getting the most out of your Office 365 investment and not letting something like profile data get in the way of users loving it.
One of the most interesting announcements, at least in my mind, this week from Ignite 2016 was the news that Microsoft is adding support for Microsoft Graph API in hybrid deployments.
This means you can call the Microsoft Graph API like you would normally for Office 365 based mailboxes, for example, but have it actually connect with a mailbox that resides in an on-prem exchange server!
Let that sink in for a moment. That is pretty sweet!
Typically, if you have an application that is not deployed behind the firewall, i.e. inside the orgnaizations network, you couldnt call things like SharePoint or Exchange APIs without doing network gymnastics like VPN, reverse proxy or putting holes in your firewall (yuk).
Now with this hybrid support in the graph you can simply call internet based REST APIs and Microsoft is doing the work of facilitating that communication back to the on-prem resource.
Currently in Preview the graph only supports Mail, Calendar and Contacts in this hybrid configuration right now, however I can only imagine that more support for other endpoints like Users, SharePoint etc… will come at some point.
You also have to have Exchange 2016 CU3 servers deployed on-prem to get this support and sync your AD to Azure AD as authentication is managed this way.
You can read more about these re-reqs here: http://graph.microsoft.io/en-us/docs/overview/hybrid_rest_support
I think this is a huge benefit for those who are looking to build applications or cloud services that connect to data wherever that sits.
Couple of interesting scenarios to think about:
Mobile app that works outside the firewall was either not possible or too hard due to connectivity issues.
Cloud service web apps that you want to connect to on-prem data
Tools and Apps that can now work with data either in Office 365 or on-prem
This to me is a huge step in the right direction for Microsoft in their quest to make developers lives better in a hybrid world.
Exciting news! Last week we released a beta of our free directory analysis product!
This is the first small step towards a much larger product we have coming later in September at Microsoft’s Ignite conference for helping organizations leverage their investments in SharePoint and Office 365 better. It will ensure organizations have rock solid people profile data in their directories to power experiences like profiles, Delve, search, personalized intranets, automated business process decision making and more. But more on that in the coming weeks (stay tuned to www.hyperfish.com also).
So what does this directory analysis do?
Simply put, it scans your on-prem Active Directory or online Azure Active Directory and tells you interesting facts and figures about the completeness and accuracy of your users profile data. We send you a report on what we find.
We have been testing it with our beta users and now we are ready to have others try it out.
It’s a Beta, you might hit a bug and hopefully will have feedback for us! If you do, please reply to the email you get from us with your thoughts.
SharePoint and I go way back. I have a special place in my heart for SharePoint. I got started with the product back in 2002 and it’s taken me places. It brought me to the USA when I transferred with Microsoft to join the Windows SharePoint Services engineering team in 2007 and onto the Technical Product Management team in 2009. I owe it a great deal. Its for this reason that although my new day job isn’t directly tied to SharePoint any longer that I care about its future. I lived its glory days in 2009-2012. SharePoint fever was rampant. SharePoint Conferences sold out at 12,000+ people. There was a gold rush in the enterprise collaboration space orbiting SharePoint.
I want SharePoint to rise again as a fantastic platform for enterprise collaboration.
I make no bones about the fact that I haven’t been SharePoint’s greatest proponent recently. I feel that Microsoft was chasing shiny things at the expense of what made SharePoint great to begin with. Things felt 80% done in it’s Enterprise Content Management and Collaboration capabilities. Like a theme park that was great for a while but started to age and you couldn’t help but think what could have been if only it had been updated. The SharePoint team went off and started building other things vs. focusing on updating its core features like Team Sites and document collaboration. Competitors like Dropbox showed what a better experience could be like as SkyDrive (now OneDrive for Business) fell further behind.
All this flailing about drove me nuts. I got disheartened honestly at Microsoft rewarding engineering for shipping new things that no one wanted while letting their golden goose slowly die.
There is nothing quite like a leadership change to signal to everyone that the status quo isn’t working.
Over a number of years SharePoint fell under a number of leaders who really didn’t understand the enterprise software in my humble opinion. They came from the consumer side of Microsoft. SharePoint got little attention. (PS: who lets a multibillion dollar business stagnate the wither like that anyway?!!? nuts!)
Last year Jeff Teper returned to lead the SharePoint Product Group. Many are hailing this as a turning point. The return of the king in many ways. After all … he is the father of SharePoint.
Jeff round 2…
Today we saw the first step in Jeff’s new era of SharePoint. Jeff has put the first stake in the ground for his new vision for SharePoint and where he wants to take it.
… empower people, teams and organizations to intelligently discover, share and collaborate on content from anywhere and on any device.
It focusses on four key areas:
File sharing on any device
Intranet/Team sites on any device
New extensibility options for developers
Enterprise features around security, compliance and privacy.
What I LOVE about this that it focusses on core SharePoint capabilities. It’s radically updating what is already there vs. adding something new.
Someone finally got the memo that to stay relevant you need to focus on what made you great to begin with, don’t lose sight of your fans and incrementally build from a solid base.
SharePoint lost it’s way … the real question is … is all this too late?
I don’t think so.
Many think SharePoint’s game is over. There are a multitude of smaller more nimble productivity apps out there that do a lot of what SharePoint introduced to businesses back in 2001. But the one thing many forget is Microsoft’s ability to execute end to end when the stars align just right. Building the features is just the first star to align. Company vision is another, Marketing is another, Sales is another. When all these align at Microsoft it is an unstoppable force of nature. None of SharePoint’s competitors can touch it when these things all come together end to end. But it hinges on it being a core imperative at the company to make it happen. I feel there is a very good chance of that given Satya personally pushed asked Jeff Teper to lead the SharePoint group again. He understands how critical owning File sync and Share is, how crucial owning team collaboration is and how important personal productivity is.
Here’s Satya talking about SharePoint and OneDrive at today’s event
New product experiences…
Like I already mentioned, the first star needed for perfect alignment is product. It all starts with compelling product capabilities.
Team sites vNext
Today the SharePoint team showed new product capabilities primarily in the team and document collaboration space with fresh web and mobile experiences. This in my view is playing catchup for the last few years of rot in the SharePoint UX.
Honestly … it looks great and end users who are not used to existing SharePoint sites will love it. Users already used to SharePoint will likely be confused about where the ribbon interface went and have a hard time finding things in the place they were used to.
This new UI is not particularly innovative or over and above what competitors have already, but it is super important table stakes that were desperately needed.
There is a new SharePoint mobile app. To be honest, I am pretty ho hum about this. In my experience companies give their intranet a brand. Microsoft does with its intranet, MSW (MS Web). I think SharePoint missed a great opportunity to give companies the ability to white label this app and deploy “MSW” to peoples phones, brand it with their colors and customize it with additional apps and capabilities.
Almost since the dawn of time one of the issues with SharePoint was the rampant proliferation of SharePoint sites with little to no ability to find any of them easily. SharePoint are revamping the Sites experience in O365 to show you all the sites you belong to.
Flow and Power Apps
This demands a whole post on its own!
SharePoint forms and workflows are still left wondering why they are not getting the attention they deserve.
Flow is a new tool (in preview) to help you build easy lightweight workflows that connect to different systems and tools. It actually shows quite a bit of promise in my view, but it’s built by the Azure team and you can tell they are used to building things for developers, not end users. Of course this will likely get better over time. To really succeed it needs to be deeply rooted in the Office and SharePoint user experiences, a little bit of which is seen in the screenshot below with the “Add Flow” button in SharePoint.
This is one area where either Power Apps + Flow really has to grow up fast, or SharePoint needs to take the reigns of these tools.
Ahhh … a topic near and dear to my heart. Prior to leaving MS I ran the technical marketing team in Office 365 that focused on extensibility and the partner product ecosystem around Office 365 and it is still something I deeply care about.
Microsoft has always been a partner lead company. Steve Jobs famously said the thing he admired most about Microsoft was it’s partner lead attitude. Extensibility and the ability to build on and beside products is critical to a vibrant and healthy product ecosystem.
SharePoint didn’t get to a $2.5B+ business for Microsoft by being a great out of the box application. It got there by partner companies building amazing tools, add-ins and products that enhance SharePoint with new capabilities.
SharePoint has had an ongoing identity crisis with extensibility. Each new version introducing enough of a change to piss off developers. Although on the one hand these new capabilities are great … developers also hate to have to change development processes and relearn best practices.
They are going to need to do that again this time around. The fantastic part about it is of course the ability to surface UI directly in SharePoint using modern web development tools and frameworks.
I have some pretty bias views on this. For a very long time there have been people bending SharePoint UI to do unnatural things. They inject script and insert, tweak and change things to make the SharePoint UI do what they want. They were driven to this by customer demands and consultants inability to say no when it comes to $.
This new framework for SharePoint UI gives developers a new UI API. Documented hooks and points at which extending the UI is condoned and encouraged and something the team will support. This should be the UI nirvana developers are looking for. But the proof will be in the pudding and we will see if it goes far enough with its ability to deliver what customers are demanding. There will be gaps, I’m 100% sure of that … but the key will be SharePoint’s ability to address those, driven in a collaborative manner with the community.
I have some concerns about the new framework with regards to its suitability in building commercial products. I need to do more digging before I draw conclusions, but from what I know now it is very much targeted at enterprise developers, not ISVs. The packaging and deployment model for example is different from what we have with Add-Ins today and I feel many of the benefits of the Add-In model that ISVs had, for example end user deployment and code isolation, are not present in this new framework. I can only hope that a re-focus on these in the not too distant future.
Where to from here…
I feel we are seeing the first glimpse of the stars aligning again in SharePointlandia:
I am impressed with the teams renewed focus on the core of what are SharePoint’s strengths and that feels good!
Having this line in the sand is great … they need to do this twice a year and keep the pressure on catching back up to the pack. They need to do this so that the shiny new things make sense again because the core is strong. You can’t innovate the body of a car if you leave the engine to languish (ala DeLorean).
I am bullish that Jeff understands this and has rallied the team around this as a core priority.
I am upset that they have not yet decided this is a core priority for ISVs building products with SharePoint as this is a critical component of what made them great the first time around.
I am also extremely happy to see some focus, care and attention on SharePoint again.
One of the ways AC and I track how many people are listening to the Microsoft Cloud Show episodes we put out is by using the logs created for Azure Storage. These track the various requests for the mp3 files for each episode. You have to turn on this logging for your account and once this is done then log files are written into the /$logs/blob folder of your storage account. You can read more about Azure Storage Analytics here: https://msdn.microsoft.com/en-us/library/hh343270.aspx
However, the way the storage logs are filed does not exactly make analyzing them easy in something like Excel. They are logged in /year/month/day/hour/5min chunk folders/files in CSV format. (detail of what is logged is here).
Until recently our process for downloading the analyzing these was rather laborious and included:
Download all the logs files.
Use some PowerShell to squash all the thousands of CSV files into one big one.
Use SQL Import tool to take the CSV and import each line into a temp table in SQL Azure
Run a T-SQL script over the data to remove all the rows we were not interested in (we only wanted the logs for downloaded MP3s, not other assets etc.…)
Mutate the data in various columns for better reporting. e.g. pulling important parts from the User Agent string and pulling an episode number out of the filenames.
Finally inserting all the new rows into the final reporting Table
Using Excel to report over the data
It was error prone, painfully slow at times and required all these things to be done each time we wanted new statistics. At times we would go months without doing this and were flying blind on how many people were listening. Not really very good.
What we really needed was an automated process!
Azure Function Apps to the rescue!
In March 2016 Microsoft announced Azure Functions. They are lightweight bits of code that can be run in Azure when certain events occur, like on a timed basis, when an HTTP request is made and when a blob is added to storage. What is nice about them is that you are not paying for a Virtual Machine to sit there mostly idle. You only pay for what you use. You can write them in a variety of languages and trigger them in various ways. If you know how to write a console app in C# they are super easy to understand, same with a hello world app in Node.JS.
They looked perfect for what I wanted to do. I wanted to read in log files as they were created, parse the content out of them & them log that to an Azure SQL Database. From there we just attach Excel to the DB and go to town with graphs and pivot tables etc.…
Problem!Currently it looks like you can’t trigger a Function off blob creation of a log file in the $logs container which is where the files are dropped 🙁 I don’t know if this is something that will change, but ill try and find out.
This means that for now we would have to do one of two things:
Manually copy log files to another container periodically. Ok solution.
Make another function that is triggered every 15 mins or so to copy the log files out to another container which would trigger or other function. Better solution.
I opted for copying the blobs on a schedule. I wrote some pretty crude code that:
Enumerates all the blobs in the source $logs container
Checks if they exist in the destination container
Copies them over if they don’t
I say this is crude because it doesn’t keep track of the last blob that it copied and therefore each time it runs it enumerates all the logs each time. This isn’t ideal and I will probably need to make it a bit more sophisticated in the future. But for now running it once a day shouldn’t be a problem.
Update: Since I published the post I updated the copy code to check to see if it needs to sync files since the last time the function ran + 1 extra day (to be safe). This means it wont scan all your logs every time.
So here is what we have end to end …
Copy logs from $logs in one subscription to /showlogs in another subscription
Process each log file as it arrives and put the data in the Database
Show me the code!
Ok so how did I do this. Below is the two Azure Functions. I have tried to comment them so you can follow along.
Copying log files from /$logs to another Azure Blob container.
Next, Processing those log files when they arrive.