Checklist for Cloud Service Operations

I knew one software company that failed their SaaS transition because they chose to cut a few corners with the operations. Since they were software engineers, they did not really want to spend time on such mundane tasks as security, auditing, backups and so on. One day they let a disgruntled employee go, the guy went to an internet cafe, logged into the hosting account with the shared admin credentials, and deleted all customer data.

There were no backups or data replicas to bring the data back, no personal admin accounts or procedures to prevent such an incident from happening, and even no monitoring to learn about the issue before customers did. This was the end of this SaaS application – it just never recovered.

Agile DevOps in the Cloud - Session recording from WSO2Con Asia 2014Cloud business is more than just putting some code online (and collecting money ;)) Whether you are offering Software-as-a-Service (SaaS) web application, Platform-as-a-Service (PaaS) or Infrastructure-as-a-Service (IaaS) – what you are offering is more than just your code – it is your service.

Even if you do not offer a formal service level agreement (SLA) and have a statement in your Terms of Service that you are not liable for anything, your online application or platform is still a service so your customers expect it to be reliable and secure.

At our recent WSO2Con, Chamith Kumarage delivered an excellent session on how our Cloud DevOps team works. If you are delivering a service online (or considering doing so) – make sure to watch the recording (quick registration required).

Here’s my quick summary of Chamith’s advice:

1. Automate everything: repetitive tasks not only are inefficient and mundane, and eat your time. When done manually they are unreliable. Humans tend to do things slightly differently each time they do them, or not do them at all.

2. Tasks are really parts of processes: when you come up with something that needs to be done, ask yourself what is the process flow for this task? For example, a data backup is really a part of a process that includes:

  • Scheduled (e.g. at 1 a.m. every day) script which creates a backup,
  • Some sort of monitoring system which verifies that the script ran and the backup got created,
  • Notifications on failures and procedures that need to be followed in not,
  • Backup testing: automated and/or regular manual recovery drills (if manual then documented and performed by different team members).

3. Design for failure: everything will be failing so make sure that your system can sustain the failures. For example, if your system uses multiple virtual machines in the cloud, keep running a “chaos monkey” script which keeps randomly killing the instances and automated tests which ensure that these instance failures do not affect the overall system (by the way, see how Netflix does that.)

4. Self-healing and success verification are critical for all tasks. Any task and operation can fail (see above) so the system should not get “surprised” but should always automatically validate the action results and if something didn’t go right – implement the healing procedures (start new instances, retry, and so on).

5. Enforce discipline, processes, automation, checklists. Document everything. This will make your processes repeatable and reliable.

Bus monkey test” (related to the above) if one of your team members gets hit by a bus – all operations should keep working: everything needs to be documented and tried by other team members. (* This is a mental experiment – do not actually hit your team-members by busses :))

6. Monitoring and analytics: the key is not to collect and show tons of data and alerts, but be able to quickly detect abnormal behavior.

7. Communications: your dashboards should quickly and clearly give you the big picture and relevant details. Key metrics and system state should be something that everyone sees and understands, effective drill-downs should make it easy to understand and fix stuff.

8. Agile delivery: waterfall processes in the cloud are bad and stressful.The smaller the changes and the more often and in more automated fashion they are – the more mundane they become: which lowers the risks and improves the skills and reliability. Cloud and big-bang releases do not go well together.

9. Use standard tools and native systems of underlying platforms – do not reinvent the wheels. For example, if the platform gives you SQL-as-a-service (Amazon RDS, Azure SQL and so on) – use those and not your own MySQL running on a virtual machine.

10. Post-mortem analysis is a must. If something did get wrong after all, you need a formal investigation process:

  • What happened?
  • Why and what needs to be done to prevent this in the future?
  • If automated monitoring didn’t catch it, why and what needs to be done to prevent this in the future?
  • If validation and self-healing didn’t catch it, why and what needs to be done to prevent this?

Full session recording and slides are available here.

Categories: Powershell

Building WSO2 Cloud

This year I changed my job and became in charge of the Cloud business at WSO2. This means that now on this blog you will have even more “cloud stuff”, so I thought I would start with a quick intro to why I joined WSO2 and our general cloud directions.

Sorry if the stuff below sounds a bit salesy. :)

WSO2 started as a middleware company – with an enterprise service bus – WSO2 ESB – that is still one of the leading products in the industry.

For those who do not know, ESB is basically a system that helps connect all the discrepant IT systems that you might have – all talking different protocols, formats, etc. – so you can create your enterprise applications that use all the building blocks that you have in the company.

What made WSO2 cool (besides amazing performance and the fact that everything they produce is open source under Apache license and that all the core product discussions happen in the open, in public newsgroups) is that early on, the company designed the product to be a platform (called Carbon) which has many building blocks (for identity management, various protocols, event processing, transformations, analytics and so on) which the company then used to deliver a huge set of successful products:

  • ESB,
  • API Manager,
  • Identity Manager,
  • Business Analytics Monitor,
  • Complex Event Processing,
  • Private PaaS,
  • App Factory,
  • Enterprise Mobile Manager,
  • and many more.

All of these built from the same Carbon platform – which is very impressive.

This basically pushed the company from middleware provider to become a supplier of one of the most comprehensive platforms that the biggest enterprises are using to turn their IT platforms into Connected Business – unified system serving APIs, mobile apps, portals, applications and services connecting the company’s resources with employees, partners and customers.

Companies like Boeing, eBay, and StubHub are using this platform to run their systems – so inadvertently you have probably been a user of WSO2 technology one way or another.

Now my goal within WSO2 is to extend the reach of our technology to companies which would like to consume it as a service.

Coming from both enterprise and cloud background, I can attest that the collection of technology that WSO2 has is probably the most comprehensive cloud platform in the industry today.

At the moment, this technology gets used mostly in private cloud scenarios. My team is now actively building the public cloud side of the story. Stay tuned!

Categories: Powershell

Website scraping to PowerShell module

I have not tried this one myself, but I have to admit: this Gargl video on turning Yahoo search site into a PowerShell module looks pretty cool:

See Joe’s blog post for details and download links.

Categories: Powershell

PowerShell Tools for Visual Studio Thanksgiving Special!

Adam Driscoll's Blog - Wed, 27-Nov-13 18:52

In celebration of the Thanksgiving holiday (and because I finished a bunch of cool stuff), I’ve released a healthy update to PowerShell Tools for Visual Studio to the Gallery! There have been a lot of enhancements to the extension. I’ll highlight some of the key features in this post.

Unit Test Adapter

This particular feature I’m pretty excited about. It exposes Pester support to enable authoring of PowerShell-based unit tests that are discoverable and executable via the Test Explorer window. The adapter picks up on Describe blocks and allows you to execute them as a set or individually. When a test fails, the stacktrace and error message are presented in Test Explorer. This should also allow you to run your unit tests during CI builds in TFS. Pester support requires Pester to be installed on the module path. PoshTools does not include it at this time.



Data Tips and Watch Window 

One feature I love in Visual Studio is the ability to hover over a variable and see it’s value. The newest version of PoshTools now has this functionality! It supports arrays and hashtables (unlike some of the interim GitHub releases). It expands multiple levels; just like other managed languages.



In addition to hovering, you can specify variables in the watch window. Handy!



Dark Theme Support

The last time I promised dark theme support, it was a lie (like the cake). I had implemented it incorrectly. This version now correctly supports the dark theme. If you are still having problems with the dark theme after an upgrade, make sure to reset your colors to default for the dark theme. The colors for the dark theme aren’t as vibrant as they were (if it worked for you) but they correctly change when themes are switched. You can still update your own colors for the theme.




I’ve also added support for Read-Host. You’ll get this handy dandy input box to specify anything you desire!


As always, this release is available on both GitHub and the Visual Studio Gallery.


Categories: Powershell

Execution Policy in PowerShell Tools for Visual Studio

Adam Driscoll's Blog - Sun, 03-Nov-13 16:46

I’ve had several people report issues with PowerShell Tools for Visual Studio behaving differently than the NuGet console. After some investigation, I’ve found that NuGet actually resets the process scoped execution policy to RemoteSigned if it is not Unrestricted, RemoteSigned or Bypass.

This is taken from the NuGet source for the VSConsole.

      private static void SetupExecutionPolicy(RunspaceDispatcher runspace)
            ExecutionPolicy policy = runspace.GetEffectiveExecutionPolicy();
            if (policy != ExecutionPolicy.Unrestricted &&
                policy != ExecutionPolicy.RemoteSigned &&
                policy != ExecutionPolicy.Bypass)
                ExecutionPolicy machinePolicy = runspace.GetExecutionPolicy(ExecutionPolicyScope.MachinePolicy);
                ExecutionPolicy userPolicy = runspace.GetExecutionPolicy(ExecutionPolicyScope.UserPolicy);
                if (machinePolicy == ExecutionPolicy.Undefined && userPolicy == ExecutionPolicy.Undefined)
                    runspace.SetExecutionPolicy(ExecutionPolicy.RemoteSigned, ExecutionPolicyScope.Process);

My question is whether the community thinks this would be ideal for PowerShell Tools for Visual Studio as well. Currently, PoshTools is behaving like the ISE rather than NuGet. What do you think? Leave a comment or shoot me an email.

Categories: Powershell

PowerShell Tools for Visual Studio Beta v0.5

Adam Driscoll's Blog - Sun, 22-Sep-13 18:32

If you haven’t seen the news, PowerShell Tools for Visual Studio is now available on the Visual Studio Gallery! The first beta brings with it many enhancements from alphas past and supports VS2013 RC. Here’s a quick overview of  the current state of the extension.


The current requirements for PoshTools are PowereShell v3 and Visual Studio 2012 or 2013RC. Several people have asked for VS2010 support and I’m researching the necessary changes. Also, there is a lot of interest in supporting VS2013 Isolated Shell. If the extension could be released in this manner it would make it completely free; just like the Python Tools for Visual Studio.

Editor Extensions

The extension adds language support to the Visual Studio editor. Some of the most important features are syntax highlighting, IntelliSense and code folding. IntelliSense with help you complete cmdlets and functions, variables and parameters. Code folding can collapse script blocks and comment blocks. Overall, syntax highlighting should be very similar to the ISE but there are known issues.


In addition to the basic functionality described above, the extension also adds support for the function navigation drop down bars on the top of editors. Using the v3 AST PowerShell API, I was able to discover the functions defined within the current file and present the navigation bars seen at the top of the editor window.

Function Navigation Drop Down

A big next step for the editor is snippet support but extensions do not support this; making it a bit tricky to implement. If you have a particular function you’d like you see in the editor, head over to GitHub.

Debugger Extensions

On top of the big changes to the editor there have also been some big changes to the debugger. The goal was the surpass the ISE in terms of debugging functionality. To accomplish this, I added support for the locals, break point and call stack windows.

The locals window shows all the variables in the current scope and enables the user to expand complex objects to dig down into the object. Just like when debugging other .NET languages, type information is provided. Next on the plate for the locals window is the ability to modify values at runtime. In addition to locals window support, I’m also looking at adding Watch window support.

Locals Window

Another key window when debugging is the break point window. It displays all current pending breakpoints that are set within all the open projects. Like any other break point, the file name and line number are provided. There is no support for conditional break points but would be a good feature to add in the future.

Breakpoints Window

Another important debugging window that is supported by PoshTools is the Call Stack window. Using the Get-PSCallStack cmdlet, the extension can display the current call stack when stepping or breaking in the debugger. There is currently no support for moving call stack frames and interrogating session state.

Call Stack Window

Finally, like with PowerGUI VSX, there is a project system dedicated to PowerShell. It’s been discussed to add different types of project systems for modules or generic collections of scripts but this isn’t currently the case. The extension exposes a project system that recognizes PSM1, PSD1 and PS1 files. In the current beta there is no support for automatic signing or manifest creation. There are dialogs created for this purpose but the MSBuild tasks and targets aren’t yet hooked into the project system.


Project System

I’d love to hear feedback about the current status of the extension. If you are trying it out please leave a review on the Gallery or enter any issues on GitHub.




Categories: Powershell

I'm a Microsoft MVP... Again

It's that time of the year which feels like having a second birthday. I'm proud to announce that I was awarded the MVP title once again. This is my 6th year in a row as a PowerShell...
Categories: Powershell

Thread Slivers eBook at Amazon

Syndicate content