Category Archives: SQL Community

SQL Agent, Maintenance Plans, Auto-Heals and How to Waste an Afternoon

At my company, we’re lucky to have a custom tool that monitors EVERYTHING happening in our SQL Server environment. But every now and then, I like to explore what’s possible using only SQL Server’s built-in features-just in case you don’t have a comprehensive monitoring suite at your disposal.

Recently, someone made an off the cuff comment about using TSQL Tasks in Maintenance Plans to handle more complex logic that SQL Agent handled out of the box. I was briefly excited by the prospect of building improved logic flows directly into SQL Server Maintenance Plans. This months Andy Levy hosts TSQL Tuesday and asks us about how we handle SQL Agent Jobs and this seemed like a great opportunity for me to share a story about how I wasted an afternoon testing a few components I have a passing knowledge with, attempting to implement auto-healing and conditional logic flow, only to snap out of it a few hours later when I realized that I was trying to solve a problem that someone else cracked nearly 2 decades ago.

The visual workflow designer and conditional branching on success or failure in Maintenance Plans seemed like a great way to automate not just routine tasks, but also smart responses to issues as they arise. I love any opportunity to implement automatic healing into anything that is likely to alert me. Getting a message saying “A thing nearly broke last night – the problem is fixed” is so much better than getting a page to fix something that retrying or a known remediation action needs to be manually performed. So I looked into a little side by side comparison of SQL Agent vs Mainteance Plans to address a scenario where a DBA performs the same boring checks and actions when a backup fails, and how both technologies allowed for implementing retries and logic flow to handle this automatically rather than have a human intervene.

The flow for this process was good at first, but as I dug deeper, I realized that while Maintenance Plans can be powerful, you can achieve the same (and often better) results with a little creative thinking in SQL Server Agent Jobs. In fact, Agent Jobs offer even more flexibility, allowing you to mix and match different step types-T-SQL, PowerShell, SSIS packages, and more-within a single job. Let me walk you through what I learned, with some real-world examples.

Maintenance Plans: Visual Logic, Limited Flexibility

Let’s start with the Maintenance Plan approach. Here’s a screenshot of a workflow I put together to handle database backups with built-in remediation and notifications but the same ideas can apply to ETL or other processes.

  • The plan tries to back up all user databases.
  • If the backup succeeds, it updates a central monitoring table.
  • If the backup fails, it runs some auto-remediation steps (like clearing disk space or restarting a service), then tries the backup again.
  • If the second attempt succeeds it updates the central repository, and if it fails, it notifies the on-call operator.

This is a solid approach, and the visual editor makes it easy to follow the logic. But there are some limitations:

  • You’re mostly limited to backup, maintenance, and T-SQL tasks.
  • Complex branching or looping is awkward.
  • Integrating with external scripts or tools isn’t straightforward.

Agent Jobs: The Real Powerhouse

What’s more, my inital enthusiasm around the workflows was misplaced. SQL Agent actually has success and failure branching built in also at a per step level, it’s just in most cases the failure action is to say the job failed. But it doesn’t need to be.

With Agent Jobs, you can:

  • Chain as many steps as you want, each with its own success/failure logic.
  • Use different step types (T-SQL, PowerShell, CmdExec, SSIS, etc.).
  • Easily implement loops, retries, and advanced remediation.
  • Integrate with external systems, scripts, or APIs.

Here’s a screenshot of a way of achieving a better outcome using a non-standard workflow with SQL Agent. The job again handles backups with layered remediation and targeted notifications:

  1. Reset Autoheal Remediation Counter – Prepares the environment for a new backup attempt.
  2. Check Retry Loop – Determines if a retry is needed based on previous attempts. If the number of retries is exceeded the job generates a failure which redirects to the final step(Which is configured to alert the oncall operator)
  3. Backup User Databases – The main backup step.
  4. Update Central Reporting Repository – On successful backup the job proceeds to this step which logs the result and ends the job.
  5. Initiate Backup Drive Maintenance – If the backup fails, attempts to resolve likely issues.
  6. Perform Additional AutoResolution Tasks – Further remediation if needed. Following the completion of this step the job is redirected back to step two which increments the loop counter and retries the backup.
  7. Job Failed Notification – If all else fails, notifies the operator and ends the job.

Each step specifies exactly what should happen on success or failure, making the workflow both robust and transparent.

Why Agent Jobs Win

After experimenting with both approaches, I decided I hadn’t infact finally found a use for Maintenance Plans. Really the only thing they have going for them in this scenario is a nice graphic interface to visualise what is going on. That’s not to undersell the importance of a clearly identifiable workflow. The job step approach is very hard to follow if your workflows become more complicated. But here’s why I think Agent Jobs are the smarter choice for most advanced workflows:

  1. Multiple Step Types – You’re not limited to T-SQL or maintenance tasks. Want to run a PowerShell script, call an external API, or execute a batch file as part of your remediation? No problem.
  2. Granular Control – You can direct the job flow on a step-by-step basis, allowing for complex logic, conditional retries, and targeted notifications.
  3. Easy Integration – Agent Jobs make it simple to integrate with external monitoring systems, send custom alerts, or update centralized logging repositories.
  4. Maintainability – It’s easier to document, update, and troubleshoot jobs when each step is clearly defined and can be of any type you need.

Conclusion – So which should you use?

So obviously I’m leaving out some additioanl options in my side by side test. SQL Agent Jobs have auto retry built in at the per step level if you want, and the loop doesn’t have to be at the task or job step level – you can build that right into the TSQL if you like. I played around with both of these approaches for a while, trying to see if I could refine them to resolve the issues that I came up with. I really wanted that graphical format, but I also wanted the flexibility of the SQL Agent jobs. Which should I use?

When you delve into these things it’s really easy to get stuck with your blinkers on. The answer here was pretty clear. Microsoft actually built a whole tool for us to handle SQL data movement with complicated workflows and event handlers that has a flexible graphical presentation layer – and that tool is called Integration Services. Yes SSIS has some of it’s own issues but flexibility of logic flows is not one of them.

Another afternoon frittered away investigating a problem that was solved years ago….

SQL Saturday Wellington 2024

I wanted to make sure that everyone in the Wellington region is aware that Anu and the team are lining up another weekend of high class SQL training THIS WEEKEND. I do apologize for the late post, but I’ve been chatting to a few Wellington based clients in the past week and letting them know and realize that not everyone is aware and I need to do my bit to get the message out.

I’m going to be sharing some of the good, the bad and the ugly that we’ve seen with live moves of various clients into the Azure cloud. It’s a session that compresses masses of experience into a (probably less than) one hour session, so come along and take the opportunity to learn from other peoples mistakes.

But check out the schedule, because there is 3 tracks of data goodness, and what looks to be a fascinating keynote from Kevin McCombe and Mike Winiata from the New Zealand Fire Service.

SQL Saturday South Island 2024

Hi all. Just a reminder if you are not already signed up to get over to:

https://sqlsaturday.com/2024-07-27-sqlsaturday1084

and sign up for SQL Saturday South Island 2024 edition. I’ll be giving a talk on Azure Database migration and sharing some of the good, the bad and the ugly we’ve experienced in moving clients to (and from) Microsoft Azure.

There’s also 4 streams of data good ness with the usual suspects all making an appearance. Check out the link above for the full list. See you there!

SQL Saturday South Island – Wrap Up

What an amazing weekend.

Sometimes you can go to a free training event and take a few notes of things to go look up and that’s a win. And some days the stars align, you pick the right sessions and you walk away with solutions to problems you are working on right now.

I was lucky enough to attend Craig Ryans session on reducing your SQL spend. This is something I focus on in most of my client engagements so generally I think we have the bases covered. Not even close! Craig had a handful of different considerations that I need to work into our own offerings including comparing pricing in different regions and dusting off the cobwebs from the Web Edition of SQL Server – which to be quite honest I haven’t considered in my calculations for years. Then add in changing operating system and we saw an example of how to actieve the same levels of performance on the same hardware(or often better hardware) for half the price. Great session well worth the price of admission(which was free, but to be honest I would have paid twice that.)

My own session went okay. We went through the 5 types of Encryption that are available within SQL Server and demo’d each, making it clear that technical know-how is not a reason to not start the process of implementing Encryption in your environments. And a great thing happened after that session. I had fumbled a couple of times with the technical setup and switching between slide decks and Management Studio and one of the audience came up to me afterwards with a recommendation on using colour coding for session connections. This is something I often do when I am connecting to prod or non-prod client environments, but I think it would be a great addition to a session where I’m demo’ing in multiple different environments – both for me and the audience. So thank you anonymous random guy.

The next session I started going to was Warwick Rudds session on Azure Data Studio, but Warwick hit a technical snag when the presentation room monitor decided it had had enough and went to sleep. Warwick always presents great content, but I also know he distrubtes it really well so I promised to catch it on online and took the opportunity to grab a Lime scooter back to the hotel and drop off my laptop so I was set for a big night after the event without having to lug it around.

I got back in time for Heidi Hasting to talk about some of the processes she has automated and the tools she uses. I picked a couple of key words out of her presentation abstract and thought it may just help with an issue that a colleague is currently working on. Heidi had flown in late and had the last session of the day and presented on 2 hours sleep. But man did she deliver just what I was after with some pieces being perfect to solve a Powershell\PowerBI analytics\MS Licensing issues that have been on my periphery over the last week or two.

And as always the real value in these sessions is that if you don’t get the exact piece of information you are after in a session, you can go out afterwards, buy someone a drink and chat about the technologies we are working with and chew over the problems we are faced with in our day jobs. I literally got thousands of dollars of value out of the event and had an amazing time doing it. Thanks to Hamish and the team for another successful and enjoyable event.

Come to SQL Saturday South Island

Hi all – a quick note telling you to come along and support SQL Saturday South Island.

Once again Hamish has pulled together a great group of speakers from New Zealand, Australia and even the USA. As always it is a free day of training, and you even get a free lunch thrown in, so if not for bettering your professional career and learning lots of cool new stuff – come along and grab a free lunch – cause it turns out there IS such a thing as that.

SQL Saturday South Island 2023 (#1061)

Virtual SQL Saturday Auckland

Well obviously an in person SQL Saturday is not something we can easily do at the moment, but the virtual version that is being organised for Auckland next month is going to be really interesting.

Scheduled for 8th August 2020, we get the opportunity to have speakers from everywhere attend this local event. Because it’s virtual we can also have attendees from everywhere, so be sure to check it out.

I’m going to be doing some introduction into security in SQL Server, in particular running through different ways and reasons to implement database encryption. This stems from a session a few years back in Christchurch where one of the presenters made the throw away comment “It’s not about whether your database will be compromised, but when.” This is a little terrifying, but let’s say someone does get a hold of an old backup of a SQL Database – is that it? Is everything gone. I’m going to talk about how we can layer up defenses, so we first prevent people knowing about our SQL Servers, then do everything we can to stop them getting to them, and finally, how to make minimise the impact when it does happen.

SQL Saturday 831 – South Island

Next weekend we have another round of SQL Saturday goodness in Christchurch with SQL Saturday South Island (The 831st SQL Saturday!). Check out the full list of speakers here. Spoilers – there’s some amazing speakers coming and some diverse topics being discussed.

I will be talking about Encryption this time around. It’s an entry level session designed to make it clear to users new to encryption that the technical implementation should not be a barrier to protecting their data. We’re going to demo setting up every type of encryption SQL Server offers in this 50 minute session and spend some time talking about the thing that is way more difficult – getting everyone to agree on what should be encrypted.