A Better Kubernetes Job Scheduler

Kubernetes Jobs are ideal for applications that are intended to perform some task and then shut down, as opposed to service type applications that run continuously. Examples include batch processes in data pipelines and machine learning tasks like model training and batch inference. A Kubernetes Job is really a controller that creates one or more Pods to perform a specific task. It ensures that n Pods successfully terminate, after which the Job and any Pods it created are deleted.

Why not CronJobs?

Kubernetes CronJobs can be used to create Jobs on a repeating cron type schedule. CronJobs may work for simple use cases but are not ideal for many scenarios due to limited features and idiosyncrasies including:

• In some cases a scheduled CronJob might create multiple or no Jobs.
• The timezone used to determine scheduled execution times is based solely on the timezone of the kube-controller-manager.
• Advanced scheduling features including start/end dates, jitter and interval based and single execution schedules are not supported.
• Parallelization options for Job instances are limited as is controlling what happens when scheduled Jobs fail to execute at the scheduled time.
• Multiple Jobs cannot be chained together in a workflow.

Scheduling Kubernetes Jobs with SaaSGlue

SaaSGlue simplifies scheduling and running containerized applications as Kubernetes Jobs. After setting up a SaaSGlue account and installing the SaaSGlue Helm chart, scheduling a Job requires a few mouse clicks and entering the docker image name and other Job parameters in the SaaSGlue console.

This article demonstrates this process using a sample application. To run the sample application you’ll need a SaaSGlue account. If you don’t have one already, sign up for a free account with your email address or your Google or GitHub credentials.

Next follow the instructions in the kubernetes-job-runner GitHub repo to configure SaaSGlue to run Kubernetes Jobs and then to run two simple python scripts as Kubernetes Jobs and view the running output in the SaaSGlue console.

The instructions show how to import the “Kubernetes Job Runner” job to your SaaSGlue account. After importing the Job, you can view and modify it in the SaaSGlue console.

This job has a single task “Run Job” which creates a Kubernetes Job via the Kubernetes API. It has two schedules which run the two applications every minute.

By clicking the “edit” link, you can view and modify the schedule’s properties including the “Runtime Variables” property which is a list of key value pairs used by the running job. The “Kubernetes Job Runner” job utilizes a yaml formatted runtime variable named FunctionKwargs to specify the docker image to use and other Kubernetes Jobs parameters. This design enables reuse of this job to schedule any number of Kubernetes Jobs. See the GitHub repo readme for a detailed description of the available parameters.

FunctionKwargs variable value:

How it works

SaaSGlue is a cloud/agent code scheduling and execution platform. Jobs are created using the SaaSGlue web console or the API and executed by agents deployed to the execution environment. You can easily install the SaaSGlue agent in a Kubernetes cluster using the SaaSGlue helm repo.

When a schedule is triggered, the SaaSGlue cloud service sends a message to the appropriate agent via a message queue with instructions for creating the Kubernetes Job. The agent then creates the Job using the Kubernetes API.

The Monitor tab in the SaaSGlue console displays status updates in real-time from start to completion as the Job executes. Historical logs are also available in the Monitor.

The running stdout tail is shown in the main view and pop-up links are available to see the full script, stdout and stderr.

Additional Features

SaaSGlue includes many powerful job scheduling features beyond those demonstrated in the sample application.

• Start/end dates, jitter and the time zone to use.
• Interval, cron and single execution schedules.
• Limit the number of instances of a job that can run in parallel.
• Optionally coalesce multiple queued job instances.
• Set the “misfire grace time” to skip the scheduled execution if more than the specified period of time has passed since the scheduled time.
• Pause future job creation if an instance of the job fails.
• Send slack/email alerts on job completion/failure.
• Chain multiple tasks together in a workflow with outcome based routing. Workflow tasks can run in any environment, not just Kubernetes.
• Export/import job definitions.
• The scripts that actually create the Kubernetes Jobs are imported with the “Kubernetes Job Runner” job and are completely customizable for specific use cases so you can leverage the full Kubernetes API unlike no/low code solutions like Argo CD.

For full details check out the documentation.