Configure Argo Workflow Executor¶
A workflow executor allows Argo to perform certain actions like monitoring Pod logs, collecting artifacts, managing the life cycle of containers, etc. Certain executors introduce some limitations in specific scenarios or environments, so the admin might need to change the executor depending on their needs.
For example, here are some known issues with certain executors:
- PNS executor: Kubeflow Pipelines fail when Pods are spawned in a newly created node or when they involve short-lived containers.
- Docker executor: Kubeflow Pipelines do not work on AKS clusters with Linux node pools running Kubernetes 1.19 or newer, since they no longer support Docker.
- Emissary executor: Kubeflow Pipelines with custom steps will fail if the user does not explicitly define a command and args for the container.
See also
This guide will walk you through configuring a custom Argo workflow executor.
Note
We suggest using the Emissary Argo workflow executor, which is the default executor for Kubeflow Pipelines, unless you want to specify custom KFP steps without explicitly specifying command and args for the containers.
Overview
What You’ll Need¶
- A configured management environment.
- An Arrikto EKF deployment.
Procedure¶
Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:~# cd ~/ops/deploymentsCreate a
patches
directory in the Kubeflow Pipelines deploy overlay:root@rok-tools:~/ops/deployments# mkdir -p kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patchesCreate the
kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml
patch:root@rok-tools:~/ops/deployments# touch kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yamlEdit the
kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yaml
patch to configure the executor that the Argo workflow controller will use. Append the following lines if the file is empty or edit the existing patch accordingly if it is not. Choose one of the following options, based on the executor you will use.apiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap data: containerRuntimeExecutor: dockerapiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap data: containerRuntimeExecutor: pnsapiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap data: containerRuntimeExecutor: emissaryEdit the
kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/kustomization.yaml
file and append the following lines to enable the previous patch. Edit the existingpatches
directive instead, if it is already present:patches: - patches/workflow-controller-configmap.yamlStage your changes:
root@rok-tools:~/ops/deployments# git add kubeflow/manifests/apps/pipeline/upstream/overlays/deploy/patches/workflow-controller-configmap.yamlCommit your changes:
root@rok-tools:~/ops/deployments# git commit -am "Configure Argo workflow executor"Apply the Kubeflow Pipelines manifests:
root@rok-tools:~/ops/deployments# rok-deploy --apply kubeflow/manifests/apps/pipeline/upstream/overlays/deploy
Verify¶
Verify that the
ConfigMap
of the Argo workflow controller has been updated with the executor you specified earlier:root@rok-tools:~# kubectl get -n kubeflow configmap workflow-controller-configmap -o json \ > | jq -r '.data.containerRuntimeExecutor' dockerroot@rok-tools:~# kubectl get -n kubeflow configmap workflow-controller-configmap -o json \ > | jq -r '.data.containerRuntimeExecutor' pnsroot@rok-tools:~# kubectl get -n kubeflow configmap workflow-controller-configmap -o json \ > | jq -r '.data.containerRuntimeExecutor' emissary
What’s Next¶
Check out the rest of the operations you can perform on your Kubeflow deployment.