Rok 1.0¶
Important
This version is no longer supported. Please follow Rok 1.4 to upgrade Rok to the latest version.
This guide assumes that you have already cloned Arrikto's deployment repository that contains needed Kubernetes manifests, i.e., https://github.com/arrikto/deployments.
Change your current directory to your local clone of Arrikto's GitOps deployment repository. For example:
$ cd ~/ops/deployments
Note
This guide uses the deploy
overlay in the commands to be executed since
this is the overlay that users are supposed to tailor based on their needs
and preferences.
Notable changes v0.15 -> v1.0¶
- Until
v0.15
Rok integrates with Istiov1.3.1
. Inv1.0
we upgraded Rok to work with Istiov1.5.7
. - Rok tools now persists the whole of
/root
and not only/root/ops
by default.
Upgrade your management environment¶
This version of Rok changes the way data is persisted in your management
environment: from now on, rok-tools
persists the whole of /root
, i.e,
not only the local GitOps repository but also user settings and credentials,
e.g., under ~/.aws
and ~/.ssh
.
Important
The location where the volume of rok-tools
is mounted has changed from
/root/ops
to /root
.
Important
The steps below will instruct you how to mirror the local GitOps repo in a
private remote, so that you can later clone it into a new, empty
volume that the upgraded instance of rok-tools
will use to persist
data.
First, add an extra remote of your choice so you can push local changes there:
root@rok-tools-0:/# cd ~/ops/deployments root@rok-tools-0:~/ops/deployments# git remote add private <repo-url>
Then, push your local GitOps repository to the remote you just added to safely keep your current changes. For example:
root@rok-tools-0:~/ops/deployments# git push private develop:develop
(Docker only) If you are running
rok-tools
as a local Docker container you need to clear the host directory that was previously used by Docker as the volume ofrok-tools
. For example:$ rm -rf rok-tools-data/*
Follow the steps described in the Upgrade your management environment section
Exec into the upgraded
rok-tools
container:For Kubernetes:
$ kubectl exec -ti statefulset/rok-tools /bin/bash
For Docker:
$ docker exec -ti <ROK_TOOLS_CONTAINER_ID> /bin/bash
Since any changes under
/root
are lost, you have to reconfigure your management environment based on the configuration sections of the Prepare Management Environment guide, as needed. From now on, changes in files under/root
will now be persisted in the external volume.Restore your GitOps repository locally to make all your latest changes available in the new
rok-tools
instance:root@rok-tools-0:/# mkdir ~/ops && cd ~/ops root@rok-tools-0:~/ops# git clone --branch develop --origin private <repo-url> deployments
Add the Arrikto provided remote:
root@rok-tools-0:/# cd ~/ops/deployments root@rok-tools-0:~/ops/deployments# git remote add origin git@github.com:arrikto/deployments.git
Update local repo:
root@rok-tools-0:~/ops/deployments# git fetch --all -p
Ensure your local branch tracks the Arrikto-managed one:
root@rok-tools-0:~/ops/deployments# git branch -u origin/develop
(Optional) Remove the private remote:
root@rok-tools-0:~/ops/deployments# git remote remove private
Reconfigure
kubectl
so that you can connect to your Kubernetes cluster. Depending on your environment and setup you might need to follow the Access EKS Cluster section or copy in your kubeconfig file.(Kubernetes only) If you are running
rok-tools
as a Kubernetes StatefulSet you can optionally delete the old PVC, i.e.,rok-ops-rok-tools-0
, since the PVC has been renamed todata-rok-tools-0
and the old one is no longer needed.
Upgrade Manifests¶
Upgrade the local deployments repo and rebase your work over the release-1.0
branch to track the 1.0
version of Rok.
Fetch latest upstream changes:
$ git fetch --all -p
Retrieve the current upstream branch on which your current work is based:
$ git rev-parse --abbrev-ref --symbolic-full-name @{u} origin/develop
Important
The next command assumes you were working off the
origin/develop
upstream branch until now. Otherwise you must use the correct branch, based on the output of this step.Rebase over
release-1.0
and favor local changes upon conflicts:$ git rebase -Xtheirs --onto origin/release-1.0 origin/develop
Important
The above may cause conflicts, e.g., when a file was modified locally but removed from upstream, for example:
CONFLICT (modify/delete): kubeflow/kfctl_config.yaml deleted in origin/develop and modified in HEAD~61. Version HEAD~61 of kubeflow/kfctl_config.yaml left in tree.
We suggest here you go ahead and delete those files, e.g.:
$ git status --porcelain | awk '{if ($1=="DU") print $2}' | xargs git rm
And proceed with the rebase:
$ git rebase --continue
Configure the local branch to track
release-1.0
:$ git branch --set-upstream-to=origin/release-1.0
Upgrade Istio¶
Note
The upgrade procedure described below deviates from the generate/commit/apply model since requires some manual deletions.
Rok v1.0
uses a newer version of Istio (v1.5.7
), containing many bug
fixes and improvements. In order to upgrade to version 1.5.7
from version
1.3.1
:
Delete the previous Istio control plane installation:
$ kubectl delete -k rok/rok-external-services/istio/istio-1-3-1/istio-install-1-3-1/overlays/deploy
Apply the new Istio control plane:
$ kubectl apply -k rok/rok-external-services/istio/istio-1-5-7/istio-crds-1-5-7/overlays/deploy $ kubectl apply -k rok/rok-external-services/istio/istio-1-5-7/istio-namespace-1-5-7/overlays/deploy $ kubectl apply -k rok/rok-external-services/istio/istio-1-5-7/istio-install-1-5-7/overlays/deploy
Check the Envoy Proxy sidecars that exist in the cluster:
$ istioctl proxy-status NAME CDS LDS EDS RDS PILOT VERSION activator-cfc66dc7-tzdzx.knative-serving SYNCED SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.3.1 autoscaler-6cc8bc459b-jghcg.knative-serving SYNCED SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.3.1 cluster-local-gateway-9d544d7db-c2xbq.istio-system STALE (Never Acknowledged) SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.3.1 istio-ingressgateway-74649669b7-x77tt.istio-system SYNCED SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.5.7 prometheus-6c846d79b9-r8v4b.istio-system SYNCED SYNCED SYNCED SYNCED istiod-7c855cc66-fdqcg 1.5.7
Note
The above list might include Kubeflow related components that will be upgraded later on.
This release of Rok comes with an upgraded version of Istio to fix an issue with the handling of
X-Forwarded-*
headers by intermediate proxies, i.e., Rok URLs showed up ashttp
URLs instead ofhttps
ones. If you have HTTP proxies in front of your Istio IngressGateway installation (e.g., ALB, NGINX, etc.), make sure to configure X-Forwarded-* settings when you exposing Istio.Also have Ingress NGINX compute full X-Forwarded-For header by patching its configuration:
$ kubectl apply -f rok/nginx-ingress-controller/patch-configmap-l7.yaml
Delete Pod Disruption Budgets for Istio resources:
$ kubectl -n istio-system delete pdb istio-ingressgateway istiod cluster-local-gateway
Upgrade Rok components¶
Upgrade Rok components using latest kustomizations as described in the Upgrade components section.
Upgrade Kubeflow¶
If you have integrated your deployment with Deploy Kubeflow then you have to perform some manual actions before proceeding to the normal upgrade. Specifically
- Upgrade Kubeflow to use new Istio.
- Delete some deprecated resources.
Upgrade Kubeflow to use new Istio¶
Using new Istio requires you to manually delete some resources that cannot be updated. Specifically:
$ kubectl delete poddisruptionbudgets -n istio-system cluster-local-gateway
and:
$ kubectl delete deployment -n istio-system cluster-local-gateway
Also, delete KFServing Pods, so that Istio sidecars get upgraded:
$ kubectl delete pods -n knative-serving -l app=activator $ kubectl delete pods -n knative-serving -l app=autoscaler
Upgrade to use new Manifests¶
In Kubeflow 1.0, the upgrade procedure for Kubeflow was not in a
fetch/rebase/apply manner since you had to re-generate kustomizations using
kfctl
. This was a limitation of using kfctl
to produce Kubeflow 1.0
manifests.
For EKF with Kubeflow 1.1, we use kustomize
directly instead of kfctl
.
Using kustomize
allows us to streamline upgrades using a fetch/rebase/apply
workflow.
To migrate your manifests to the new Kubeflow 1.1 structure, follow the steps below:
Follow the Upgrade manifests guide to get the latest manifests.
At this point you have to recreate any changes you possibly made during the initial deployment of Kubeflow, which updated the content of
- the
kubeflow/kustomize
directory. - the
kubeflow/manifests
directory, which might have been overridden during rebase (see Upgrade manifests).
Make sure you move these changes to the new manifests structure where
- the
kustomize
folder does not exist anymore. - all custom changes should end up in the
deploy
overlay and not inekf
one of the corresponding application.
You are essentially configuring Kubeflow with the new style manifests from scratch. This is a one-time task. So you need to go through the following sections and ensure the configuration matches your needs:
- Configure Authentication (this section is no longer available in this version) and Tweak Dex (this section is no longer available in this version) to set users and OIDC credentials for Dex.
- GitLab to integrate AuthService with GitLab SSO.
- Extend List of JWA Default Images to add custom Images for Notebook servers.
- Grant JWA Access to Private Registry to change PodDefaults in Jupyter Web App.
The above steps will modify the
deploy
overlays of the new manifests. In case you want to compare the new changes with the ones previously made, here is a mapping between the two trees:¶ Kubeflow 1.0 (old, kustomize/, ekf overlays) Kubeflow 1.1 (new, manifests/, deploy overlays) kubeflow/kustomize/dex/overlays/ekf/secret_params.env kubeflow/manifests/dex-auth/dex-crds/overlays/deploy/secret_params.env kubeflow/kustomize/dex/overlays/ekf/patches/config-map.yaml kubeflow/manifests/dex-auth/dex-crds/overlays/deploy/patches/config-map.yaml kubeflow/kustomize/oidc-authservice/overlays/ekf/params.env kubeflow/manifests/istio/oidc-authservice/overlays/deploy/params.env kubeflow/kustomize/oidc-authservice/overlays/ekf/secret_params.env kubeflow/manifests/istio/oidc-authservice/overlays/deploy/secret_params.env kubeflow/kustomize/jupyter-web-app/overlays/ekf/patches/config-map.yaml kubeflow/manifests/jupyter/jupyter-web-app/overlays/deploy/patches/config-map.yaml kubeflow/manifests/dex-auth/dex-crds/overlays/ekf/secret_params.env kubeflow/manifests/dex-auth/dex-crds/overlays/deploy/secret_params.env kubeflow/manifests/istio/oidc-authservice/overlays/ekf/secret_params.env kubeflow/manifests/istio/oidc-authservice/overlays/deploy/secret_params.env kubeflow/manifests/jupyter/jupyter-web-app/overlays/ekf/patches/config-map.yaml kubeflow/manifests/jupyter/jupyter-web-app/overlays/deploy/patches/config-map.yaml - the
Undo custom changes in
ekf
overlays:$ git checkout origin/develop --\ > kubeflow/manifests/dex-auth/dex-crds/overlays/ekf/secret_params.env \ > kubeflow/manifests/istio/oidc-authservice/overlays/ekf/secret_params.env \ > kubeflow/manifests/jupyter/jupyter-web-app/overlays/ekf/patches/config-map.yaml
Commit changes:
$ git add kubeflow $ git commit -m "kubeflow: Refactor to 1.1 manifests"
Delete deprecated files related to
kfctl
and commit changes:$ git rm -r kubeflow/kustomize $ rm -rf kubeflow/.cache $ git commit -m "kubeflow: Delete deprecated kustomizations"
Apply:
$ rok-deploy --apply install/kubeflow --force
Warning
We have to force update because there are changes to immutable fields of resources and thus we should delete and recreate them. Note that this is done only for specific kind of resources that are safe to be deleted. These kinds are specified by
--force-kinds
and are by default Deployments and StatefulSets.
Delete deprecated resources¶
Previous versions were creating some resources which you do not need anymore. Follow this section to clean them up.
Gather all user namespaces:
$ NAMESPACES=$(kubectl get profiles -o jsonpath="{.items[*].metadata.name}")
Select the namespace-scoped and cluster-scoped resources to delete:
$ NAMESPACE_RESOURCES="rolebinding.rbac.authorization.k8s.io/sa-pipeline-runner-edit rolebinding.rbac.authorization.k8s.io/sa-default-editor-edit rolebinding.rbac.authorization.k8s.io/pipeline-runner-admin" $ CLUSTER_RESOURCES="clusterrole.rbac.authorization.k8s.io/cr-empty"
Delete them:
$ for ns in $NAMESPACES; do kubectl delete -n $ns $NAMESPACE_RESOURCES; done $ kubectl delete $CLUSTER_RESOURCES