Update Retention Policy of all Buckets¶
This guide will walk you through updating the version retention policy of existing buckets in a Rok API installation to the default options you have configured for new buckets in guide Configure Default Retention Policy for New Buckets.
Note
This guide’s purpose is to peform a bulk update of multiple version retention policies across multiple accounts. If you wish to configure the version retention policy of a single bucket, you can also do it via the Rok UI or CLI by following the Snapshot policies for Backup guide.
See also
Overview
What You’ll Need¶
- A configured management environment.
- Your clone of the Arrikto GitOps repository.
- An existing Kubeflow deployment.
- Configured settings for default retention policies.
Procedure¶
Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:~# cd ~/ops/deploymentsSpecify the accounts in which you wish to update version retention policies:
root@rok-tools:~# export ACCOUNTS=<ACCOUNTS>Replace
<ACCOUNTS>
with a comma-separated list of account names in which to update version retention policies. Set toALL
to apply changes to all accounts:root@rok-tools:~# export ACCOUNTS=ALLSpecify the accounts to exclude from the operation:
root@rok-tools:~# export EXCLUDE_ACCOUNTS=<ACCOUNTS>Replace
<ACCOUNTS>
with a comma-separated list of account names to exclude. Leave empty to apply to all accounts selected in the previous step:root@rok-tools:~# export EXCLUDE_ACCOUNTS=Specify the buckets of which you wish to update the version retention policy:
root@rok-tools:~# export BUCKETS=<BUCKETS>Replace
<BUCKETS>
with a comma-separated list of bucket names to update the version retention policy of. Set toALL
to apply to all buckets:root@rok-tools:~# export BUCKETS=ALLSpecify the buckets to exclude from the operation:
root@rok-tools:~# export EXCLUDE_BUCKETS=<BUCKETS>Replace
<BUCKETS>
with a comma-separated list of bucket names to exclude. You must always include thenotebooks
andauto-backup
buckets in this list, to retain the version retention policy settings that you have configured via your EKF manifests:root@rok-tools:~# export EXCLUDE_BUCKETS=notebooks,auto-backupImportant
EKF manifests already contain
RokBucketConfiguration
custom resources for the version retention policies of thenotebooks
andauto-backup
buckets. This means that even if you modify these policies as part of this guide, the operator will automatically revert the change.However, it is still possible that a new task will be created before the Rok operator has time to revert the change. This can lead to versions being deleted in these buckets, which especially in the case of the
notebooks
bucket may lead to malfunctioning Kubeflow Pipelines. Therefore, we strongly recommend to always exclude thenotebooks
andauto-backup
buckets when running this management command.Optional
By default the command will only affect version retention policies with out-of-the-box settings, i.e., no schedule and a single retention rule to retain all versions forever. If you wish to instead affect all policies, even if they have a custom configuration, specify the following setting:
root@rok-tools:~# export EXTRA_OPTIONS=--overwriteRun the command to update the version retention policies. Inspect the buckets that will be affected and rules that will be applied, and type
y
,<enter>
to proceed with the update:root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok-gw-manage retention-policy-update \ > --accounts "${ACCOUNTS?}" --exclude-accounts "${EXCLUDE_ACCOUNTS?}" \ > --buckets "${BUCKETS?}" --exclude-buckets "${EXCLUDE_BUCKETS?}" \ > ${EXTRA_OPTIONS} 2022-09-14T17:00:09.002161+0000 api pid=3032/tid=3032/pytid=140096893715904 cache:178 [INFO] Initialized authentication backends: AuthService, KubernetesToken ------------------------------------------------------------------------- Policy ID Account Bucket ------------------------------------------------------------------------- 2e1733e8-9543-4866-aa0d-29cef79ac3c1 kubeflow-user open-vaccine 6a9ac4fe-c649-4802-b15b-7838f3b690f0 kubeflow-user titanic ------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------- Default Version Retention Rules -------------------------------------------------------------------------------------------------------- Retain the current version of each object forever Retain all versions of each object for 1 week Retain one version of each object every 1 week (starting at 2022-09-12T00:00:00+00:00) for 6 months Retain one version of each object every 3 months (starting at 2022-09-01T00:00:00+00:00) for 5 years -------------------------------------------------------------------------------------------------------- ---------------------------------------------------------- Default Version Retention Schedule ---------------------------------------------------------- Run every 1 day, starting at 2022-09-14T00:00:00+00:00 ---------------------------------------------------------- This command will update the retention policies displayed above to use the default retention rules and schedule of the Rok Gateway. You can modify the Gateway's default rules and schedule by updating the VERSION_RETENTION_DEFAULT_RULES andVERSION_RETENTION_DEFAULT_SCHEDULE settings in '/etc/rok/gw/backend.conf.py'. Apply these changes? [y/N]: y 2022-09-14T17:05:20.991803+0000 api pid=3667/tid=3667/pytid=140560480060864 retention-policy-update:135 [INFO] Successfully updated 2 policies
Verify¶
Specify the namespace in which to verify the version retention policy of a bucket:
root@rok-tools:~# export NAMESPACE=<NAMESPACE>Replace
NAMESPACE
with the name of the namespace that you wish to verify:root@rok-tools:~# export NAMESPACE=kubeflow-userSpecify the name of the bucket to verify:
root@rok-tools:~# export BUCKET_NAME=<BUCKET_NAME>Replace
BUCKET_NAME
with the name of the bucket you wish to verify. Pick an existing bucket outside of the list of buckets you excluded above, for example:root@rok-tools:~# export BUCKET_NAME=verify-retentionRetrieve the ID of the version retention policy for the bucket:
root@rok-tools:~# export POLICY_ID=$(kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE} -o json \ > policy-list --bucket ${BUCKET_NAME?} | \ > jq -r -c ".[] | select(.action == \"version_retention\") | .policy_id")Display information about the retention policy. Verify that the scheduling rules and version retention rules match the default ones you have configured in the Configure Default Retention Policy for New Buckets guide:
root@rok-tools:~# kubectl exec -ti -n rok svc/rok -- \ > rok --account ${NAMESPACE?} policy-show ${POLICY_ID?} Policy Information Policy ID bea1e8a1-38fa-4786-a612-299f39ef060a Action version_retention Account kubeflow-user Bucket verify-retention Created At 2022-09-05T21:49:56.761403+00:00 Description Version Retention Policy Details Retention policy for versions in bucket `verify-retention' Paused False Current Run ID Next Run ID fbf4f09cbe114d7ca7f0f84ab1f6612f Next Run At 2022-09-09T00:00:00+00:00 Last Run ID bcac41be04094d12a2a005f27e29155c Last Run Status success Last Run Started At 2022-09-05T21:57:35.205674+00:00 Last Run Completed At 2022-09-05T21:57:35.560008+00:00 Scheduling Rule ID Interval Start Description 2e7c4c72-75da-442e-8f74-b8e5768701c5 1day 2022-09-05T00:00:00+00:00 Run every 1 day, starting at 2022-09-05T00:00:00+00:00 Version Retention Rule ID Strategy Params Description df216760-f2bd-4906-992f-8aacfc58d04b version_count 1,forever Retain the current version of each object forever e6602f83-d9c7-415b-887b-7872890e9c01 age all,1weeks Retain all versions of each object for 1 week 649c472a-f629-4d8a-b2df-845015269552 periodic 1per1weeks,2022-09-05T00:00:00+00:00,6months Retain one version of each object every 1 week (starting at 2022-09-05T00:00:00+00:00) for 6 months a1d912fb-6cac-4a8f-baa8-c1169f5cce0b periodic 1per3months,2022-09-01T00:00:00+00:00,5years Retain one version of each object every 3 months (starting at 2022-09-01T00:00:00+00:00) for 5 years