Switch Between Host-based and Path-based Serving¶
This guide will walk you through switching between host-based and path-based serving.
- Host-based: Each model is exposed under a different subdomain. This requires a wildcard certificate and a wildcard DNS entry. This option is the only one available in EKF 1.5.
- Path-based: All models are exposed under a single subdomain, but under a different prefix. EKF 2.0 supports this option and we recommend you use it.
Important
This guide assumes that you have already exposed serving during the initial deployment. It also assumes that you will use the existing load balancer and TLS configuration.
Overview
What You’ll Need¶
- A configured management environment.
- Your clone of the Arrikto GitOps repository.
- An existing Kubernetes cluster.
- An existing Kubeflow deployment.
- A configured domain for Serving.
Procedure¶
Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:~# cd ~/ops/deploymentsSpecify the serving type you want. Choose one of the following options based on whether you want to use path-based or host-based serving.
root@rok-tools:~/ops/deployments# export SERVING_TYPE=path-basedroot@rok-tools:~/ops/deployments# export SERVING_TYPE=host-basedSpecify the serving subdomain:
root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=<SUBDOMAIN>Replace
<SUBDOMAIN>
with your serving domain, for example:root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=arrikto-cluster-serving.serving.example.comImportant
This should be the same with the domain you used when generating your certificate.
Specify the serving prefix. Choose one of the following options based on whether you want to use path-based or host-based serving.
root@rok-tools:~/ops/deployments# export SERVING_PREFIX=<PREFIX>Replace
<PREFIX>
with your desired prefix, for example:root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/servingroot@rok-tools:~/ops/deployments# export SERVING_PREFIX=/Set the KServe URL template. Choose one of the following options based on whether you want to use path-based or host-based serving.
root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE="https://${SERVING_SUBDOMAIN?}${SERVING_PREFIX?}/{{ .Namespace }}/{{ .Name }}"root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE=Render the patch for KServe:
root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml.j2 \ > -o kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yamlEdit
kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/kustomization.yaml
and ensure that theingress
patch is enabled:patches: - path: patches/ingress.yamlRender the patch for Knative:
root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yaml.j2 \ > -o kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yamlEdit
kubeflow/manifests/common/knative/knative-serving/overlays/deploy/kustomization.yaml
and ensure that theconfig-domain
patch is enabled:patches: - path: patches/config-domain.yamlRender the Istio ingress patch template with the variables you have specified:
root@rok-tools:~/ops/deployments# j2 \ > rok/expose-serving/overlays/deploy/patches/ingress-host.yaml.j2 \ > -o rok/expose-serving/overlays/deploy/patches/ingress-host.yamlEdit
rok/expose-serving/overlays/deploy/kustomization.yaml
and ensure that theingress-host
patch is enabled:patches: - path: patches/ingress-host.yaml target: kind: Ingress name: knative-serving-ingressRender the patch for TLS termination. Choose one of the following options based on your cloud provider and the load balancer you are using.
TLS termination takes place on ALB with an ACM certificate. You do not have to use a TLS patch, so skip this step.
root@rok-tools:~/ops/deployments# j2 \ > rok/expose-serving/overlays/deploy/patches/ingress-tls.yaml.j2 \ > -o rok/expose-serving/overlays/deploy/patches/ingress-tls.yamlroot@rok-tools:~/ops/deployments# j2 \ > rok/expose-serving/overlays/deploy/patches/ingress-tls.yaml.j2 \ > -o rok/expose-serving/overlays/deploy/patches/ingress-tls.yamlroot@rok-tools:~/ops/deployments# j2 \ > rok/expose-serving/overlays/deploy/patches/ingress-tls.yaml.j2 \ > -o rok/expose-serving/overlays/deploy/patches/ingress-tls.yamlOptional
If you are running on AWS EKS and have exposed serving with Application Load Balancer (ALB), save your new state:
root@rok-tools:~/ops/deployments# j2 deploy/env.serving.j2 \ > -o deploy/env.servingCommit your changes:
root@rok-tools:~/ops/deployments# git commit -am "Switch Between Host-based and Path-based Serving"Apply the manifests:
root@rok-tools:~/ops/deployments# rok-deploy --apply \ > kubeflow/manifests/common/knative/knative-serving/overlays/deploy \ > kubeflow/manifests/contrib/kserve/kserve/overlays/deploy \ > rok/expose-serving/overlays/deploy
Verify¶
Ensure that the
urlTemplate
for KServe is equal to the one you previously set. Choose one of the following options based on whether you use path-based or host-based serving.root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate "https://arrikto-cluster-serving.serving.example.com/serving/{{ .Namespace }}/{{ .Name }}"root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate ""Ensure that you have configured Knative for you serving subdomain. Ensure that the output of the following command is equal to your serving subdomain:
root@rok-tools:~# kubectl get configmap config-domain \ > -n knative-serving -o jsonpath={.data} \ > | jq keys[] "arrikto-cluster-serving.serving.example.com"Verify that you have successfully created the Ingress object for Istio. Ensure that the HOSTS field includes both your subdomain and your wildcard subdomain:
root@rok-tools:~# kubectl get ingress -n knative-serving NAME CLASS HOSTS ADDRESS PORTS AGE knative-serving-ingress nginx-serving *.arrikto-cluster-serving.serving.example.com,arrikto-cluster-serving.serving.example.com e53a524a-ingressnginx-ingr-8872-592794601.us-east-1.elb.amazonaws.com 80 1m
What’s Next¶
Check out the rest of the operations you can perform on your Kubeflow deployment.