Configure KServe¶
In this section you will configure KServe for exposing your served models to the outer world.
Fast Forward
If you have already configured KServe, expand this box to fast-forward.
Specify the type of serving you want. Choose one of the following options based on whether you use path-based or host-based serving.
root@rok-tools:~# export SERVING_TYPE=path-basedroot@rok-tools:~# export SERVING_TYPE=host-basedSet the serving subdomain:
root@rok-tools:~# export SERVING_SUBDOMAIN=${SERVING_EKS_ALB_ACM_SUBDOMAIN?}Specify the serving prefix. Choose one of the following options based on whether you use path-based or host-based serving.
root@rok-tools:~# export SERVING_PREFIX=<PREFIX>Replace
<PREFIX>
with your desired prefix, for example:root@rok-tools:~# export SERVING_PREFIX=/servingroot@rok-tools:~# export SERVING_PREFIX=/Set the KServe URL template. Choose one of the following options based on whether you use path-based or host-based serving.
root@rok-tools:~# export SERVING_URL_TEMPLATE="https://${SERVING_SUBDOMAIN?}${SERVING_PREFIX?}/{{ .Namespace }}/{{ .Name }}"root@rok-tools:~# export SERVING_URL_TEMPLATE=Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:/# cd ~/ops/deploymentsSave your state:
root@rok-tools:~/ops/deployments# j2 deploy/env.serving.j2 \ > -o deploy/env.servingProceed to the Verify section.
Choose one of the following options:
Overview
What You’ll Need¶
- A configured management environment.
- An existing EKS cluster.
Option 1: Configure KServe Automatically (preferred)¶
Under construction
This section is a work in progress. Use Option 2: Configure KServe Manually instead.
Configure KServe by following the on-screen instructions on the rok-deploy
user interface.
If rok-deploy
is not already running, start it with:
Proceed to the Summary section.
Option 2: Configure KServe Manually¶
If you want to configure KServe manually, follow the instructions below.
Procedure¶
Go to your GitOps repository, inside your
rok-tools
management environment:root@rok-tools:/# cd ~/ops/deploymentsRestore the required context from previous sections:
root@rok-tools:~/ops/deployments# source deploy/env.serving-eks-alb-acmroot@rok-tools:~/ops/deployments# export SERVING_EKS_ALB_ACM_SUBDOMAINSpecify the type of serving you want. Choose one of the following options based on whether you want path-based or host-based serving.
root@rok-tools:~/ops/deployments# export SERVING_TYPE=path-basedroot@rok-tools:~/ops/deployments# export SERVING_TYPE=host-basedSet the serving subdomain:
root@rok-tools:~/ops/deployments# export SERVING_SUBDOMAIN=${SERVING_EKS_ALB_ACM_SUBDOMAIN?}Note
Use the one you have specified when creating the SSL certificate.
Specify the serving prefix. Choose one of the following options based on whether you use path-based or host-based serving.
root@rok-tools:~/ops/deployments# export SERVING_PREFIX=<PREFIX>Replace
<PREFIX>
with your desired prefix, for example:root@rok-tools:~/ops/deployments# export SERVING_PREFIX=/servingroot@rok-tools:~/ops/deployments# export SERVING_PREFIX=/Set the KServe URL template. Choose one of the following options based on whether you use path-based or host-based serving.
root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE="https://${SERVING_SUBDOMAIN?}${SERVING_PREFIX?}/{{ .Namespace }}/{{ .Name }}"root@rok-tools:~/ops/deployments# export SERVING_URL_TEMPLATE=Render the patch for KServe:
root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yaml.j2 \ > -o kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/patches/ingress.yamlEdit
kubeflow/manifests/contrib/kserve/kserve/overlays/deploy/kustomization.yaml
and ensure that theingress
patch is enabled:patches: - path: patches/ingress.yamlRender the patch for Knative:
root@rok-tools:~/ops/deployments# j2 kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yaml.j2 \ > -o kubeflow/manifests/common/knative/knative-serving/overlays/deploy/patches/config-domain.yamlEdit
kubeflow/manifests/common/knative/knative-serving/overlays/deploy/kustomization.yaml
and ensure that theconfig-domain
patch is enabled:patches: - path: patches/config-domain.yamlSave your state:
root@rok-tools:~/ops/deployments# j2 deploy/env.serving.j2 \ > -o deploy/env.servingCommit your changes:
root@rok-tools:~/ops/deployments# git commit -am "Configure KServe"Apply the manifests:
root@rok-tools:~/ops/deployments# rok-deploy --apply \ > kubeflow/manifests/common/knative/knative-serving/overlays/deploy \ > kubeflow/manifests/contrib/kserve/kserve/overlays/deploy
Verify¶
Ensure that the
urlTemplate
for KServe is equal to the one you previously set. Choose one of the following options based on whether you use path-based or host-based serving.root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate "https://arrikto-cluster-serving.serving.example.com/serving/{{ .Namespace }}/{{ .Name }}"root@rok-tools:~# kubectl get configmap inferenceservice-config \ > -n kserve -o jsonpath={.data.ingress} \ > | jq .urlTemplate ""Ensure that you have configured Knative for you serving subdomain. Ensure that the output of the following command is equal to your serving subdomain:
root@rok-tools:~# kubectl get configmap config-domain \ > -n knative-serving -o jsonpath={.data} \ > | jq keys[] "arrikto-cluster-serving.serving.example.com"