Exploring Path-Based and Header-Based Routing Solutions in Knative
Currently, in Knative version 1.1.0, traffic routing is primarily based on domain names.
However, in many use cases:
- Services often have fixed external domain names, and there may be multiple of them.
- Services are typically organized under specific paths within a domain, meaning one domain consists of multiple services.
- Grey releases are based on multiple header values with
and
andor
relationships.
Let’s discuss how to address these requirements.
1 Domain Binding
Apart from using Knative’s default domain, Knative provides domainMapping to bind additional domains.
Create a Knative service named example-nginx
, and it will generate the domain example-nginx.default.example.com
.
Then, you can bind another domain, such as out.com
, to example-nginx
.
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
annotations:
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
name: example-nginx
namespace: default
spec:
template:
metadata:
creationTimestamp: null
spec:
containerConcurrency: 0
containers:
- image: xxx.com/k8s/test-nginx:v1.21
name: nginx
ports:
- containerPort: 80
protocol: TCP
readinessProbe:
successThreshold: 1
tcpSocket:
port: 0
resources: {}
enableServiceLinks: false
timeoutSeconds: 300
traffic:
- latestRevision: true
percent: 50
- latestRevision: false
percent: 50
revisionName: example-nginx-00006
tag: canary
status:
address:
url: http://example-nginx.default.svc.cluster.local
conditions:
- lastTransitionTime: "2022-01-12T13:07:07Z"
status: "True"
type: ConfigurationsReady
- lastTransitionTime: "2022-01-12T13:08:15Z"
status: "True"
type: Ready
- lastTransitionTime: "2022-01-12T13:08:15Z"
status: "True"
type: RoutesReady
latestCreatedRevisionName: example-nginx-00006
latestReadyRevisionName: example-nginx-00006
observedGeneration: 13
traffic:
- latestRevision: true
percent: 50
revisionName: example-nginx-00006
- latestRevision: false
percent: 50
revisionName: example-nginx-00006
tag: canary
url: http://canary-example-nginx.default.example.com
url: http://example-nginx.default.example.com
apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata:
name: out.com
namespace: default
spec:
ref:
name: example-nginx
namespace: default
kind: Service
apiVersion: serving.knative.dev/v1
1.1 Underlying implementation principles
Knative services and DomainMappings both generate Knative Ingress resources. Knative utilizes Ingress to abstract traffic routing, while Net Istio uses Istio to implement traffic routing. The Net Istio component translates Knative’s Ingress objects into Istio’s VirtualService objects.
Therefore, domain binding effectively involves working with Knative Ingress and Istio VirtualService.
The hierarchy of these resources is as follows: Knative Service -> Knative Route -> Knative Ingress -> VirtualService.
In the context of example-nginx
, it corresponds to Ingress resources and VirtualService resources."
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
metadata:
annotations:
networking.internal.knative.dev/rollout: '{"configurations":[{"configurationName":"example-nginx","percent":100,"revisions":[{"revisionName":"example-nginx-00006","percent":100}],"stepParams":{}}]}'
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
creationTimestamp: "2022-01-11T06:21:28Z"
finalizers:
- ingresses.networking.internal.knative.dev
generation: 8
labels:
serving.knative.dev/route: example-nginx
serving.knative.dev/routeNamespace: default
serving.knative.dev/service: example-nginx
name: example-nginx
namespace: default
spec:
httpOption: Enabled
rules:
- hosts:
- example-nginx.default
- example-nginx.default.svc
- example-nginx.default.svc.cluster.local
http:
paths:
- headers:
Knative-Serving-Tag:
exact: canary
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
- appendHeaders:
Knative-Serving-Default-Route: "true"
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ClusterLocal
- hosts:
- example-nginx.default.example.com
http:
paths:
- headers:
Knative-Serving-Tag:
exact: canary
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
- appendHeaders:
Knative-Serving-Default-Route: "true"
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ExternalIP
- hosts:
- canary-example-nginx.default
- canary-example-nginx.default.svc
- canary-example-nginx.default.svc.cluster.local
http:
paths:
- appendHeaders:
Knative-Serving-Tag: canary
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ClusterLocal
- hosts:
- canary-example-nginx.default.example.com
http:
paths:
- appendHeaders:
Knative-Serving-Tag: canary
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ExternalIP
status:
conditions:
- lastTransitionTime: "2022-01-12T13:08:15Z"
status: "True"
type: LoadBalancerReady
- lastTransitionTime: "2022-01-11T06:21:28Z"
status: "True"
type: NetworkConfigured
- lastTransitionTime: "2022-01-12T13:08:15Z"
status: "True"
type: Ready
observedGeneration: 8
privateLoadBalancer:
ingress:
- domainInternal: knative-local-gateway.istio-system.svc.cluster.local
publicLoadBalancer:
ingress:
- domainInternal: istio-ingressgateway.istio-system.svc.cluster.local
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
networking.internal.knative.dev/rollout: '{"configurations":[{"configurationName":"example-nginx","percent":100,"revisions":[{"revisionName":"example-nginx-00006","percent":100}],"stepParams":{}}]}'
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
creationTimestamp: "2022-01-11T06:21:28Z"
generation: 32
labels:
networking.internal.knative.dev/ingress: example-nginx
serving.knative.dev/route: example-nginx
serving.knative.dev/routeNamespace: default
name: example-nginx-ingress
namespace: default
ownerReferences:
- apiVersion: networking.internal.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: example-nginx
uid: 94592245-c9ef-4321-88a5-a7d3cd57b383
spec:
gateways:
- knative/knative-ingress-gateway
- knative/knative-local-gateway
hosts:
- canary-example-nginx.default
- canary-example-nginx.default.example.com
- canary-example-nginx.default.svc
- canary-example-nginx.default.svc.cluster.local
- example-nginx.default
- example-nginx.default.example.com
- example-nginx.default.svc
- example-nginx.default.svc.cluster.local
http:
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
match:
- authority:
prefix: example-nginx.default
gateways:
- knative/knative-local-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
Knative-Serving-Default-Route: "true"
match:
- authority:
prefix: example-nginx.default
gateways:
- knative/knative-local-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- match:
- authority:
prefix: example-nginx.default
gateways:
- knative/knative-local-gateway
headers:
Knative-Serving-Tag:
exact: canary
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
Knative-Serving-Default-Route: "true"
match:
- authority:
prefix: example-nginx.default
gateways:
- knative/knative-local-gateway
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
match:
- authority:
prefix: example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
Knative-Serving-Default-Route: "true"
match:
- authority:
prefix: example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- match:
- authority:
prefix: example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
headers:
Knative-Serving-Tag:
exact: canary
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
Knative-Serving-Default-Route: "true"
match:
- authority:
prefix: example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
Knative-Serving-Tag: canary
match:
- authority:
prefix: canary-example-nginx.default
gateways:
- knative/knative-local-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
Knative-Serving-Tag: canary
match:
- authority:
prefix: canary-example-nginx.default
gateways:
- knative/knative-local-gateway
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
Knative-Serving-Tag: canary
match:
- authority:
prefix: canary-example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
Knative-Serving-Tag: canary
match:
- authority:
prefix: canary-example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
The ingress and virtualservice corresponding to DomainMapping
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
metadata:
annotations:
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
creationTimestamp: "2022-01-12T06:06:48Z"
finalizers:
- ingresses.networking.internal.knative.dev
generation: 1
labels:
serving.knative.dev/domainMappingNamespace: default
serving.knative.dev/domainMappingUID: 111ff4ba-a357-4861-a089-5c35b8760770
name: out.com
namespace: default
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: DomainMapping
name: out.com
uid: 111ff4ba-a357-4861-a089-5c35b8760770
resourceVersion: "393747308"
uid: 81fa388f-1211-4d82-9cc4-46c4e38c79e6
spec:
httpOption: Enabled
rules:
- hosts:
- out.com
http:
paths:
- rewriteHost: example-nginx.default.svc.cluster.local
splits:
- appendHeaders:
K-Original-Host: out.com
percent: 100
serviceName: example-nginx
serviceNamespace: default
servicePort: 80
visibility: ExternalIP
status:
conditions:
- lastTransitionTime: "2022-01-12T06:06:49Z"
status: "True"
type: LoadBalancerReady
- lastTransitionTime: "2022-01-12T06:06:48Z"
status: "True"
type: NetworkConfigured
- lastTransitionTime: "2022-01-12T06:06:49Z"
status: "True"
type: Ready
observedGeneration: 1
privateLoadBalancer:
ingress:
- domainInternal: knative-local-gateway.istio-system.svc.cluster.local
publicLoadBalancer:
ingress:
- domainInternal: istio-ingressgateway.istio-system.svc.cluster.local
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
creationTimestamp: "2022-01-12T06:06:48Z"
generation: 1
labels:
networking.internal.knative.dev/ingress: out.com
name: out.com-ingress
namespace: default
ownerReferences:
- apiVersion: networking.internal.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: out.com
uid: 81fa388f-1211-4d82-9cc4-46c4e38c79e6
resourceVersion: "393747283"
uid: 61148c5f-5541-4b35-90d8-1c90c6601c66
spec:
gateways:
- knative/knative-ingress-gateway
hosts:
- out.com
http:
- headers:
request:
set:
K-Network-Hash: b5c1fc426bf11303c158115fde3a06530e29d7aac00c60d6f469b6a0948ae362
match:
- authority:
prefix: out.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
rewrite:
authority: example-nginx.default.svc.cluster.local
route:
- destination:
host: example-nginx.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
K-Original-Host: out.com
weight: 100
- match:
- authority:
prefix: out.com
gateways:
- knative/knative-ingress-gateway
retries: {}
rewrite:
authority: example-nginx.default.svc.cluster.local
route:
- destination:
host: example-nginx.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
K-Original-Host: out.com
weight: 100
2 Forwarding Based on Paths
If a domain is composed of multiple services, you’ll need to deploy multiple Knative services. A Knative service will generate a domain, so for one external service domain, there will correspond to multiple Knative domains.
How can you achieve a single domain for external access with path-based forwarding?
- You can use the official example method and directly manipulate VirtualService.
- Currently, Knative Ingress supports forwarding based on headers and paths. You can directly manipulate Knative Ingress to achieve this.
Here, let’s discuss how to use Knative Ingress to implement it:
Create a Knative Ingress configuration with the domain as “url.com” and the path as “/url-1”.
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
metadata:
annotations:
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
name: url-1
spec:
httpOption: Enabled
rules:
- hosts:
- url.com
http:
paths:
- headers:
version:
exact: canary
abc:
exact: abc
path: "/url-1"
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ExternalIP
the generated virtualservice
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
creationTimestamp: "2022-01-15T10:35:46Z"
generation: 1
labels:
networking.internal.knative.dev/ingress: url-1
name: url-1-ingress
namespace: default
ownerReferences:
- apiVersion: networking.internal.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: url-1
uid: 0a1f2e17-cacf-4656-8b75-8f3df9296372
resourceVersion: "399816649"
selfLink: /apis/networking.istio.io/v1beta1/namespaces/default/virtualservices/url-1-ingress
uid: 58d750d0-150a-4baa-8100-030e7c8bd40c
spec:
gateways:
- knative/knative-ingress-gateway
hosts:
- url.com
http:
- headers:
request:
set:
K-Network-Hash: 1eb7ef10f7c01124526d9c594a088852b7146b167a1641af4b62eb3278e2ff59
match:
- authority:
prefix: url.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
uri:
prefix: /url-1
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- match:
- authority:
prefix: url.com
gateways:
- knative/knative-ingress-gateway
headers:
version:
exact: canary
uri:
prefix: /url-1
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
3 Implementing Header-Based Routing with Multiple Headers for Canary Deployment
How can you achieve routing based on multiple headers?
- You can directly manipulate VirtualService.
- You can use Knative Ingress to implement this.
Here, we’ll focus on using Knative Ingress to achieve it.
Currently, Knative Ingress supports precise header matching and supports both “OR” and “AND” relationships between headers.
In the example below, version:canary
and abc:abc
have an “AND” relationship.
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
.....
spec:
httpOption: Enabled
rules:
- hosts:
- url.com
http:
paths:
- headers:
version:
exact: canary
abc:
exact: abc
path: "/url-1"
In the example below, version:canary
and abc:abc
have an “OR” relationship.
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
.....
spec:
httpOption: Enabled
rules:
- hosts:
- url.com
http:
paths:
- headers:
version:
exact: canary
path: "/url-1"
- headers:
abc:
exact: abc
path: "/url-1"
3.1 Issues in net-istio Implementation
While Knative Ingress supports path-based routing and header-based routing, there are several issues in the net-istio implementation:
In the example of path-based routing above, the defined headers
version:canary
andabc:abc
are in an “AND” relationship, but the resulting VirtualService only contains one header matching condition. Relevant issue link: https://github.com/knative-sandbox/net-istio/issues/847- match: - authority: prefix: url.com gateways: - knative/knative-ingress-gateway headers: version: exact: canary uri: prefix: /url-1
net-istio performs probe detection on newly created ingress resources. It does this by accessing the Istio Ingress Gateway at
url.com:80/healthz
. If it returns a 200 status (or returns 200 and the K-Network-Hash header in the response matches the spec hash value of the ingress), it updates the condition in the ingress status to true.However, the generated VirtualService rules do not include a path for
/healthz
, causing the probe to return a 404 status. While this does not affect VirtualService generation, the ingress status remains in an “unknown” state. Relevant issue link: https://github.com/knative-sandbox/net-istio/issues/848Currently, you can set up an EnvoyFilter rule to match the
/healthz
path ofurl.com
and return a direct 200 status to resolve this issue.status: conditions: - lastTransitionTime: "2022-01-15T10:35:46Z" message: Waiting for load balancer to be ready reason: Uninitialized status: Unknown type: LoadBalancerReady - lastTransitionTime: "2022-01-15T10:35:46Z" status: "True" type: NetworkConfigured - lastTransitionTime: "2022-01-15T10:35:46Z" message: Waiting for load balancer to be ready reason: Uninitialized status: Unknown type: Ready observedGeneration: 1
Related issue link: https://github.com/knative-sandbox/net-istio/issues/848
Currently, you can set up an EnvoyFilter rule to match the
/healthz
path ofurl.com
and return a direct 200 status to resolve this issue.apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: net-istio-probe namespace: namespace-of-sidecar spec: workloadSelector: labels: app: istio-ingressgateway configPatches: - applyTo: HTTP_ROUTE match: context: GATEWAY routeConfiguration: vhost: name: url.com:80 patch: operation: INSERT_FIRST value: name: direct-healthz-response match: prefix: /healthz directResponse: body: inlineString: 'ok' status: 200
4 Community Progress
The community is proposing the definition of a Dispatcher to achieve path and header routing forwarding, with the underlying design based on ingress and VirtualService.
Design Prototype: Use DomainMapping to bind domains and Dispatcher, configure path and header routing forwarding in Dispatcher, and finally generate Knative Ingress.
Related issue: https://github.com/knative/serving/issues/11997
Related links:
https://docs.google.com/document/d/1q3kkBhSqWMHm-TCFo56R-32C7-fo6G8KQH4lIGtc-U0/edit
https://github.com/knative/serving/issues/540
https://www.likakuli.com/posts/knative-pathfilter/
https://github.com/Kong/kubernetes-ingress-controller/issues/584
https://github.com/knative/docs/tree/main/code-samples/serving/knative-routing-go
5 Conclusion
Knative’s Ingress has already implemented routing based on routing rules, but Knative Service does not yet support it. So, based on the current Knative version 1.1.0, there are two approaches to binding domains and implementing path and header forwarding:
Directly manipulate VirtualService.
Pros: Supports not only path and header matching but also other features like path regex matching and domain regex matching.
Cons: Only supports Istio.
Operate Knative Ingress.
Pros: Uses Knative-native resources, follows community standards, and has good compatibility. It can work with any implementation.
Cons: Only supports exact matching, lacks some advanced features. However, you can extend functionality similar to Kubernetes Ingress resources using labels or annotations.