探索knative里根据路径转发和header转发解决方案
目前knative v1.1.0版本还是基于域名进行转发流量到服务。
而大多数使用场景是:
- 服务对外域名一般都是固定的,而且可能有多个。
- 服务一般都是在域名的某一路径下面,即一个域名由多个服务组成。
- 灰度基于多个header头之间
and
or
关系。
下面讨论如何实现这些需求
1 绑定域名
除了使用knative的默认域名,knative提供了domainMapping来绑定额外域名。
创建一个name为example-nginx
的knative service,会生成example-nginx.default.example.com
域名。
example-nginx
再绑定一个域名out.com
。
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
annotations:
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
name: example-nginx
namespace: default
spec:
template:
metadata:
creationTimestamp: null
spec:
containerConcurrency: 0
containers:
- image: xxx.com/k8s/test-nginx:v1.21
name: nginx
ports:
- containerPort: 80
protocol: TCP
readinessProbe:
successThreshold: 1
tcpSocket:
port: 0
resources: {}
enableServiceLinks: false
timeoutSeconds: 300
traffic:
- latestRevision: true
percent: 50
- latestRevision: false
percent: 50
revisionName: example-nginx-00006
tag: canary
status:
address:
url: http://example-nginx.default.svc.cluster.local
conditions:
- lastTransitionTime: "2022-01-12T13:07:07Z"
status: "True"
type: ConfigurationsReady
- lastTransitionTime: "2022-01-12T13:08:15Z"
status: "True"
type: Ready
- lastTransitionTime: "2022-01-12T13:08:15Z"
status: "True"
type: RoutesReady
latestCreatedRevisionName: example-nginx-00006
latestReadyRevisionName: example-nginx-00006
observedGeneration: 13
traffic:
- latestRevision: true
percent: 50
revisionName: example-nginx-00006
- latestRevision: false
percent: 50
revisionName: example-nginx-00006
tag: canary
url: http://canary-example-nginx.default.example.com
url: http://example-nginx.default.example.com
apiVersion: serving.knative.dev/v1beta1
kind: DomainMapping
metadata:
name: out.com
namespace: default
spec:
ref:
name: example-nginx
namespace: default
kind: Service
apiVersion: serving.knative.dev/v1
1.1 底层实现原理
knative service和DomainMapping都会生成knative ingress资源,knative使用ingress抽象了流量转发,而net-istio是使用istio来做流量转发流量转发的实现。net-istio组件将knative的ingresss对象转成istio的virtualservice。
所以绑定域名,其实就是在操作knative ingress和istio virtualservice。
各个资源的父子关系是knative service–> knative route–> knative ingress–>virtualservice。
example-nginx
对应的ingress资源和virtualservice资源
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
metadata:
annotations:
networking.internal.knative.dev/rollout: '{"configurations":[{"configurationName":"example-nginx","percent":100,"revisions":[{"revisionName":"example-nginx-00006","percent":100}],"stepParams":{}}]}'
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
creationTimestamp: "2022-01-11T06:21:28Z"
finalizers:
- ingresses.networking.internal.knative.dev
generation: 8
labels:
serving.knative.dev/route: example-nginx
serving.knative.dev/routeNamespace: default
serving.knative.dev/service: example-nginx
name: example-nginx
namespace: default
spec:
httpOption: Enabled
rules:
- hosts:
- example-nginx.default
- example-nginx.default.svc
- example-nginx.default.svc.cluster.local
http:
paths:
- headers:
Knative-Serving-Tag:
exact: canary
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
- appendHeaders:
Knative-Serving-Default-Route: "true"
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ClusterLocal
- hosts:
- example-nginx.default.example.com
http:
paths:
- headers:
Knative-Serving-Tag:
exact: canary
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
- appendHeaders:
Knative-Serving-Default-Route: "true"
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ExternalIP
- hosts:
- canary-example-nginx.default
- canary-example-nginx.default.svc
- canary-example-nginx.default.svc.cluster.local
http:
paths:
- appendHeaders:
Knative-Serving-Tag: canary
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ClusterLocal
- hosts:
- canary-example-nginx.default.example.com
http:
paths:
- appendHeaders:
Knative-Serving-Tag: canary
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ExternalIP
status:
conditions:
- lastTransitionTime: "2022-01-12T13:08:15Z"
status: "True"
type: LoadBalancerReady
- lastTransitionTime: "2022-01-11T06:21:28Z"
status: "True"
type: NetworkConfigured
- lastTransitionTime: "2022-01-12T13:08:15Z"
status: "True"
type: Ready
observedGeneration: 8
privateLoadBalancer:
ingress:
- domainInternal: knative-local-gateway.istio-system.svc.cluster.local
publicLoadBalancer:
ingress:
- domainInternal: istio-ingressgateway.istio-system.svc.cluster.local
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
networking.internal.knative.dev/rollout: '{"configurations":[{"configurationName":"example-nginx","percent":100,"revisions":[{"revisionName":"example-nginx-00006","percent":100}],"stepParams":{}}]}'
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
creationTimestamp: "2022-01-11T06:21:28Z"
generation: 32
labels:
networking.internal.knative.dev/ingress: example-nginx
serving.knative.dev/route: example-nginx
serving.knative.dev/routeNamespace: default
name: example-nginx-ingress
namespace: default
ownerReferences:
- apiVersion: networking.internal.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: example-nginx
uid: 94592245-c9ef-4321-88a5-a7d3cd57b383
spec:
gateways:
- knative/knative-ingress-gateway
- knative/knative-local-gateway
hosts:
- canary-example-nginx.default
- canary-example-nginx.default.example.com
- canary-example-nginx.default.svc
- canary-example-nginx.default.svc.cluster.local
- example-nginx.default
- example-nginx.default.example.com
- example-nginx.default.svc
- example-nginx.default.svc.cluster.local
http:
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
match:
- authority:
prefix: example-nginx.default
gateways:
- knative/knative-local-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
Knative-Serving-Default-Route: "true"
match:
- authority:
prefix: example-nginx.default
gateways:
- knative/knative-local-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- match:
- authority:
prefix: example-nginx.default
gateways:
- knative/knative-local-gateway
headers:
Knative-Serving-Tag:
exact: canary
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
Knative-Serving-Default-Route: "true"
match:
- authority:
prefix: example-nginx.default
gateways:
- knative/knative-local-gateway
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
match:
- authority:
prefix: example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
Knative-Serving-Default-Route: "true"
match:
- authority:
prefix: example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- match:
- authority:
prefix: example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
headers:
Knative-Serving-Tag:
exact: canary
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
Knative-Serving-Default-Route: "true"
match:
- authority:
prefix: example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
Knative-Serving-Tag: canary
match:
- authority:
prefix: canary-example-nginx.default
gateways:
- knative/knative-local-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
Knative-Serving-Tag: canary
match:
- authority:
prefix: canary-example-nginx.default
gateways:
- knative/knative-local-gateway
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
K-Network-Hash: 1d94c6d2b2727715d4e9ef39ef170cfe3863d7bf72b0f373841b77e47be1c3cc
Knative-Serving-Tag: canary
match:
- authority:
prefix: canary-example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- headers:
request:
set:
Knative-Serving-Tag: canary
match:
- authority:
prefix: canary-example-nginx.default.example.com
gateways:
- knative/knative-ingress-gateway
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
DomainMapping对应的ingress和virtualservice
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
metadata:
annotations:
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
creationTimestamp: "2022-01-12T06:06:48Z"
finalizers:
- ingresses.networking.internal.knative.dev
generation: 1
labels:
serving.knative.dev/domainMappingNamespace: default
serving.knative.dev/domainMappingUID: 111ff4ba-a357-4861-a089-5c35b8760770
name: out.com
namespace: default
ownerReferences:
- apiVersion: serving.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: DomainMapping
name: out.com
uid: 111ff4ba-a357-4861-a089-5c35b8760770
resourceVersion: "393747308"
uid: 81fa388f-1211-4d82-9cc4-46c4e38c79e6
spec:
httpOption: Enabled
rules:
- hosts:
- out.com
http:
paths:
- rewriteHost: example-nginx.default.svc.cluster.local
splits:
- appendHeaders:
K-Original-Host: out.com
percent: 100
serviceName: example-nginx
serviceNamespace: default
servicePort: 80
visibility: ExternalIP
status:
conditions:
- lastTransitionTime: "2022-01-12T06:06:49Z"
status: "True"
type: LoadBalancerReady
- lastTransitionTime: "2022-01-12T06:06:48Z"
status: "True"
type: NetworkConfigured
- lastTransitionTime: "2022-01-12T06:06:49Z"
status: "True"
type: Ready
observedGeneration: 1
privateLoadBalancer:
ingress:
- domainInternal: knative-local-gateway.istio-system.svc.cluster.local
publicLoadBalancer:
ingress:
- domainInternal: istio-ingressgateway.istio-system.svc.cluster.local
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
serving.knative.dev/creator: admin
serving.knative.dev/lastModifier: admin
creationTimestamp: "2022-01-12T06:06:48Z"
generation: 1
labels:
networking.internal.knative.dev/ingress: out.com
name: out.com-ingress
namespace: default
ownerReferences:
- apiVersion: networking.internal.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: out.com
uid: 81fa388f-1211-4d82-9cc4-46c4e38c79e6
resourceVersion: "393747283"
uid: 61148c5f-5541-4b35-90d8-1c90c6601c66
spec:
gateways:
- knative/knative-ingress-gateway
hosts:
- out.com
http:
- headers:
request:
set:
K-Network-Hash: b5c1fc426bf11303c158115fde3a06530e29d7aac00c60d6f469b6a0948ae362
match:
- authority:
prefix: out.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
retries: {}
rewrite:
authority: example-nginx.default.svc.cluster.local
route:
- destination:
host: example-nginx.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
K-Original-Host: out.com
weight: 100
- match:
- authority:
prefix: out.com
gateways:
- knative/knative-ingress-gateway
retries: {}
rewrite:
authority: example-nginx.default.svc.cluster.local
route:
- destination:
host: example-nginx.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
K-Original-Host: out.com
weight: 100
2 根据路径进行转发
一个域名由多个服务组成,则需要部署多个knative service。一个knative service会生成一个域名,那么一个对外服务的域名,会对应多个knative域名。
如何实现对外一个域名且根据路径进行转发呢?
- 使用官方的样例方法,直接操作virtualservice。
- 目前knative ingress支持根据header和path进行转发,可以直接操作knative ingress来实现。
这里讨论操作knative ingress来实现:
创建域名为url.com和路径为/url-1的knative ingress配置
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
metadata:
annotations:
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
name: url-1
spec:
httpOption: Enabled
rules:
- hosts:
- url.com
http:
paths:
- headers:
version:
exact: canary
abc:
exact: abc
path: "/url-1"
splits:
- appendHeaders:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
percent: 100
serviceName: example-nginx-00006
serviceNamespace: default
servicePort: 80
visibility: ExternalIP
生成的virtualservice
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
annotations:
networking.knative.dev/ingress.class: istio.ingress.networking.knative.dev
creationTimestamp: "2022-01-15T10:35:46Z"
generation: 1
labels:
networking.internal.knative.dev/ingress: url-1
name: url-1-ingress
namespace: default
ownerReferences:
- apiVersion: networking.internal.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Ingress
name: url-1
uid: 0a1f2e17-cacf-4656-8b75-8f3df9296372
resourceVersion: "399816649"
selfLink: /apis/networking.istio.io/v1beta1/namespaces/default/virtualservices/url-1-ingress
uid: 58d750d0-150a-4baa-8100-030e7c8bd40c
spec:
gateways:
- knative/knative-ingress-gateway
hosts:
- url.com
http:
- headers:
request:
set:
K-Network-Hash: 1eb7ef10f7c01124526d9c594a088852b7146b167a1641af4b62eb3278e2ff59
match:
- authority:
prefix: url.com
gateways:
- knative/knative-ingress-gateway
headers:
K-Network-Hash:
exact: override
uri:
prefix: /url-1
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
- match:
- authority:
prefix: url.com
gateways:
- knative/knative-ingress-gateway
headers:
version:
exact: canary
uri:
prefix: /url-1
retries: {}
route:
- destination:
host: example-nginx-00006.default.svc.cluster.local
port:
number: 80
headers:
request:
set:
Knative-Serving-Namespace: default
Knative-Serving-Revision: example-nginx-00006
weight: 100
3 根据多个header的进行灰度
如何实现根据多个header进行转发呢?
- 直接操作virtualservice。
- 直接操作knative ingress来实现。
这里只讨论操作knative ingress来实现。
目前knative ingress只支持header头精确匹配,支持header头的or和and关系。
下面的例子中,version:canary
和abc:abc
是and关系
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
.....
spec:
httpOption: Enabled
rules:
- hosts:
- url.com
http:
paths:
- headers:
version:
exact: canary
abc:
exact: abc
path: "/url-1"
下面例子中,version:canary
和abc:abc
是or关系
apiVersion: networking.internal.knative.dev/v1alpha1
kind: Ingress
.....
spec:
httpOption: Enabled
rules:
- hosts:
- url.com
http:
paths:
- headers:
version:
exact: canary
path: "/url-1"
- headers:
abc:
exact: abc
path: "/url-1"
3.1 net-istio实现中存在的问题
虽然knative ingress支持按路径转发和header转发,但是在net-istio实现中存在这几个问题:
在上面路径转发中的例子,定义的header
version:canary
和abc:abc
是and关系,但是最终生成的virtualservice里只有一个header匹配条件。相关issue链接https://github.com/knative-sandbox/net-istio/issues/847- match: - authority: prefix: url.com gateways: - knative/knative-ingress-gateway headers: version: exact: canary uri: prefix: /url-1
net-istio会对新创建的ingress进行probe探测,它通过访问istio ingress gateway访问
url.com:80/healthz
,如果返回200(或返回200且返回header头里K-Network-Hash与ingress的spec hash值相等)则修改ingress status里的condition为true。但是生成的virtualservice规中并没有配置/healthz路径转发,导致探测返回404。虽然不影响virtualservice生成,但是ingress status一直是unknown状态。
status: conditions: - lastTransitionTime: "2022-01-15T10:35:46Z" message: Waiting for load balancer to be ready reason: Uninitialized status: Unknown type: LoadBalancerReady - lastTransitionTime: "2022-01-15T10:35:46Z" status: "True" type: NetworkConfigured - lastTransitionTime: "2022-01-15T10:35:46Z" message: Waiting for load balancer to be ready reason: Uninitialized status: Unknown type: Ready observedGeneration: 1
相关issue链接https://github.com/knative-sandbox/net-istio/issues/848
目前可以设置envoyfilter规则,匹配url.com的/healthz路径直接返回200来解决这个问题
apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: net-istio-probe namespace: namespace-of-sidecar spec: workloadSelector: labels: app: istio-ingressgateway configPatches: - applyTo: HTTP_ROUTE match: context: GATEWAY routeConfiguration: vhost: name: url.com:80 patch: operation: INSERT_FIRST value: name: direct-healthz-response match: prefix: /healthz directResponse: body: inlineString: 'ok' status: 200
4 社区的进展
社区正在提议定义Dispatcher来完成基于path和header路由转发,底层还是基于ingress和virtualservice进行设计 。
设计原型:使用DomainMapping绑定域名和Dispatcher,Dispatcher里配置path和header路由转发,最终生成knative ingress。
相关issue:https://github.com/knative/serving/issues/11997
相关链接:
https://docs.google.com/document/d/1q3kkBhSqWMHm-TCFo56R-32C7-fo6G8KQH4lIGtc-U0/edit
https://github.com/knative/serving/issues/540
https://www.likakuli.com/posts/knative-pathfilter/
https://github.com/Kong/kubernetes-ingress-controller/issues/584
https://github.com/knative/docs/tree/main/code-samples/serving/knative-routing-go
5 总结
knative的ingress已经实现了基于路由转发规则,只是目前knative service还不支持。所以基于现在knative v1.1.0版本,实现绑定域名、基于路径和header转发的思路有两个。
直接操作virtualservice
优点:除了支持路径匹配和header匹配外,还支持路径正则匹配、域名正则匹配等其他功能
缺点: 只支持istio
操作knative ingress
优点:使用knative 原生资源,跟随社区兼容性好,可以使用任意实现。
缺点:只支持精确匹配 ,功能不够丰富。当然可以像k8s的ingress资源那样,具体实现通过label或annotation来扩展功能。