Ingress-Nginx进阶学习扩展实践

link之家

链接快照平台

输入网页链接，自动生成快照
标签化管理网页链接

相关文章推荐

自信的热水瓶 · DeadObjectException ...· 1 年前 ·

朝气蓬勃的黑框眼镜 · vue.js - Vue 3 ...· 2 年前 ·

爱逃课的墨镜 · 循环中的异步&&循环中的闭包-腾讯云开发者社 ...· 2 年前 ·

宽容的自行车 · TypeError:无法实例化抽象类-腾讯云 ...· 2 年前 ·

温柔的围巾 · NoNullAllowedException ...· 2 年前 ·

tee > test.yaml << 'EOF'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
spec:
  defaultBackend:
    service:
      name: test
      port:
        number: 80
kubectl apply -f

# 1.下载 v1.1.1 版本的ingress YAML资源清单到本地。
$ wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
# 2.为了加快国内环境拉取镜像的速度，此处将k8s.gcr.io替换为阿里云提供的镜像源。
$ sed -i -r 's#k8s.gcr.io/ingress-nginx/controller:(.*)@.*$#registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:\1#g' ingress-nginx-deploy.yaml
$ sed -i -r 's#k8s.gcr.io/ingress-nginx/(.*)@.*$#registry.cn-hangzhou.aliyuncs.com/google_containers/\1#g' ingress-nginx-deploy.yaml
#  或者使用skopeo同步到私有仓库
# ./skopeo copy --insecure-policy --src-tls-verify=false --dest-tls-verify=false --dest-authfile /root/.docker/config.json docker://registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.2.0 docker://harbor.weiyigeek.top/devops/nginx-ingress-controller:v1.2.0
# 3.在ingress-nginx命名空间中，部署 YAML 清单中ingress相关的控制器资源
$ kubectl apply -f ingress-nginx-deploy.yaml
# 4.指定条件等待Ingress相关资源部署完成以及查看namespace中部署ingress的相关Pod及其状态
$ kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s
$ kubectl get pods --namespace=ingress-nginx
# 5.创建的默认后端的资源名称一定要与ingress-nginx-controller中的 --default-backend-service 字段值相对应
  # 使用如下命令利用deployment 资源控制器快速创建default-backend 对象
$ kubectl create deployment default-backend --image=nginx:latest --namespace ingress-nginx
  # 为default-backend Pod 创建clusterip 类型的服务
$ kubectl create service clusterip default-backend --namespace ingress-nginx --tcp=80
  # 或者资源清单的方式
tee default-backend.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/path: /metrics
    prometheus.io/port: "80"
    prometheus.io/scheme: http
  labels:
    app: weiyigeek.top
    ref: default-backend
  name: default-backend
  namespace: ingress-nginx
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: weiyigeek.top
    ref: default-backend
  type: ClusterIP
# 6.查看Ingress的ClassName，此处名称为nginx
$ kubectl get ingressclasses.networking.k8s.io
  # NAME      CONTROLLER             PARAMETERS        AGE
  # nginx     k8s.io/ingress-nginx        <none>       10d
# 将ingress controler服务Pod运行在所有Work节点上，有几台机器就可以设置几个副本，当然master要去除污点后使用
~$ kubectl scale deployment -n ingress-nginx ingress-nginx-controller --replicas=4

# 方式1，创建一个名为demo-myweb-blog的入口，控制器名称为nginx，规则是将访问demo.weiyigeek.top请求转发到后端myweb-blog:80 服务之上
kubectl create ingress demo-myweb-blog --class=nginx --rule="demo.weiyigeek.top/=web-blog:80" --namespace devtest
# kubectl create ingress test-app -n devtest --rule="test.app.weiyigeek.top/oa(/|\$)(.*)=weiyigeek-oa-prod:8080" --annotation nginx.ingress.kubernetes.io/rewrite-target=/\$2
# 方式2.Ingress资源清单如下所示，其中重点是ingressClassName与rules字段
cat > www-myweb-blog.ingress <<'EOF'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: www-blog
  namespace: devtest
  annotations:
    kubernetes.io/ingress.class: "nginx"              # 方式1.根据该注解指定发现 Ingress
    ingressclass.kubernetes.io/is-default-class: true # 如未设置IngressClassName名称则采用默认Ingress
    nginx.ingress.kubernetes.io/ssl-redirect : false  # 禁用强制跳转
    nginx.ingress.kubernetes.io/rewrite-target: /
  labels:
    app: blog
spec:
  ingressClassName: nginx  # 方式2.根据该字段指定发现 Ingress
  defaultBackend:
    service:
      name: default-http
      port:
        number: 80
  rules:
  - host: "www.weiyigeek.top"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: www-blog
            port:
              number: 80
  - host: "web.weiyigeek.top"
    http:
      paths:
      - pathType: ImplementationSpecific
        path: "/"
        backend:
          service:
            name: web-blog
            port:
              number: 80 
  - hosts:
      - www.weiyigeek.top
    secretName: testsecret-tls   
# 部署资源清单
kubectl apply -f www-myweb-blog.ingress
# 查看上面部署的ingress入口信息及其规则默认后端信息。
kubectl get ingress -n devtest
kubectl describe ingress -n devtest
# 调用修改后的 Ingress yaml 文件。
kubectl replace -f www-myweb-blog.ingress
# 通过本机的hosts文件，手动将www.weiyigeek.top、demo.weiyigeek.top域名绑定到K8S中Node节点IP地址上，即可通过浏览器访问。
192.168.12.222 www.weiyigeek.top demo.weiyigeek.top

# 方式1.资源清单方式
apiVersion: v1
kind: Secret
metadata:
  name: www.weiyigeek.top
  namespace: devtest
data:
  tls.crt: `base64 编码的 cert`
  tls.key: `base64 编码的 key`
type: kubernetes.io/tls
# 方式2.kubectl 命令行方式
kubectl create secret tls www.weiyigeek.top --cert=server.crt --key=server.key -n devtest
# 使用示例:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-example-ingress
spec:
  - hosts:
      - www.weiyigeek.top 
    secretName: testsecret-tls

$ kubectl edit deployments.apps -n ingress-nginx ingress-nginx-controller
    spec:
      hostNetwork: true
      containers:
      - args:
        - /nginx-ingress-controller

~$ kubectl get pod -n ingress-nginx  -o wide | grep "ingress-nginx-controller"
ingress-nginx-controller-6b8bd48548-dfnvc   1/1     Running     0          4m26s   192.168.12.224   weiyigeek-224
ingress-nginx-controller-6b8bd48548-r9kpl   1/1     Running     0          5m      192.168.12.226   weiyigeek-226
ingress-nginx-controller-6b8bd48548-x9w9j   1/1     Running     0          5m33s   192.168.12.225   weiyigeek-225
ingress-nginx-controller-6b8bd48548-xqpcc   1/1     Running     0          5m33s   192.168.12.223   weiyigeek-223 
~$ curl 192.168.12.225
Hello ingress-Nginx , Index /

# 设置工作节点标签
~$ kubectl label nodes weiyigeek-223 weiyigeek-224 weiyigeek-225 weiyigeek-226 node=work
  # node/weiyigeek-223 labeled
  # node/weiyigeek-224 labeled
  # node/weiyigeek-225 labeled
  # node/weiyigeek-226 labeled
# 设置节点硬亲和、pod软亲和
~$ kubectl edit deployments.apps -n ingress-nginx ingress-nginx-controller
    spec: 
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node
                operator: In
                values:
                - work
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: ingress
                  operator: In
                  values:
                  - nginx
              topologyKey: kubernetes.io/hostname
            weight: 100

# 启用 RBAC 支持
$ helm install --name nginx-ingress --set "rbac.create=true,controller.service.externalIPs[0]=192.168.12.211,controller.service.externalIPs[1]=192.168.12.212,controller.service.externalIPs[2]=192.168.12.213" stable/nginx-ingress

$ kubectl get svc | grep "nginx-ingress-controller"
nginx-ingress-controller        LoadBalancer   10.254.84.72     192.168.12.211,192.168.12.212,192.168.12.213   80:8410/TCP,443:8948/TCP   46s

$ kubectl --namespace default get services -o wide -w nginx-ingress-controller

# 返回 200
$ curl -I  http://192.168.12.211
$ curl -I --insecure http://192.168.12.211

$ sudo netstat -tlunp|grep kube-proxy|grep -E '80|443'
tcp        0      0 192.168.12.211:80      0.0.0.0:*               LISTEN      714/kube-proxy
tcp        0      0 192.168.12.211:443     0.0.0.0:*               LISTEN      714/kube-proxy
$ sudo netstat -tlunp|grep kube-proxy|grep -E '80|443'
tcp        0      0 192.168.12.212:80      0.0.0.0:*               LISTEN      690/kube-proxy
tcp        0      0 192.168.12.212:443     0.0.0.0:*               LISTEN      690/kube-proxy
$ sudo netstat -tlunp|grep kube-proxy|grep -E '80|443'
tcp        0      0 192.168.12.213:80      0.0.0.0:*               LISTEN      748/kube-proxy
tcp        0      0 192.168.12.213:443     0.0.0.0:*               LISTEN      748/kube-proxy

cat '
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: rewrite
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: rewrite.weiyigeek.top
    http:
      paths:
      - path: /something(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: http-svc
            port: 
              number: 80
' | kubectl create -f -

# tcp-services-configmap.yaml
cat > tcp-services-configmap.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  2181: "kafka/kafka-zookeeper:2181"
  50000: "devops/jenkins:50000"
cat  tcp-services-configmap.yaml | kubectl create -f -
# udp-services-configmap.yaml
tee udp-services-configmap.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: udp-services
  namespace: ingress-nginx
data:
  53: "kube-system/kube-dns:53"
cat udp-services-configmap.yaml | kubectl create -f -

~$ kubectl get cm -n ingress-nginx  | grep "services"
tcp-services                2      15h
udp-services                1      15h

~$ telnet 192.168.12.225 50000
Trying 192.168.12.225...
Connected to 192.168.12.225.
Escape character is '^]'.
~$ dig -x jenkins.devops.svc @192.168.12.225

# ngress-nginx/ci/deployment-tcp-udp-values.yaml
controller:
  image:
    repository: ingress-controller/controller
    tag: 1.0.0-dev
    digest: null
  admissionWebhooks:
    enabled: false
  service:
    type: ClusterIP
  9000: "default/test:8080"
  9001: "default/test:8080"
# 执行更新
helm upgrade nginx-ingress stable/nginx-ingress \
-f ./ci/deployment-tcp-udp-values.yaml

sepc:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node
            operator: In
            values:
            - app
            - ali
            - www
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: kubernetes.io/os
              operator: In
              values:
              - linux
          topologyKey: kubernetes.io/hostname
        weight: 100
  nodeSelector:
    k8s.io/ingress-nginx: "true"

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: https://{server_domain}/auth/ldap/$remote_user/$http_authorization

# app.py
# -*- coding: utf-8 -*-
# description: ldap 服务用户账号密码验证
from ldap3 import Server, Connection, ALL, SUBTREE, ALL_ATTRIBUTES, MODIFY_REPLACE, MODIFY_ADD, MODIFY_DELETE
from ldap3.utils.hashed import hashed
from ldap3 import (
    HASHED_SALTED_SHA, MODIFY_REPLACE
import json,os
import base64
from flask import (
    Flask,
    Response,
    request,
    render_template,
    redirect,
    jsonify,
    make_response,
    url_for,
    abort,
from helpers import (
    status_code
app = Flask(__name__)
class LdapUtils(object):
    def __init__(self, ldap_host=None, port=None, base_dn=None, user=None, password=None):
        self.base_dn = base_dn
            server = Server(ldap_host, port, get_info=ALL)
            self.ldapconn = Connection(server, user=None, password=None,
                                       auto_bind='NONE', version=3, authentication='SIMPLE',
                                       client_strategy='SYNC',
                                       auto_referrals=True, check_names=True, read_only=False, lazy=False,
                                       raise_exceptions=False)
            self.ldapconn.rebind(user=user, password=password)
        except Exception as e:
            print(e)
    def ldap_search_dn(self, uid=None):
        obj = self.ldapconn
        search_base = self.base_dn
        search_scope = SUBTREE
        search_filter = "(cn={0})".format(uid)
            obj.search(search_base, search_filter, search_scope, attributes=['cn'], paged_size=1)
            if len(obj.response) == 1:
                return obj.response[0]['dn']
            else:
                return None
        except Exception as e:
            print(e)
    def ldap_update_pass(self, uid=None, oldpass=None, newpass=None):
        target_cn = self.ldap_search_dn(uid)
            hashed_password = hashed(HASHED_SALTED_SHA, newpass)
            print("password:" + hashed_password)
            changes = {
                'userPassword': [(MODIFY_REPLACE, [hashed_password])]
            return self.ldapconn.modify(target_cn, changes=changes)
        except Exception as e:
            print(e)
            return False
    def ldap_get_vaild(self, uid=None, passwd=None):
        if not uid or not passwd:
            return False
        obj = self.ldapconn
        # 这里注意修改成自己ldap上定义的user dn 
        dn = "uid={0},ou=Users,dc=xxxxx,dc=com".format(uid)
            if obj.rebind(dn, passwd):
                return True
            else:
                return False
        except Exception as e:
            print('e:' + str(e))
# 这里我是通过docker部署服务，这些配置在环境变量传入，你可以根据需要选择直接填写对应值
ldap_host = os.getenv("host")
port = os.getenv("port")
base_dn = os.getenv("base_dn")
user = os.getenv("admin_user")
password = os.getenv("password")
ldap = LdapUtils(ldap_host, int(port), base_dn, user,password)  
@app.route('/auth/<user>/<passwd>')
def auth_user(user="user", passwd="passwd"):
    if not ldap.ldap_get_vaild(uid=user,passwd=passwd):
        return status_code(401)
    return jsonify(authenticated=True, user=user)
@app.route('/auth/<qop>/<user>/<passwd>')
def auth_qop_user(qop=None, user="user", passwd="passwd"):
    temp = base64.b64decode(passwd.split(" ")[1]).decode()
    if not ldap.ldap_get_vaild(uid=user,passwd=temp.split(":")[1]):
        return status_code(401)
    return jsonify(authenticated=True, user=user)
@app.route('/', defaults={'path': ''})
@app.route('/<path:path>')
def catch_all(path):
    auth = request.headers.get('Authorization')
    if auth is not None:
        uid = base64.b64decode(auth.split(" ")[1]).decode().split(":")[0]
        passwd = base64.b64decode(auth.split(" ")[1]).decode().split(":")[1]
        if not ldap.ldap_get_vaild(uid=uid,passwd=passwd):
            return status_code(401)
        return render_template('password.html',uid=uid)
    return status_code(401)
if __name__ == '__main__':
   app.run(host = '0.0.0.0')

# 问题补充：
# 用下面的命令进入 nginx-ingress 容器
kubectl exec -it daemonset/nginx-ingress -n nginx-ingress cat /etc/nginx/conf.d/production-cnblogs-ingress.conf
# 发现问题是下面的配置引起的
proxy_set_header X-Forwarded-Proto $scheme;

Annotation: nginx.org/redirect-to-https
ConfigMap Key redirect-to-https
Description Sets the 301 redirect rule based on the value of the http_x_forwarded_proto header on the server block to force incoming traffic to be over HTTPS. Useful when terminating SSL in a load balancer in front of the Ingress controller
# 根据服务器块上http_x_forwarded_proto报头的值设置301重定向规则，以强制传入的流量使用HTTPS。在入口控制器前的负载均衡器中终止SSL时非常有用;

apiVersion: extensions/v1beta1 
kind: Ingress
metadata:
  name: cnblogs-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/redirect-to-https: "true"
spec:
  rules:
  - host: q.cnblogs.com
    http:
      paths:
        - backend:
            serviceName: q-web
            servicePort: 80

kubectl apply -f cnblogs-ingress.yaml

kubectl rollout restart daemonset/nginx-ingress -n nginx-ingress && \
kubectl rollout status daemonset/nginx-ingress -n nginx-ingress

kubectl exec -it daemonset/nginx-ingress -n nginx-ingress cat /etc/nginx/conf.d/production-cnblogs-ingress.conf
# 发现 proxy_set_header X-Forwarded-Proto $scheme; 变成了 proxy_set_header X-Forwarded-Proto https; ，并且增加了下面的 http 重定向 https 的配置。
if ($http_x_forwarded_proto = 'http') {
    return 301 https://$host$request_uri;
}

~/K8s/Day7/demo1$ kubectl api-resources | grep "ingresses"
ingresses                         ing          extensions                     true         Ingress
ingresses                         ing          networking.k8s.io              true         Ingress
# Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
apiVersion: networking.k8s.io/v1 # 注意点否则将报上面的预警

$ kubectl apply -f demo-ingress.yaml
# 错误1
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"kubernetes.io/ingress.class\":\"nginx\"},\"name\":\"nginx\",\"namespace\":\"default\"},\"spec\":{\"rules\":[{\"host\":\"nginx.ieasou.cn\",\"http\":{\"paths\":[{\"backend\":{\"service\":{\"name\":\"nginx\",\"port\":{\"number\":80}}},\"path\":\"/\",\"pathType\":\"Prefix\"}]}}]}}\n","kubernetes.io/ingress.class":"nginx"}},"spec":{"rules":[{"host":"nginx.ieasou.cn","http":{"paths":[{"backend":{"service":{"name":"nginx","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}
Resource: "networking.k8s.io/v1, Resource=ingresses", GroupVersionKind: "networking.k8s.io/v1, Kind=Ingress"
Name: "nginx", Namespace: "default"
for: "demo-ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s": x509: certificate is valid for k8s-master002, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not 
ingress-nginx-controller-admission.ingress-nginx.svc
# 错误2
kubectl create -f ingress-nginx-http-v1.yaml
  # Error from server (InternalError): error when creating "ingress-nginx-http-v1.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": an error on the server ("") has prevented the request from succeeding

weiyigeek@ubuntu:~/K8s/Day7/demo2$ kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
  # validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-admission" deleted
weiyigeek@ubuntu:~/K8s/Day7/demo2$ kubectl create -f ingress-nginx-http-v1.yaml  
  # ingress.networking.k8s.io/nginx-ingress-http created  # 未其他报错正常创建

$ kubectl describe ing
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)

# Ingress 控制器
spec:
  defaultbackend:
    service:
      name: nginx-ingress-v1-svc
      port:
        number: 80

# 1.查看部署的ingress-nginx控制器名称k8s.io/ingress-nginx。
~$ kubectl get deployments.apps -n ingress-nginx ingress-nginx-controller -o yaml | grep " --controller-class"
        - --controller-class=k8s.io/ingress-nginx
# 2.查看ingressclass的Name信息
~$ kubectl get ingressclasses.networking.k8s.io
  # NAME    CONTROLLER             PARAMETERS   AGE
  # nginx   k8s.io/ingress-nginx   <none>       12d
# 3.编辑ingressclass
~$ kubectl get ingressclasses.networking.k8s.io
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 1.1.1
    helm.sh/chart: ingress-nginx-4.0.15
spec:
  controller: k8s.io/ingress-nginx
  parameters:
    apiGroup: k8s.example.com
    kind: IngressParameters
    name: external-lb
    namespace: external-configuration
    scope: Namespace
# 4.为ingress规则指定默认的ingresclass的名称。
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blog
  namespace: devtest
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    # kubernetes.io/ingress.class: nginx         #  当使用多个 Ingress 控制器时进行配置，如果不定义 ingress.class，云提供商可能使用默认的 Ingress 控制器。
    ingressclass.kubernetes.io/is-default-class: true
spec:
  ingressClassName: nginx                        #  推荐
  rules:
  - host: "www.weiyigeek.top"
    http:
      paths:
      - pathType: Prefix

WeiyiGeek

Ingress-Nginx进阶学习扩展实践

0x00 Ingress-Nginx 快速安装配置实践

1.快速安装

2.服务验证

3.配置虚拟主机证书

4.使用hostNetwork网络方式

5.使用externalIP网络方式

6.对外暴露端口配置

7.节点与Pod亲和性设置

8.基于 auth-url 模块认证方式实践

0x01 入坑出坑

问题1.K8s Nginx Ingress Controller 转发 X-Forwarded-Proto 请求头的问题

问题2:extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

问题3.Error from server (InternalError): error when applying patch:

问题4.查看ingress规则时提示 kubernetes之ingress error: endpoints "default-http-backend" not found 警告

问题5.设置了ingress无法正常显示页面，由于没有在资源清单中指定设置ingressclass名称