部署k3s、使用kubectl安装portainer

K3s 是一个高可用的、经过 CNCF 认证的轻量级Kubernetes发行版，专为物联网及边缘计算设计。

k3s 将安装 Kubernetes 所需的一切打包进仅有 60MB 大小的二进制文件中，并且完全实现了 Kubernetes API。为了减少运行 Kubernetes 所需的内存，k3s 删除了很多不必要的驱动程序，并用附加组件对其进行替换。由于它只需要极低的资源就可以运行，因此它能够在任何 512MB 内存以上的设备上运行集群。

前提

安装Docker （略）可看我其它文章或者自行百度

一、安装

1、国内源安装

curl –sfL \
     https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | \
     INSTALL_K3S_MIRROR=cn sh -s - \
     --system-default-registry "registry.cn-hangzhou.aliyuncs.com" \
     --write-kubeconfig ~/.kube/config \
     --write-kubeconfig-mode 666 \
     --disable traefik

... 日志
[INFO]  systemd: Starting k3s

出现systemd: Starting k3s就完成了
如果你的带宽充足，一分钟之内即可完成 K3s 的安装和系统服务的启动，然后可以通过执行kubectl get pods -n kube-system 来验证

2、验证

kubectl get pods -n kube-system

...
NAME                                      READY   STATUS    RESTARTS   AGE
local-path-provisioner-858c864885-vktw4   1/1     Running   0          89s
coredns-6c4b5c5567-c77wf                  1/1     Running   0          89s
metrics-server-5d7f9fc86-f6hhh            1/1     Running   0          89s

3、配置 containerd 的 mirror

以上这些步骤只是为了加速 K3s 的安装和启动。启动 K3s 后你可能会在 K3s 上部署自己的业务(例如 nginx)，而这些镜像默认也是从 DockerHub 拉取。如果使用 docker 容器运行时，你可能会在 docker 上配置 mirror 来加速镜像的拉取。

K3s 默认使用的containerd容器运行时。而且，可以通过 K3s 的参数来设置 containerd 的 mirror，设置方式如下:

cat > /etc/rancher/k3s/registries.yaml <<EOF
mirrors:
  docker.io:
    endpoint:
      - "http://hub-mirror.c.163.com"
      - "https://docker.mirrors.ustc.edu.cn"
      - "https://registry.docker-cn.com"
EOF


systemctl restart k3s

4、验证k3s

# 查看集群运行情况
kubectl get pod --all-namespaces
kubectl get nodes -o wide
kubectl get all -A -owide
kubectl describe node vm01
kubectl -n kube-system describe deploy coredns
kubectl config view

k3s kubectl get node

指定config

方案一：通过KUBECONFIG环境变量指定kubeconfig文件:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

方案二：在命令中指定 kubeconfig 文件的位置

kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml

注意: 此命令允许访问 kubeconfig(默认路径:/etc/rancher/k3s/k3s.yaml)。例如,当使用 Kubectl 时。

5、Traefik

安装完成后K3S默认使用Traefik作为网络反向代理应用，作为一种IngressController资源提供给Ingress资源使用，其中需要给IngressController配置IngressClass来设置其身份标识，尽管K3S中默认只有Traefik这一个IngressController。创建traefik-ingress-class文件，内容参考如下

# traefik-ingress-class.yaml
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: traefik
  annotations:
    ingressclass.kubernetes.io/is-default-class: "true"
spec:
  controller: traefik.io/ingress-controller

在K8S中所有的资源都可以通过apply指定参数来创建，也可以通过apply来修改和删除资源。

kubectl apply -f traefik-ingress-class.yaml

K8S默认限制NodePort范围，可以修改启动命令设置端口范围

systemctl stop k3s
vim /etc/systemd/system/k3s.service  # 修改配置，保存并退出
systemctl daemon-reload
systemctl start k3s

修改内容最后添加一行参数指定--kube-apiserver-arg service-node-port-range=1-65535

ExecStart=/usr/local/bin/k3s \
    server \
    --kube-apiserver-arg service-node-port-range=1-65535

6、https的配置和使用

如K3S平台有公网访问需求和https的需求，可以通过cert-manager、Let’s Encrypt和Trafike来自动申请和管理https证书。

6.1 证书配置

# 安装cert-manager
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

# 查看pod安装进度
kubectl get po -n cert-manager

创建签发机构资源对象，配置证书信息（https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml）

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: https-prod
spec:
  acme:
    email: cctomato@outlook.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: https-prod
    solvers:
    - http01:
        ingress:
          class: traefik

6.2 创建https跳转网关

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: redirect-https
spec:
  redirectScheme:
    scheme: https
    permanent: true

7、 Portainer使用

部署一个Portainer应用来测试https证书。在内网部署应用不多的情况下，推荐使用Portainer。

kubectl apply -n portainer -f https://downloads.portainer.io/ce2-15/portainer.yaml

# portainer.yaml
---
# Source: portainer/templates/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: portainer
---
# Source: portainer/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: portainer-sa-clusteradmin
  namespace: portainer
  labels:
    app.kubernetes.io/name: portainer
    app.kubernetes.io/instance: portainer
    app.kubernetes.io/version: "ce-latest-ee-2.19.1"
---
# Source: portainer/templates/pvc.yaml
kind: "PersistentVolumeClaim"
apiVersion: "v1"
metadata:
  name: portainer
  namespace: portainer  
  annotations:
    volume.alpha.kubernetes.io/storage-class: "generic"
  labels:
    io.portainer.kubernetes.application.stack: portainer
    app.kubernetes.io/name: portainer
    app.kubernetes.io/instance: portainer
    app.kubernetes.io/version: "ce-latest-ee-2.19.1"
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: "10Gi"
---
# Source: portainer/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: portainer
  labels:
    app.kubernetes.io/name: portainer
    app.kubernetes.io/instance: portainer
    app.kubernetes.io/version: "ce-latest-ee-2.19.1"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  namespace: portainer
  name: portainer-sa-clusteradmin
---
# Source: portainer/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: portainer
  namespace: portainer
  labels:
    io.portainer.kubernetes.application.stack: portainer
    app.kubernetes.io/name: portainer
    app.kubernetes.io/instance: portainer
    app.kubernetes.io/version: "ce-latest-ee-2.19.1"
spec:
  type: NodePort
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
      name: http
      nodePort: 10000
    - port: 9443
      targetPort: 9443
      protocol: TCP
      name: https
      nodePort: 30779      
    - port: 30776
      targetPort: 30776
      protocol: TCP
      name: edge
      nodePort: 30776
  selector:
    app.kubernetes.io/name: portainer
    app.kubernetes.io/instance: portainer
---
# Source: portainer/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: portainer
  namespace: portainer
  labels:
    io.portainer.kubernetes.application.stack: portainer
    app.kubernetes.io/name: portainer
    app.kubernetes.io/instance: portainer
    app.kubernetes.io/version: "ce-latest-ee-2.19.1"
spec:
  replicas: 1
  strategy:
    type: "Recreate"
  selector:
    matchLabels:
      app.kubernetes.io/name: portainer
      app.kubernetes.io/instance: portainer
  template:
    metadata:
      labels:
        app.kubernetes.io/name: portainer
        app.kubernetes.io/instance: portainer
    spec:
      nodeSelector:
        {}
      serviceAccountName: portainer-sa-clusteradmin
      volumes:
        - name: "data"
          persistentVolumeClaim:
            claimName: portainer
      containers:
        - name: portainer
          image: "summary/portainer-ce:2.19.1"
          imagePullPolicy: Always
          args:
          - '--tunnel-port=30776'          
          volumeMounts:
            - name: data
              mountPath: /data              
          ports:
            - name: http
              containerPort: 9000
              protocol: TCP
            - name: https
              containerPort: 9443
              protocol: TCP              
            - name: tcp-edge
              containerPort: 8000
              protocol: TCP              
          livenessProbe:
            httpGet:
              path: /
              port: 9443
              scheme: HTTPS
          readinessProbe:
            httpGet:
              path: /
              port: 9443
              scheme: HTTPS
          resources:
            {}
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.portainer.io/ingress-type: traefik
    cert-manager.io/cluster-issuer: https-prod
    traefik.ingress.kubernetes.io/router.middlewares: default-redirect-https@kubernetescrd
  name: portainer
  namespace: portainer
spec:
  ingressClassName: traefik
  rules:
  - host: portainer.dev.cctomato.com
    http:
      paths:
      - backend:
          service:
            name: portainer
            port:
              number: 9000
        path: /
        pathType: ImplementationSpecific
  tls:
    - secretName: portainer-https
      hosts:
        - portainer.dev.cctomato.com

Portainer中部署不支持设置HostPath的本地目录设置，所以只能手动编辑文件来配置。

kubectl edit deploy/portainer -n portainer

#在containers同级声明要挂载的目录
volumes:                                    
- hostPath:                    
    path: /alidata/dev/gitea/data          
    type: Directory                           
  name: gitea-data
#在containers下级具体的容器设置中声明挂载的路径
volumeMounts:                                
- mountPath: /data                    
  name: gitea-data

Portainer已经支持OAuth登录，可以在Settings-Authentication中选择OAuth-Custom配置。使用前需要先在portainer中创建用户。

Authorization URL	https://git.dev.cctomato.com/login/oauth/authorize
Access token URL	https://git.dev.cctomato.com/login/oauth/access_token
Resource URL	https://git.dev.cctomato.com/api/v1/user
Redirect URL	https://portainer.dev.cctomato.com/
User identifier	username

9、卸载k3s

# 一键卸载 server
k3s-uninstall.sh

# 一键卸载 agent
k3s-agent-uninstall.sh

或shell

# 一键卸载 server
/usr/local/bin/k3s-uninstall.sh

# 一键卸载 agent
/usr/local/bin/k3s-agent-uninstall.sh