k8s部署rediscluster集群的实现

⽬录

Redis 介绍为什么要⽤Redis

什么是Redis Cluster集群

k8s以StatefulSet⽅式部署redis cluster集群：部署nfs创建pv部署redis初始化redis集群

Redis 介绍

Redis代表REmote DIctionary Server是⼀种开源的内存中数据存储，通常⽤作数据库，缓存或消息代理。它可以存储和操作⾼级数据类型，例如列表，地图，集合和排序集合。

由于Redis接受多种格式的密钥，因此可以在服务器上执⾏操作，从⽽减少了客户端的⼯作量。它仅将磁盘⽤于持久性，⽽将数据完全保存在内存中。

Redis是⼀种流⾏的数据存储解决⽅案，并被GitHub，Pinterest，Snapchat，Twitter，StackOverflow，Flickr等技术巨头所使⽤。

为什么要⽤Redis

它的速度⾮常快。它是⽤ANSI C编写的，并且可以在POSIX系统上运⾏，例如Linux，Mac OS X和Solaris。Redis通常被排名为最流⾏的键/值数据库和最流⾏的与容器⼀起使⽤的NoSQL数据库。其缓存解决⽅案减少了对云数据库后端的调⽤次数。应⽤程序可以通过其客户端API库对其进⾏访问。所有流⾏的编程语⾔都⽀持Redis。它是开源且稳定的。

什么是Redis Cluster集群

Redis Cluster是⼀组Redis实例，旨在通过对数据库进⾏分区来扩展数据库，从⽽使其更具弹性。

群集中的每个成员（⽆论是主副本还是辅助副本）都管理哈希槽的⼦集。如果主机⽆法访问，则其从机将升级为主机。在由三个主节点组成的最⼩Redis群集中，每个主节点都有⼀个从节点（以实现最⼩的故障转移），每个主节点都分配有⼀个介于0到16,383之间的哈希槽范围。节点A包含从0到5000的哈希槽，节点B从5001到10000，节点C从10001到16383。群集内部的通信是通过内部总线进⾏的，使⽤协议传播有关群集的信息或发现新节点。

k8s以StatefulSet⽅式部署redis cluster集群：

1. 部署nfs2. 创建pv3. 部署redis4. 初始化redis集群主机说明：

系统

ip

⾓⾊cpu内存hostname

CentOS 7.8192.168.30.128master>=2>=2Gmaster1CentOS 7.8192.168.30.129master>=2>=2Gmaster2CentOS 7.8192.168.30.130nodeCentOS 7.8192.168.30.131nodeCentOS 7.8192.168.30.132node

kubectl get node

NAME STATUS ROLES AGE VERSIONmaster1 Ready master 33d v1.14.0master2 Ready master 33d v1.14.0

>=2>=2Gnode1>=2>=2Gnode2>=2>=2Gnode3

node1 Ready 33d v1.14.0node2 Ready 33d v1.14.0node3 Ready 33d v1.14.0

部署nfs

在master2节点上做nfs共享，

yum -y install nfs-utils rpcbind

mkdir -p /data/redis/{cluster0,cluster1,cluster2,cluster3,cluster4,cluster5}vim /etc/exports

/data/redis/cluster0 192.168.30.0/24(rw,sync,no_root_squash)/data/redis/cluster1 192.168.30.0/24(rw,sync,no_root_squash)/data/redis/cluster2 192.168.30.0/24(rw,sync,no_root_squash)/data/redis/cluster3 192.168.30.0/24(rw,sync,no_root_squash)/data/redis/cluster4 192.168.30.0/24(rw,sync,no_root_squash)/data/redis/cluster5 192.168.30.0/24(rw,sync,no_root_squash)chmod -R 755 /data/redisexportfs -arv

systemctl enable rpcbind && systemctl start rpcbindsystemctl enable nfs && systemctl start nfs

nfs部署完毕。对于需要使⽤nfs的node节点，都要安装nfs：

yum -y install nfs-utils

创建pv

创建pv：

mkdir -p /home/k8s/redis && cd /home/k8s/redisvim pv.yaml

apiVersion: v1

kind: PersistentVolumemetadata:

name: nfs-pv0spec:

capacity:

storage: 1Gi accessModes: - ReadWriteMany nfs:

server: 192.168.30.129 path: /data/redis/cluster0---apiVersion: v1

kind: PersistentVolumemetadata:

name: nfs-pv1spec:

capacity:

storage: 1Gi accessModes: - ReadWriteMany nfs:

server: 192.168.30.129 path: /data/redis/cluster1---apiVersion: v1

kind: PersistentVolumemetadata:

name: nfs-pv2spec:

capacity:

storage: 1Gi accessModes: - ReadWriteMany nfs:

server: 192.168.30.129 path: /data/redis/cluster2---

apiVersion: v1

kind: PersistentVolumemetadata:

name: nfs-pv3spec:

capacity:

storage: 1Gi accessModes: - ReadWriteMany nfs:

server: 192.168.30.129 path: /data/redis/cluster3---apiVersion: v1

kind: PersistentVolumemetadata:

name: nfs-pv4spec:

capacity:

storage: 1Gi accessModes: - ReadWriteMany nfs:

server: 192.168.30.129 path: /data/redis/cluster4---apiVersion: v1

kind: PersistentVolumemetadata:

name: nfs-pv5spec:

capacity:

storage: 1Gi accessModes: - ReadWriteMany nfs:

server: 192.168.30.129 path: /data/redis/cluster5kubectl apply -f pv.yamlkubectl get pv

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGEnfs-pv0 1Gi RWX Retain Available 3snfs-pv1 1Gi RWX Retain Available 3snfs-pv2 1Gi RWX Retain Available 3snfs-pv3 1Gi RWX Retain Available 3snfs-pv4 1Gi RWX Retain Available 3snfs-pv5 1Gi RWX Retain Available 3s

pv创建完毕。

部署redis

vim namespace.yamlapiVersion: v1kind: Namespacemetadata:

name: public-servicevim redis.conf

appendonly yescluster-enabled yes

cluster-config-file /var/lib/redis/nodes.confcluster-node-timeout 5000dir /var/lib/redisport 6379vim redis.yamlapiVersion: v1kind: Servicemetadata:

name: redis

namespace: public-service labels:

app: redisspec:

selector: app: redis

appCluster: redis-cluster ports:

- name: redis port: 6379 clusterIP: None ---apiVersion: v1kind: Servicemetadata:

name: redis-access

namespace: public-service labels:

app: redisspec:

selector: app: redis

appCluster: redis-cluster ports:

- name: redis-access protocol: TCP port: 6379

targetPort: 6379

---apiVersion: apps/v1kind: StatefulSetmetadata: name: redis

namespace: public-servicespec:

serviceName: redis replicas: 6 selector:

matchLabels: app: redis

appCluster: redis-cluster template: metadata: labels:

app: redis

appCluster: redis-cluster spec:

terminationGracePeriodSeconds: 20 affinity:

podAntiAffinity:

preferredDuringSchedulingIgnoredDuringExecution: - weight: 100

podAffinityTerm: labelSelector:

matchExpressions: - key: app operator: In values: - redis

topologyKey: kubernetes.io/hostname containers: - name: redis

image: redis:latest command:

- \"redis-server\" args:

- \"/etc/redis/redis.conf\" - \"--protected-mode\" - \"no\" resources: requests: cpu: \"500m\"

memory: \"500Mi\" ports:

- containerPort: 6379 name: redis

protocol: TCP

- containerPort: 16379 name: cluster protocol: TCP volumeMounts: - name: conf

mountPath: /etc/redis - name: data

mountPath: /var/lib/redis volumes: - name: conf configMap:

name: redis-conf items:

- key: redis.conf path: redis.conf volumeClaimTemplates: - metadata: name: data

namespace: public-service spec:

accessModes: [ \"ReadWriteMany\" ] resources: requests: storage: 1Gi

kubectl apply -f namespace.yaml

kubectl create configmap redis-conf --from-file=redis.conf -n public-servicekubectl apply -f redis.yamlkubectl get svc -n public-service

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEredis ClusterIP None 6379/TCP 20s

redis-access ClusterIP 10.96.2.100 6379/TCP 20skubectl get pod -n public-service

NAME READY STATUS RESTARTS AGEredis-0 1/1 Running 0 2m43sredis-1 1/1 Running 0 2m18sredis-2 1/1 Running 0 108sredis-3 1/1 Running 0 80sredis-4 1/1 Running 0 48sredis-5 1/1 Running 0 30skubectl get pvc -n public-service

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGEdata-redis-0 Bound nfs-pv4 1Gi RWX 3m4sdata-redis-1 Bound nfs-pv0 1Gi RWX 2m39sdata-redis-2 Bound nfs-pv1 1Gi RWX 2m9sdata-redis-3 Bound nfs-pv2 1Gi RWX 101sdata-redis-4 Bound nfs-pv3 1Gi RWX 69sdata-redis-5 Bound nfs-pv5 1Gi RWX 51s

redis-access这个service⽅便集群内访问redis集群，redis部署完毕。

初始化redis集群

使⽤Redis-tribe⼯具进⾏集群的初始化。下载redis-tribe：

kubectl run -it ubuntu --image=ubuntu --restart=Never -n public-service bash

root@ubuntu:/# cat > /etc/apt/sources.list << EOF

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial main restricted

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-updates main restricteddeb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial universe

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-updates universedeb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial multiverse

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-updates multiverse

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-backports main restricted universe multiversedeb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-security main restricteddeb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-security universe

deb http://mirrors.tuna.tsinghua.edu.cn/ubuntu/ xenial-security multiverseEOF

root@ubuntu:/# apt-get update

root@ubuntu:/# apt-get install -y libncursesw5 libreadline6 libtinfo5 --allow-remove-essentialroot@ubuntu:/# apt-get install -y libpython2.7-stdlib python2.7 python-pip redis-tools dnsutilsroot@ubuntu:/# pip install --upgrade piproot@ubuntu:/# pip install redis-trib==0.5.1

初始化集群：

root@ubuntu:/# redis-trib.py create \\

`dig +short redis-0.redis.public-service.svc.cluster.local`:6379 \\ `dig +short redis-1.redis.public-service.svc.cluster.local`:6379 \\ `dig +short redis-2.redis.public-service.svc.cluster.local`:6379

root@ubuntu:/# redis-trib.py replicate \\

--master-addr `dig +short redis-0.redis.public-service.svc.cluster.local`:6379 \\ --slave-addr `dig +short redis-3.redis.public-service.svc.cluster.local`:6379root@ubuntu:/# redis-trib.py replicate \\

--master-addr `dig +short redis-1.redis.public-service.svc.cluster.local`:6379 \\ --slave-addr `dig +short redis-4.redis.public-service.svc.cluster.local`:6379root@ubuntu:/# redis-trib.py replicate \\

--master-addr `dig +short redis-2.redis.public-service.svc.cluster.local`:6379 \\ --slave-addr `dig +short redis-5.redis.public-service.svc.cluster.local`:6379

root@ubuntu:/# exit

查看集群：

kubectl exec -it -n public-service redis-0 bashroot@redis-0:/data# redis-cli -c

127.0.0.1:6379> CLUSTER NODES #列出节点信息

aac2b3d320da67eedf3512ed0e38a1cdce5bc8fe 172.10.2.55:6379@16379 slave 7c4d60cf32685484ea6c5cb4493a937dfbf6b8a5 0 1592276224727 3 connected2efad514b2f3c7fe4530dd6dc63c0df8ffdb793d 172.10.2.54:6379@16379 master - 0 1592276224224 1 connected 0-5461

524f03526a4b683d7d4de19296431810bfdc22cf 172.10.3.60:6379@16379 slave df5bc3c2e2851d63cdb9f762efde6e1b0d38efed 0 1592276223117 5 connected7c4d60cf32685484ea6c5cb4493a937dfbf6b8a5 172.10.4.77:6379@16379 myself,master - 0 1592276224000 2 connected 5462-10922df5bc3c2e2851d63cdb9f762efde6e1b0d38efed 172.10.3.59:6379@16379 master - 0 1592276223217 0 connected 10923-16383

c1dbaaef4a583e372c43eed52c22cd9ad7184d18 172.10.4.78:6379@16379 slave 2efad514b2f3c7fe4530dd6dc63c0df8ffdb793d 0 1592276223719 4 connected127.0.0.1:6379> CLUSTER INFO #集群状态cluster_state:ok

cluster_slots_assigned:16384cluster_slots_ok:16384cluster_slots_pfail:0cluster_slots_fail:0

cluster_known_nodes:6cluster_size:3

cluster_current_epoch:5cluster_my_epoch:2

cluster_stats_messages_ping_sent:514cluster_stats_messages_pong_sent:491cluster_stats_messages_meet_sent:2cluster_stats_messages_sent:1007

cluster_stats_messages_ping_received:491cluster_stats_messages_pong_received:516cluster_stats_messages_received:1007

redis集群初始化完成，已经形成3主3从的cluster集群。写⼊数据：

127.0.0.1:6379> SET key1 aaaOK

127.0.0.1:6379> SET key2 bbb

-> Redirected to slot [4998] located at 172.10.2.54:6379OK

172.10.2.54:6379> SET key3 cccOK

kubectl exec -it -n public-service redis-4 bashroot@redis-4:/data# redis-cli -c

127.0.0.1:6379> GET key1

-> Redirected to slot [9189] located at 172.10.4.77:6379\"aaa\"

172.10.4.77:6379> GET key2

-> Redirected to slot [4998] located at 172.10.2.54:6379\"bbb\"

172.10.2.54:6379> GET key3\"ccc\"

可以看出redis cluster集群是去中⼼化的，每个节点都是平等的，连接哪个节点都可以获取和设置数据。主从切换：

选择redis-2模拟宕掉，测试主从切换，

kubectl get pod -n public-service -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESredis-0 1/1 Running 0 62m 172.10.4.77 node1 redis-1 1/1 Running 0 62m 172.10.2.54 node2 redis-2 1/1 Running 0 61m 172.10.3.59 node3 redis-3 1/1 Running 0 61m 172.10.2.55 node2 redis-4 1/1 Running 0 61m 172.10.4.78 node1 redis-5 1/1 Running 0 61m 172.10.3.60 node3 ubuntu 0/1 Completed 0 57m 172.10.2.56 node2 kubectl exec -it -n public-service redis-2 bashroot@redis-2:/data# redis-cli -c127.0.0.1:6379> ROLE1) \"master\"

2) (integer) 2898

3) 1) 1) \"172.10.3.60\" 2) \"6379\" 3) \"2898\"

可以看到，redis-2是master，它的slave是172.10.3.60，即redis-5。

kubectl delete pod -n public-service redis-2 #模拟节点宕掉kubectl get pod -n public-service redis-2 -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESredis-2 1/1 Running 0 38s 172.10.3.61 node3 kubectl exec -it -n public-service redis-2 bashroot@redis-2:/data# redis-cli -c127.0.0.1:6379> ROLE1) \"slave\"

2) \"172.10.3.60\"3) (integer) 63794) \"connected\"5) (integer) 3430

kubectl exec -it -n public-service redis-5 bashroot@redis-5:/data# redis-cli -c127.0.0.1:6379> ROLE1) \"master\"

2) (integer) 3584

3) 1) 1) \"172.10.3.61\" 2) \"6379\" 3) \"3570\"

127.0.0.1:6379> CLUSTER NODES

aac2b3d320da67eedf3512ed0e38a1cdce5bc8fe 172.10.2.55:6379@16379 slave 7c4d60cf32685484ea6c5cb4493a937dfbf6b8a5 0 1592278859530 2 connected2efad514b2f3c7fe4530dd6dc63c0df8ffdb793d 172.10.2.54:6379@16379 master - 0 1592278859000 1 connected 0-5461

c1dbaaef4a583e372c43eed52c22cd9ad7184d18 172.10.4.78:6379@16379 slave 2efad514b2f3c7fe4530dd6dc63c0df8ffdb793d 0 1592278859000 1 connected524f03526a4b683d7d4de19296431810bfdc22cf 172.10.3.60:6379@16379 myself,master - 0 1592278857000 6 connected 10923-163837c4d60cf32685484ea6c5cb4493a937dfbf6b8a5 172.10.4.77:6379@16379 master - 0 1592278858021 2 connected 5462-10922

df5bc3c2e2851d63cdb9f762efde6e1b0d38efed 172.10.3.61:6379@16379 slave 524f03526a4b683d7d4de19296431810bfdc22cf 0 1592278859000 6 connected

可以看到，redis-2在重启之后变为slave，⽽它之前的slave——redis-5变为master，⽽且是新redis-2的master。集群的主从切换没有问题。k8s部署redis cluster集群完成。

到此这篇关于k8s部署redis cluster集群的实现的⽂章就介绍到这了,更多相关k8s部署redis cluster内容请搜索以前的⽂章或继续浏览下⾯的相关⽂章希望⼤家以后多多⽀持！