redis k8s kubernetes

如何在k8s上搭建redis cluster

Bob Lien 2020/09/28 17:08:40
10907

本文將在已建立的k8s集群上建立redis集群，並且將redis集群資料、集群設定持久化在file server上。

昕力大學已經有文章介紹過redis和k8s，以下提供連結參考

    ⋅ 如何於Kubernetes上佈建Apache Kafka Cluster運行環境

    ⋅ Redis實作

redis cluster

    優點:

       1. 無中心架構。

       2. 可擴展性。

       3. 高可用性。

    缺點

        1. Key 批量操作限制。

       2. Key 事務操作限制。   

        3. 不支持多資料庫空間。

環境準備

說明 OS ip domain
k8s cluster master node

CentOS 7

 192.168.43.113 k8s-master-node-01
k8s cluster worker node 1 CentOS 7 192.168.43.25 k8s-worker-node-01
k8s cluster worker node 2 CentOS 7 192.168.43.250 k8s-worker-node-02
k8s cluster worker node 3 CentOS 7 192.168.43.195 k8s-worker-node-03
nfs file server CentOS 7 192.168.43.205  
harbor(測試用) CentOS 7 192.168.43.117 k8s-harbor.com

Step1. 創建 redis-cluster namespace

    本文將redis資源建立在redis-cluster namespace下

kubectl create namespace redis-cluster

Step2. 創建 ConfigMap

    redis 設定檔

    說明：

    appendonly yes  #開啟aof

    cluster-enabled yes  # 開啟集群

    cluster-config-file /var/lib/redis/nodes.conf # 集群設定檔位置

apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-conf
  namespace: redis-cluster
data:
  redis.conf: |
    appendonly yes
    cluster-enabled yes
    cluster-config-file /var/lib/redis/nodes.conf
    dir /var/lib/redis
    port 6379

Step3. 創建 PV

    建立六個pv對應六個redis cluster pod

    pv6, pv7用來測試增加redis node

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfspv0
spec:
  storageClassName: redis-storage-class
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: "/redis0"
    server: 192.168.43.205
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfspv1
spec:
  storageClassName: redis-storage-class
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: "/redis1"
    server: 192.168.43.205
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfspv2
spec:
  storageClassName: redis-storage-class
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: "/redis2"
    server: 192.168.43.205
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfspv3
spec:
  storageClassName: redis-storage-class
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: "/redis3"
    server: 192.168.43.205
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfspv4
spec:
  storageClassName: redis-storage-class
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: "/redis4"
    server: 192.168.43.205
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfspv5
spec:
  storageClassName: redis-storage-class
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: "/redis5"
    server: 192.168.43.205
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfspv6
spec:
  storageClassName: redis-storage-class
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: "/redis6"
    server: 192.168.43.205
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfspv7
spec:
  storageClassName: redis-storage-class
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    path: "/redis7"
    server: 192.168.43.205

Step4. 創建 HeadlessService

    StatefulSet需建立headlessService

apiVersion: v1
kind: Service
metadata:
  name: redis-headless-server
  namespace: redis-cluster
  labels:
    app: redis-cluster-app
spec:
  ports:
  - name: redis-port
    port: 6379
  clusterIP: None
  selector:
    app: redis-cluster-app

Step5. 創建 StatefulSet

     --cluster-announce-ip  $(MY_POD_IP) 可在pod重啟集群自動恢復時帶入新ip

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-app
  namespace: redis-cluster
spec:
  selector:
    matchLabels:
      app: redis-cluster-app
  serviceName: "redis-cluster-app"
  replicas: 6
  template:
    metadata:
      labels:
        app: redis-cluster-app
    spec:
      containers:
      - name: redis
        image: "redis:6.0.6"
        command:
          - "redis-server"
        args:
          - "/etc/redis/redis.conf"
          - "--protected-mode"
          - "no"
          - "--cluster-announce-ip"
          - " $(MY_POD_IP)"
        ports:
          - name: redis
            containerPort: 6379
            protocol: "TCP"
          - name: cluster
            containerPort: 16379
            protocol: "TCP"
        volumeMounts:
          - name: "redis-conf"
            mountPath: "/etc/redis"
          - name: "redis-data"
            mountPath: "/var/lib/redis"
        env:
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
      volumes:
      - name: "redis-conf"
        configMap:
          name: "redis-conf"
          items:
            - key: "redis.conf"
              path: "redis.conf"
  volumeClaimTemplates:
  - metadata:
      name: redis-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "redis-storage-class"
      resources:
        requests:
          storage: 1Gi

Step6. 創建 redis cluster

kubectl get pods -l app=redis-cluster-app -o jsonpath='{range.items[*]}{.status.podIP}:6379 ' -n redis-cluster #取得redis pod ip

kubectl -n redis-cluster exec -it redis-app-0 -- redis-cli --cluster create --cluster-replicas 1 \
$(kubectl get pods -l app=redis-cluster-app -o jsonpath='{range.items[*]}{.status.podIP}:6379 ' -n redis-cluster )

Step7. 創建 Service

創建服務service

apiVersion: v1
kind: Service
metadata:
  name: redis-service
  namespace: redis-cluster
  labels:
    app: redis-cluster-app
spec:
  ports:
  - name: redis-port
    protocol: "TCP"
    port: 6379
    targetPort: 6379
  selector:
    app: redis-cluster-app

測試1 集群自動恢復

    下圖是目前redis cluster pod 現況以及cluster nodes現況

    pod name 為 redis-app-0 的 ip 是 10.244.1.78,redis cluster nodes myself ip為10.244.1.78

    這邊會做一個實驗嘗試把這個pod刪除，讓k8s重新建立pod並且測試redis集群是否會自動恢復

測試結果

    刪除後redis-app-0 name 的 pod自動重新創建，ip更換為10.244.1.80但並不影響redis集群，redis cluster nodes運行狀態正常 myself ip也更新為10.244.1.80

測試2 增加redis node

  增加到8個redis pod

kubectl scale statefulsets redis-app --replicas=8 -n redis-cluster

    將新增的兩個redis node 加入到cluster中。

kubectl -n redis-cluster exec -it redis-app-0 -- redis-cli --cluster add-node 10.244.1.81:6379 10.244.1.80:6379
kubectl -n redis-cluster exec -it redis-app-0 -- redis-cli --cluster add-node 10.244.3.73:6379 10.244.1.81:6379  --cluster-slave

kubectl -n redis-cluster exec -it redis-app-0 -- redis-cli --cluster reshard 10.244.2.64:6379

    將1000個slots從10.244.1.80移至10.244.1.81

測試3 佈建spring boot redis api 測試專案

maven dependency

    使用jib產生並上傳docker image，這邊有事先準備了一個私有image registry harbor(也可用docker hub)，會先將image傳到harbor，k8s再將image拉下測試。

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.12.RELEASE</version>
        <relativePath/>
    </parent>
    <groupId>org.example</groupId>
    <artifactId>RedisK8sDemo</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-redis</artifactId>
        </dependency>
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>com.google.cloud.tools</groupId>
                <artifactId>jib-maven-plugin</artifactId>
                <version>2.4.0</version>
                <configuration>
                    <to>
                        <image>k8s-harbor.com/library/redis-k8s-demo</image>
                    </to>
                    <allowInsecureRegistries>true</allowInsecureRegistries>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

application.yml

spring:
  redis:
    cluster:
      nodes: redis-service.redis-cluster.svc.cluster.local:6379
      timeout: 2000
      max-redirects: 7

啟動類

@SpringBootApplication
public class RedisK8sDemoApplication {
    public static void main(String[] args) {
        SpringApplication.run(RedisK8sDemoApplication.class, args);
    }
}

redis config

@Configuration
@EnableCaching
public class RedisConfig extends CachingConfigurerSupport {


    private RedisConnectionFactory connectionFactory(RedisClusterConfiguration configuration) {
        JedisConnectionFactory connectionFactory = new JedisConnectionFactory(configuration);
        connectionFactory.afterPropertiesSet();
        return connectionFactory;
    }

    @Bean
    public RedisClusterConfiguration getClusterConfiguration(@Value("${spring.redis.cluster.nodes}") String clusterNodes,
                                                             @Value("${spring.redis.cluster.timeout}") Long timeout,
                                                             @Value("${spring.redis.cluster.max-redirects}") int redirects) {
        Map<String, Object> source = new HashMap<>();
        source.put("spring.redis.cluster.nodes", clusterNodes);
        source.put("spring.redis.cluster.timeout", timeout);
        source.put("spring.redis.cluster.max-redirects", redirects);
        return new RedisClusterConfiguration(new MapPropertySource("RedisClusterConfiguration", source));
    }

    @Bean
    public CacheManager cacheManager(RedisConnectionFactory factory) {
        return RedisCacheManager.builder(factory).build();
    }
}

controller

@RestController
public class TestRedisController {

    @Autowired
    private RedisService redisService;

    private final static Logger LOGGER = LoggerFactory.getLogger(TestRedisController.class);

    @GetMapping("/{key}")
    public String testCache(@PathVariable String key) {
        LOGGER.info("controller start .......");
        String name = redisService.getName(key);
        LOGGER.info("controller end .......");
        return name;
    }
}

service

@Service
public class RedisService {
    
    private final static Logger LOGGER = LoggerFactory.getLogger(RedisService.class);

    @Cacheable(value = "name")
    public String getName(String key) {
        LOGGER.info("key: {}", key);
        return key;
    }
}

test-redis-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-redis-deployment
  namespace: redis-cluster
  labels:
    app: test-redis-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-redis-app
  template:
    metadata:
      labels:
        app: test-redis-app
    spec:
      containers:
      - name: test-redis-app
        image: k8s-harbor.com/library/redis-k8s-demo

    app-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: app-service
  namespace: redis-cluster
spec:
  selector:
    app: test-redis-app
  ports:
    - name: http
      port: 80
      targetPort: 8080

    ingress-service.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: app-ingress
  namespace: redis-cluster
spec:
  rules:
  - host: k8s-test.com
    http:
      paths:
      - path: /test-redis
        backend:
          serviceName: app-service
          servicePort: 80

    k8s事先有採用Bare-metal方式安裝nginx ingress

    詳細安裝方式可以參考官方網站

    先將專案包成image推到harbor後，在k8s 建立相對應資源進行測試。

    包成image

mvn compile jib:dockerBuild -e -DsendCredentialsOverHttp=true -DallowInsecureRegistries=true

    push到harbor registry

docker push k8s-harbor.com/library/redis-k8s-demo:3.0.0

    建立測試deployment

kubectl apply -f test-redis-deployment.yaml

    建立測試service

kubectl apply -f app-service.yaml

    建立測試ingress-service

kubectl apply -f ingress-service.yaml

測試結果

    這邊使用網頁連續呼叫api兩次只有第一次印出緩存方法內的log，第二次直接走redis緩存，因此第二次未印出method內log

參考資源

⋅ redis官方網站

⋅ K8S官方網站

⋅ ingress nginx官方網站

⋅ 在K8S上搭建Redis集群

⋅ SpringBoot 使用redis並實現cache機制

⋅ github redis cluster sample

 

 

Bob Lien
相關文章
MongoDB管理最佳拍檔 - 監控管理組合拳Part1
digiC 簡易介紹
MongoDB管理最佳拍檔 - digiMars一站式管理
深入淺出digiRunner - 透過批次更新 API 標籤的方式，根據指定的標籤條件查詢並獲取 API 列表
Composer 如何處理需要 redirect 的 API
使用 Composer 製作取得病歷 API
最新文章
Business Story - Logistics Management in Smart Warehousing using digiRunner
Redefining Manufacturing with digiRunner's Digital Transformation
Bridging Healthcare with digiRunner: FHIR Interoperability
Revolutionizing Government Services with digiRunner
Transforming Financial Services with digiRunner API Management
Industry Transformation with digiRunner: Best Practices Across Sectors