As we all know, the basic units running in a Kubernetes cluster are docker containers, docker has its native logging drivers that can provide basic logging functionality, whether to write to stdout and stderr, or write to json file with rotations.

However, it’s not enough for a full logging solution. For example, your API services may have 3 replicas, which in turn will bring up 3 containers on which your logs are distributed across, how do you aggregate all the logs and be able to view them at one place? Another example is if your container crashes, the container will die and will be replaced with another one on the node, or could even gets scheduled on a different node, in this case, the logs of that container could get lost. Due to the above reasons, when it comes to logging, Kubernetes promotes cluster-level-logging, it means we need to have a separate backend to store, analyze and query logs, so that the logs are independent of the lifecycle of of any containers, pods or the nodes in the cluster.

Kubernetes does not ship with any logging facility that suffices the cluster-level-logging requirement, however there are many open source solutions you can leverage on to implement a full logging service. Among them, fluentd + elasticsearch form a great combination to provide a cluster level logging solution. In this post, I’ll introduce how to get it done.

In a nutshell, the ideal is:

  • Use fluentd as an agent on each node to scrape all the container logs
  • Push the logs to Elasticsearch for storing, analyzing and querying using logstash-ish indices.

Use fluentd as node log agents

Deploy Fluentd

We need to make sure we have a fluentd agent running on each node, and this fits the DaemonSet of Kubernetes perfectly. In order to make sure the pods can only access relevant resources in the cluster, we will create a service account and use it to create the DaemonSet later.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluentd
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-system

---

You can see that we only grant this service account the read access to namespaces and pods, which will be enough.

Now we move on to the DaemonSet definition, fluentd has published a set of images that support logging with Kubernetes meta support, and here we use fluent/fluentd-kubernetes-daemonset:v0.12-alpine-elasticsearch

By default, you don’t need to make any configuration changes, this image will try to collect the following logs if they exists in /var/log/ directory, pack them with kubernetes metadata and push to an elasticsearch backend with logstash-ish indices.

  • all your docker containers logs
  • kubelet
  • kube-api-server
  • kube-controller-manager
  • kube-scheduler
  • kube-proxy
  • etcd

In our case, we are only interested in the containerized application logs for our workload but not the system applications, in order to achieve this, we can customize the the fluentd configuration file kubernetes.conf to replace the default one. We will only follow the docker container logs that we are interested. Typically you will have different namespaces for your application instead of using default and kube-system.

We can create a config map that contains fluentd.conf and kubernetes.conf and mount it to the /fluentd/etc path in the fluentd pods. The configuration file is shown as follows.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
apiVersion: v1
kind: ConfigMap
metadata:
name: fluentd-kubernetes-config
namespace: kube-system
data:
fluent.conf: |-
@include kubernetes.conf

<match **>
@type elasticsearch
@id out_es
log_level info
include_tag_key true
host "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
logstash_format true
buffer_chunk_limit 2M
buffer_queue_limit 32
flush_interval 5s
max_retry_wait 30
disable_retry_limit
num_threads 8
</match>
kubernetes.conf: |-
<match fluent.**>
@type null
</match>

<source>
@type tail
@id in_tail_container_logs
path /var/log/containers/*tds-*.log
pos_file /var/log/fluentd-containers.log.pos
tag kubernetes.*
read_from_head true
format json
time_format %Y-%m-%dT%H:%M:%S.%NZ
</source>

<filter kubernetes.**>
@type kubernetes_metadata
@id filter_kube_metadata
</filter>

---

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: fluentd
namespace: kube-system
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
k8s-app: fluentd-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: fluentd
image: fluent/fluentd-kubernetes-daemonset:v0.12-alpine-elasticsearch
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
resources:
limits:
cpu: 100m
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
- name: kubernetesconfig
mountPath: /fluentd/etc
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: kubernetesconfig
configMap:
name: fluentd-kubernetes-config

You might wonder why the host path /var/lib/docker/containers is also mounted as volume. It’s because the docker container logs in /var/log/ are actually symbol links which points to /var/lib/docker/containers/(container_id)/***.log. If you don’t mount /var/lib/docker/container, fluentd won’t be able to read the actually log files.

Wait for the Daemonset to be deployed, you should find the pods count equals your node count.

You might notice that we set the FLUENT_ELASTICSEARCH_HOST to elasticsearch, this actually implicates that we should have an elasticsearch service running within the cluster itself. If you are using some external elasticsearch service, you can just set the three environment variables accordingly to point to your elasticsearch service and you will be all set:

  • FLUENT_ELASTICSEARCH_HOST
  • FLUENT_ELASTICSEARCH_PORT
  • FLUENT_ELASTICSEARCH_SCHEME

You should start to receive logstash-ish logs in your elasticsearch cluster once you have the DaemonSet up and running.

1
2
3
4
5
curl -X GET 'http://your_elastic_service_url:9200/_cat/indices?v='

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-2018.07.04 bRiQJCnrTZqOQQI4ol1mfw 5 1 16744 0 24.4mb 12.1mb
green open .kibana 6UUqrS5_RUSS5Ozy2tSYTg 1 1 19 1 115kb 57.5kb

If you want to use an in-cluster elasticsearch cluster, obviously you need to deploy one, please continue reading.

Deploy Elasticsearch

This part is assuming that your Kubernetes cluster is running on AWS EC2 instances across different availability zones to achieve HA. However even if it’s not, it can also provide a reference for you.

Here we’ll deploy a minimum HA elasticsearch cluster which consists:

  • 3 master nodes
  • 2 data nodes

The 3 master nodes manage the cluster and opens API access to clients. 1 of the 3 at all times will be elected as the leader, so that we have an active-passive HA master cluster.

In order not to put the masters under pressure, we separate the data nodes from master nodes. The 2 dedicated data nodes are used to perform resources heavy data operations such as CURD, search and etc. We use 2 nodes to make sure that for each index we have at least a replica on a different node, so if one node is offline, the cluster is still able to function.

Use Kubernetes Persistent Volume

In Kubernetes, usually your applications are stateless, which means it does not matter where the pods running them are deployed and how many times they are restarted, you don’t reply on any storage which outlives the life time of the pods. However, it is clearly the opposite case for elasticsearch cluster, as it provides a data service, you don’t want to lose your data when one or two pods running elasticsearch application get restarted.

And this is where Persistent Volume (PV) comes in to play. A PV in Kubernetes is a storage resource in the cluster just like a node is a computing resource in the cluster. The lifecycle of a PV is independent to any pods that use it.

A pod uses PV via Persistent Volume Claim (PVC), which translates to: I want to claim X size of storage with Y access mode from the cluster. So PVCs are like pods, as PVCs uses PV resource while pods use node resource.

Dynamic Provisioning Of PV

When a pod requests PV using PVC and the cluster can’t find a matching PV for it, it will try to dynamically provision a PV and bind it with the PVC. This is called dynamic provisioning. To enable dynamic provisioning, the API server must have been configured to support it, the PVC also needs to specify a StorageClass because that’s what the cluster will refer to when it provisions the PV.

AWSElasticBlockStore StorageClass and PV

Since the Kuberentes cluster is deployed on AWS EC2 instances, it’s nature to use EBS PV. To achieve HA, I assume that you have your node distributed across multiple zones, in this example, let’s say we have us-east-1a, us-east-1b and us-east-1c. The first thing to do is to create a StorageClass for each zone.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2-us-east-1a
labels:
failure-domain.beta.kubernetes.io/region: us-east-1
failure-domain.beta.kubernetes.io/zone: us-east-1a
k8s-addon: storage-aws.addons.k8s.io
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zone: us-east-1a

---

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2-us-east-1b
labels:
failure-domain.beta.kubernetes.io/region: us-east-1
failure-domain.beta.kubernetes.io/zone: us-east-1b
k8s-addon: storage-aws.addons.k8s.io
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zone: us-east-1b

---

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: gp2-us-east-1c
labels:
failure-domain.beta.kubernetes.io/region: us-east-1
failure-domain.beta.kubernetes.io/zone: us-east-1c
k8s-addon: storage-aws.addons.k8s.io
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zone: us-east-1c

Notes: You can specify one StorageClass with zones: us-east-1a,us-east-1b,us-east-1c, however, when you claim volume using this StorageClass, you can’t guarantee which zone the EBS volumes the cluster creates lies in, and it’s every likely that the EBS volume created lies in a different zone from the node where the pod gets scheduled on, thus the binding can never be successfully. This is the reason why we create multiple StorageClass objects and point each of them to a specific zone.

Deploy data nodes using Kubernetes StatefulSet

StatefulSet, on the contrast of Deployment, is used for stateful applications. The major differences between StatefulSet and Deployment are StatefulSet:

  • Maintenance a sticky identity of pods.
  • Have stable network ID. (with the help of a headless service)
  • Have stable and persistent storage.

It makes perfect sense to use StatefulSet to deploy the elasticsearch data nodes, because we want the nodes to be across different zones and we want the persistent EBS volumes to be dynamically created, we need to create two StatefulSet, 1 per zone.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
---
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-data
namespace: kube-system
labels:
component: elasticsearch
role: data
spec:
selector:
component: elasticsearch
role: data
ports:
- name: transport
port: 9300
protocol: TCP
clusterIP: None

---

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: es-data-us-east-1a
namespace: kube-system
labels:
component: elasticsearch
role: data
spec:
updateStrategy:
type: RollingUpdate
serviceName: elasticsearch-data
replicas: 1
template:
metadata:
labels:
component: elasticsearch
role: data
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- us-east-1a
initContainers:
- name: init-sysctl
image: busybox:1.27.2
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
- name: update-volume-permission
image: busybox:1.27.2
command:
- chown
- -R
- 1000:1000
- /data
volumeMounts:
- name: storage
mountPath: /data
securityContext:
privileged: true
containers:
- name: es-data
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.0
env:
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: cluster.name
value: tds-es-db
- name: discovery.zen.ping.unicast.hosts
value: "elasticsearch-discovery"
- name: discovery.zen.minimum_master_nodes
value: "2"
- name: node.master
value: "false"
- name: node.ingest
value: "false"
- name: node.data
value: "true"
- name: xpack.ml.enabled
value: "false"
- name: xpack.security.enabled
value: "false"
- name: xpack.watcher.enabled
value: "false"
- name: xpack.graph.enabled
value: "false"
- name: xpack.logstash.enabled
value: "false"
- name: node.master
value: "false"
- name: node.ingest
value: "false"
- name: node.data
value: "true"
- name: http.enabled
value: "false"
- name: ES_JAVA_OPTS
value: "-Xms2g -Xmx2g"
- name: network.host
value: _eth0:ipv4_
- name: path.data
value: /data
resources:
limits:
cpu: 0.5
ports:
- containerPort: 9300
name: transport
livenessProbe:
tcpSocket:
port: transport
initialDelaySeconds: 60
periodSeconds: 10
volumeMounts:
- name: storage
mountPath: /data
volumeClaimTemplates:
- metadata:
name: storage
spec:
storageClassName: gp2-us-east-1a
accessModes: [ ReadWriteOnce ]
resources:
requests:
storage: 150Gi

---

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: es-data-us-east-1c
namespace: kube-system
labels:
component: elasticsearch
role: data
spec:
updateStrategy:
type: RollingUpdate
serviceName: elasticsearch-data
replicas: 1
template:
metadata:
labels:
component: elasticsearch
role: data
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- us-east-1c
initContainers:
- name: init-sysctl
image: busybox:1.27.2
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
- name: update-volume-permission
image: busybox:1.27.2
command:
- chown
- -R
- 1000:1000
- /data
volumeMounts:
- name: storage
mountPath: /data
securityContext:
privileged: true
containers:
- name: es-data
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.0
env:
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: cluster.name
value: tds-es-db
- name: discovery.zen.ping.unicast.hosts
value: "elasticsearch-discovery"
- name: discovery.zen.minimum_master_nodes
value: "2"
- name: node.master
value: "false"
- name: node.ingest
value: "false"
- name: node.data
value: "true"
- name: xpack.ml.enabled
value: "false"
- name: xpack.security.enabled
value: "false"
- name: xpack.watcher.enabled
value: "false"
- name: xpack.graph.enabled
value: "false"
- name: xpack.logstash.enabled
value: "false"
- name: node.master
value: "false"
- name: node.ingest
value: "false"
- name: node.data
value: "true"
- name: http.enabled
value: "false"
- name: ES_JAVA_OPTS
value: "-Xms2g -Xmx2g"
- name: network.host
value: _eth0:ipv4_
- name: path.data
value: /data
resources:
limits:
cpu: 0.5
ports:
- containerPort: 9300
name: transport
livenessProbe:
tcpSocket:
port: transport
initialDelaySeconds: 60
periodSeconds: 10
volumeMounts:
- name: storage
mountPath: /data
volumeClaimTemplates:
- metadata:
name: storage
spec:
storageClassName: gp2-us-east-1c
accessModes: [ ReadWriteOnce ]
resources:
requests:
storage: 150Gi

A few highlights in the above configuration file:

  • A headless service is created and points to the pods.

  • Each StatefulSet is restricted to deploy pods on only 1 zone, with the PVC pointing to the StorageClass for that zone.

  • We turn off node.master so that they are dedicated data nodes.

  • In order to let elasticsearch user access to mounted volume, we added a initial container to grant the access to the data path.

Deploy master nodes using Kubernetes Deployment

Deploying the master nodes are much easier than data nodes as they don’t need persistent storage. We use Deployment to manage that. In order to let the clients access the API, we need to expose the service as well, in my case, I use ingress to expose it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: elasticsearch-ingress
namespace: kube-system
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: xxx.xxx.com
http:
paths:
- backend:
serviceName: elasticsearch
servicePort: 9200

---

apiVersion: v1
kind: Service
metadata:
name: elasticsearch-discovery
namespace: kube-system
labels:
component: elasticsearch
role: master
spec:
selector:
component: elasticsearch
role: master
ports:
- name: transport
port: 9300
protocol: TCP
clusterIP: None

---

apiVersion: v1
kind: Service
metadata:
name: elasticsearch
namespace: kube-system
labels:
component: elasticsearch
role: master
spec:
selector:
component: elasticsearch
role: master
type: ClusterIP
ports:
- name: http
protocol: TCP
port: 9200
targetPort: 9200

---

apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: es-master
namespace: kube-system
labels:
component: elasticsearch
role: master
spec:
replicas: 3
template:
metadata:
labels:
component: elasticsearch
role: master
spec:
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
initContainers:
- name: init-sysctl
image: busybox:1.27.2
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
containers:
- name: es-master
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.0
env:
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: cluster.name
value: tds-es-db
- name: discovery.zen.ping.unicast.hosts
value: "elasticsearch-discovery"
- name: discovery.zen.minimum_master_nodes
value: "2"
- name: xpack.ml.enabled
value: "false"
- name: xpack.security.enabled
value: "false"
- name: xpack.watcher.enabled
value: "false"
- name: xpack.graph.enabled
value: "false"
- name: xpack.logstash.enabled
value: "false"
- name: node.master
value: "true"
- name: node.ingest
value: "false"
- name: node.data
value: "false"
- name: http.enabled
value: "true"
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
- name: network.host
value: _eth0:ipv4_
resources:
limits:
cpu: 0.5
ports:
- containerPort: 9200
name: http
- containerPort: 9300
name: transport
livenessProbe:
tcpSocket:
port: transport
initialDelaySeconds: 60
readinessProbe:
httpGet:
path: /_cluster/health
port: http
initialDelaySeconds: 60
timeoutSeconds: 5
volumeMounts:
- name: storage
mountPath: /data
volumes:
- emptyDir:
medium: ""
name: "storage"

Some highlights in the above configuration:

  • node.master is enabled and node.data is disabled

  • A 60 seconds waiting time are set to allow the pods become alive and to able to health checked

Notes: when you kubectl apply to deploy the data nodes, you might encounter timeout for binding the EBS volumes, don’t panic, just do it again. It’s because it takes time for AWS to provision the volume and make it become available to mount.

Deploy Kibana to visualise your elasticsearch data

Your fluentd + elasticsearch cluster combination should be working fine now, check the status by running:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
curl -X GET \
'http://your.exposed.elasticsearch.domain/_cluster/health?pretty='

{
"cluster_name": "xx-xx-xxx",
"status": "green",
"timed_out": false,
"number_of_nodes": 5,
"number_of_data_nodes": 2,
"active_primary_shards": 46,
"active_shards": 92,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0,
"delayed_unassigned_shards": 0,
"number_of_pending_tasks": 0,
"number_of_in_flight_fetch": 0,
"task_max_waiting_in_queue_millis": 0,
"active_shards_percent_as_number": 100
}

Now to play with your data, you might want to install Kibana, it’s quite straightforward, we just need to deploy 1 instance of Kibana using Deployment, and we also expose the service using Ingress so that we can access it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kibana-ingress
namespace: kube-system
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: xxx.xxx.xxx
http:
paths:
- backend:
serviceName: kibana
servicePort: 80
---

apiVersion: v1
kind: Service
metadata:
name: kibana
namespace: kube-system
labels:
component: kibana
spec:
selector:
component: kibana
ports:
- name: http
port: 80
targetPort: http
---

apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: kibana
namespace: kube-system
labels:
component: kibana
spec:
replicas: 1
selector:
matchLabels:
component: kibana
template:
metadata:
labels:
component: kibana
spec:
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:6.3.0
env:
- name: ELASTICSEARCH_URL
value: "http://elasticsearch:9200"
- name: CLUSTER_NAME
value: tds-es-db
- name: XPACK_SECURITY_ENABLED
value: "false"
- name : XPACK_GRAPH_ENABLED
value: "false"
- name : XPACK_REPORTING_ENABLED
value: "false"
- name : XPACK_ML_ENABLED
value: "false"
resources:
limits:
cpu: 200m
requests:
cpu: 100m
ports:
- containerPort: 5601
name: http

Go to your Kibana website and create an index pattern like logstash-*, you should see all the available fields extracted from your applications’ logs and the kubernetes meta data.

You can now use the Kibana’s Discover to monitor and query all of your application logs, happy monitoring!