LAB K105 - Load Balancing and Service Discovery with Services

In this lab, you would not only publish the application deployed with replication controller earlier, but also learn about the load balancing and service discovery features offered by kubernetes.

Concepts related to Kubernetes Services are depicted in the following diagram,

kubernetes service.\label{fig:captioned_image}

Publishing external facing app with NodePort

Kubernetes comes with four types of services viz.

  • ClusterIP
  • NodePort
  • LoadBalancer
  • ExternalName

Lets create a service of type NodePort to understand how it works.

To check the status of kubernetes objects,

kubectl get pods,rs,svc

You could also start watching the above output for changes. To do so, open a separate terminal window and run,

watch -n 1 kubectl get  pod,deploy,rs,svc

Refer to Service Specs to understand the properties that you could write.

filename: vote-svc.yaml

apiVersion: v1
kind: Service
  name: vote
    role: vote
    role: vote
    - port: 80
      targetPort: 80
      nodePort: 30000
  type: NodePort

Apply this file to to create a service

kubectl apply -f vote-svc.yaml --dry-run
kubectl apply -f vote-svc.yaml
kubectl get svc
kubectl describe service vote

[Sample Output of describe command]

Name:                     vote
Namespace:                instavote
Labels:                   role=svc
Annotations:    {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"role":"svc","tier":"front"},"name":"vote","namespace":"instavote"},"spec":{...
Selector:                 app=vote
Type:                     NodePort
Port:                     <unset>  80/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31429/TCP
Endpoints:      ,, + 2 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Observe the following

  • Selector
  • TargetPort
  • NodePort
  • Endpoints

Go to browser and check http://HOSTIP:NODEPORT

Here the node port is 30000 (as defined by nodePort in service spec).

Sample output will be:


If you refresh the page, you should also notice its sending traffic to diffent pod each time, in round robin fashion.


  • Change the selector criteria to use a non existant label. Use kubectl edit svc./vote to update and apply the configuration. Observe the output of describe command and check the endpoints. Do you see any ? How are selectors and pod labels related ?
  • Observe the number of endpoints. Then change the scale of replicas created by the replication controllers. Does it have any effect on the number of endpoints ?

Services Under the Hood

Lets traverse the route of the network packet that comes in on port 30000 on any node in your cluster.

iptables -nvL -t nat  
iptables -nvL -t nat  | grep 30000

Anything that comes on dpt:3000, gets forwarded to the chain created for that service.

iptables -nvL -t nat  | grep KUBE-SVC-VIQHAVHDK4QE7NA4  -A 10

Chain KUBE-SVC-VIQHAVHDK4QE7NA4 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-RFJGHFMXUDJXIEW6  all  --  *      *              /* instavote/vote: */ statistic mode random probability 0.20000000019
    0     0 KUBE-SEP-GBR5YQCVRYY3BA6U  all  --  *      *              /* instavote/vote: */ statistic mode random probability 0.25000000000
    0     0 KUBE-SEP-BAI3HQ7SV7RZ2CI6  all  --  *      *              /* instavote/vote: */ statistic mode random probability 0.33332999982
    0     0 KUBE-SEP-2EQSLPEP3WDOTI5J  all  --  *      *              /* instavote/vote: */ statistic mode random probability 0.50000000000
    0     0 KUBE-SEP-2CJQISP4W7F2HCRW  all  --  *      *              /* instavote/vote: */

Where, * counrt of KUBE-SEP-xxx matches number of pods * KUBE-SEP-BAI3HQ7SV7RZ2CI6 is a chain created for one of host. examine that next

iptables -nvL -t nat  | grep KUBE-SEP-BAI3HQ7SV7RZ2CI6  -A 3


pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-MARK-MASQ  all  --  *      *              /* instavote/vote: */
    0     0 DNAT       tcp  --  *      *              /* instavote/vote: */ tcp to:

where the packet is being forwarded to, which should corraborate with the ip of the pod


kubectl get pods -o wide

NAME         READY     STATUS    RESTARTS   AGE       IP          NODE
vote-58bpv   1/1       Running   0          1h   k-02
vote-986cl   1/1       Running   0          1h   k-03
vote-9rrfz   1/1       Running   0          1h   k-03
vote-dx8f4   1/1       Running   0          1h   k-02
vote-qxmfl   1/1       Running   0          1h   k-02 matches ip of vote-58bpv

to check how the packet is routed next use,

route -n


Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface         UG    0      0        0 eth0     U     0      0        0 eth0     U     0      0        0 weave     U     0      0        0 docker0   U     0      0        0 eth0

where, is going over weave interface.

Exposing app with ExternalIP

Observe the output of service list, specifically note the EXTERNAL-IP colum in the output.

kubectl  get svc

Now, update the service spec and add external IP configs. Pick IP addresses of any two nodes (You could add one or more) and it to the spec as,

kubectl edit svc vote

[sample file edit]

apiVersion: v1
kind: Service
  name: vote
    role: vote
    role: vote
    - port: 80
      targetPort: 80
      nodePort: 30000
  type: NodePort
    - xx.xx.xx.xx
    - yy.yy.yy.yy


replace xx.xx.xx.xx and yy.yy.yy.yy with IP addresses of the nodes on two of the kubernetes hosts.


kubectl  get svc
kubectl apply -f vote-svc.yaml
kubectl  get svc
kubectl describe svc vote

[sample output]

NAME      TYPE       CLUSTER-IP      EXTERNAL-IP                    PORT(S)        AGE
vote      NodePort,   80:30000/TCP   11m


EXTERNAL-IP column shows which IPs the application is been exposed on. You could go to http://: to access this application. e.g. where you should replace with the actual IP address of the node that you exposed this on.

Internal Service Discovery

Kubernetes not only allows you to publish external facing apps with the services, but also allows you to discover other components of your application stack with the clusterIP and DNS attached to it.

Before you begin adding service discovery,

  • Visit the vote app from browser
  • Attempt to vote by clicking on one of the options

observe what happens. Does it go through?


kubectl get pod
kubectl exec vote-xxxx nslookup redis

[replace xxxx with the actual pod id of one of the vote pods ]

keep the above command on a watch. You should create a new terminal to run the watch command.


kubectl exec -it vote-xxxx sh
watch  kubectl exec vote-xxxx ping redis

where, vote-xxxx is one of the vote pods that I am running. Replace this with the actual pod id.

Now create redis service

kubectl apply -f redis-svc.yaml

kubectl get svc

kubectl describe svc redis

Watch the nslookup screen and observe if its able to resolve redis by hostname and its pointing to an IP address.


Name:      redis
Address 1: redis.instavote.svc.cluster.local


  • is the ClusterIP assigned to redis service
  • redis.instavote.svc.cluster.local is the dns attached to the ClusterIP above

What happened here?

  • Service redis was created with a ClusterIP e.g.
  • A DNS entry was created for this service. The fqdn of the service is redis.instavote.svc.cluster.local and it takes the form of
  • Each pod points to internal DNS server running in the cluster. You could see the details of this by running the following commands
kubectl exec vote-xxxx cat /etc/resolv.conf

[replace vote-xxxx with actual pod id]

[sample output]

search instavote.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

where is the ClusterIP assigned to the DNS service. You could co relate that with,

kubectl get svc -n kube-system

NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
kube-dns               ClusterIP     <none>        53/UDP,53/TCP   1h
kubernetes-dashboard   NodePort   <none>        80:31000/TCP    23m

where, is the ClusterIP assigned to kube-dns and matches the configuration in /etc/resolv.conf above.

Creating Endpoints for Redis

Service is been created, but you still need to launch the actual pods running redis application.

Create the endpoints now,

kubectl apply -f redis-deploy.yaml
kubectl describe svc redis

[sample output]

Name:              redis
Namespace:         instavote
Labels:            role=redis
Selector:          app=redis
Type:              ClusterIP
Port:              <unset>  6379/TCP
TargetPort:        6379/TCP
Session Affinity:  None
Events:            <none>

Again, visit the vote app from browser, attempt to register your vote. observe what happens. This time the vote should be registered successfully.


In this lab, you have published a front facing application, learnt how services are implemented under the hood as well as added service discovery to provide connection strings automatically.