How to: Why do I see 504 errors from my Ingress during deploys?
This occurs due to a race condition during pod deletion between the Ingress and the pod. When a pod is deleted, it can shut down before the Ingress knows to stop sending traffic. So the Ingress may continue to send traffic to a disabled pod.

The simplest way to avoid this is to prevent the pod from shutting down immediately with a `preStop` hook. Adding in a `preStop` hook to the deployment which does `sleep 5` should delay the pod termination long enough to let the Ingress update and remove the disabled pod from its upstream list.

https://github.com/kubernetes/kubernetes/issues/43576
https://github.com/kubernetes/ingress/issues/322
https://github.com/kubernetes/contrib/issues/1140
https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods