A health check endpoint was required in the Open Event Server be used by Kubernetes to know when the web instance is ready to receive requests.
Following are the checks that were our primary focus for health checks:
- Connection to the database.
- Ensure sql-alchemy models are inline with the migrations.
- Connection to celery workers.
- Connection to redis instance.
Runscope/healthcheck seemed like the way to go for the same. Healthcheck wraps a Flask app object and adds a way to write simple health-check functions that can be used to monitor your application. It’s useful for asserting that your dependencies are up and running and your application can respond to HTTP requests. The Healthcheck functions are exposed via a user defined flask route so you can use an external monitoring application (monit, nagios, Runscope, etc.) to check the status and uptime of your application.
Health check endpoint was implemented at /health-check as following:
from healthcheck import HealthCheck health = HealthCheck(current_app, "/health-check")
Following is the function for checking the connection to the database:
def health_check_db(): """ Check health status of db :return: """ try: db.session.execute('SELECT 1') return True, 'database ok' except: sentry.captureException() return False, 'Error connecting to database'
Check functions take no arguments and should return a tuple of (bool, str). The boolean is whether or not the check passed. The message is any string or output that should be rendered for this check. Useful for error messages/debugging.
The above function executes a query on the database to check whether it is connected properly. If the query runs successfully, it returns a tuple True, ‘database ok’. sentry.captureException() makes sure that the sentry instance receives a proper exception event with all the information about the exception. If there is an error connecting to the database, the exception will be thrown. The tuple returned in this case will be return False, ‘Error connecting to database’.
Finally to add this to the endpoint:
health.add_check(health_check_db)
Following is the response for a successful health check:
{ "status": "success", "timestamp": 1500915121.52474, "hostname": "shubham", "results": [ { "output": "database ok", "checker": "health_check_db", "expires": 1500915148.524729, "passed": true, "timestamp": 1500915121.524729 } ] }
If the database is not connected the following error will be shown:
{ "output": "Error connecting to database", "checker": "health_check_db", "expires": 1500965798.307425, "passed": false, "timestamp": 1500965789.307425 }
Related:
- Health Endpoint in API Design : http://byterot.blogspot.in/2014/11/health-endpoint-in-api-design-slippery-rest-api-design-canary-endpoint-hysterix-asp-net-web-api.html
- Kubernetes Health Checking:
https://kubernetes.io/docs/user-guide/walkthrough/k8s201/#application-health-checking