In the world of stateless microservices, which are usually written in Go, we need to discover them. This is where Hashicorp’s Consul helps. Services register within Consul so other services can discover them via simple DNS or HTTP queries.
Go has a Consul client library, alas, I didn’t see any real examples of how to integrate it into your services. So here I’m going to show you how to do exactly this.
I’m going to write a service that will serve at some HTTP endpoint and will
serve key-value data – I believe this resembles a lot of existing microservices
that people write these days. Ours is called webkv and it’s on Github.
Choose the “v1” tag and you’re good to go.
This service will register itself in Consul with TTL check that will, well, check internal health status and send a heartbeat like signals to Consul. Should Consul not receive a signal from our service within a TTL interval it will mark it as failed and remove it from queries results.
Side note: Consul has also simple port checks when Consul agent will judge the health of the service based on the port availability. While it’s much simpler, e.g. you don’t have to add anything to your code, it’s not that powerful as a TTL check. With TTL checks you can inspect internal state of your service which is a huge advantage in comparison with simple availability – you can accept queries but your data may be stale or invalid. Also, with TTL checks service status can be not only in binary state – good/bad – but also with a warning.
All right, to the point! The “v1” version of webkv uses only the standard
library and the bare minimum of dependencies like Redis client and Consul API
lib. Later I’m going to extend it with other niceties like Prometheus
integration, structured logging, and sane configuration management.
Let’s start with a basic web service that will serve key-value data from Redis.
First, parse port, ttl, and addrs commandline flags. The last one is the
list of Redis addresses separated with ;.
func main() {
port := flag.Int("port", 8080, "Port to listen on")
addrsStr := flag.String("addrs", "", "(Required) Redis addrs (may be delimited by ;)")
ttl := flag.Duration("ttl", time.Second*15, "Service TTL check duration")
flag.Parse()
if len(*addrsStr) == 0 {
fmt.Fprintln(os.Stderr, "addrs argument is required")
flag.PrintDefaults()
os.Exit(1)
}
addrs := strings.Split(*addrsStr, ";")
Now, we create a service that should implement Handler interface and
launch it.
s, err := service.New(addrs, *ttl)
if err != nil {
log.Fatal(err)
}
http.Handle("/", s)
l := fmt.Sprintf(":%d", *port)
log.Print("Listening on ", l)
log.Fatal(http.ListenAndServe(l, nil))
Nothing fancy here. Now let’s look at the service itself.
import (
"time"
"github.com/go-redis/redis"
)
type Service struct {
Name string
TTL time.Duration
RedisClient redis.UniversalClient
}
The Service is a type that holds a name, TTL and Redis client handler. It’s
instantiated like this:
func New(addrs []string, ttl time.Duration) (*Service, error) {
s := new(Service)
s.Name = "webkv"
s.TTL = ttl
s.RedisClient = redis.NewUniversalClient(&redis.UniversalOptions{
Addrs: addrs,
})
ok, err := s.Check()
if !ok {
return nil, err
}
return s, nil
}
Check method issues PING Redis command to check if we’re ok. This will be
used later with Consul registration.
func (s *Service) Check() (bool, error) {
_, err := s.RedisClient.Ping().Result()
if err != nil {
return false, err
}
return true, nil
And now the implementation of ServeHTTP method that will be invoked for
request processing:
func (s *Service) ServeHTTP(w http.ResponseWriter, r *http.Request) {
status := 200
key := strings.Trim(r.URL.Path, "/")
val, err := s.RedisClient.Get(key).Result()
if err != nil {
http.Error(w, "Key not found", http.StatusNotFound)
status = 404
}
fmt.Fprint(w, val)
log.Printf("url=\"%s\" remote=\"%s\" key=\"%s\" status=%d\n",
r.URL, r.RemoteAddr, key, status)
}
Basically, what we do is retrieve URL path from request and use it as a key for Redis “GET” command. After that we return the value or 404 in case of an error. Last, we log the request with a quick and dirty structured logging message in logfmt format.
Launch it:
$ ./webkv -addrs 'localhost:6379'
2017/12/13 21:44:15 Listening on :8080
Query it:
$ curl 'localhost:8080/blink'
182
And see the log message:
2017/12/13 21:44:29 url="/blink" remote="[::1]:35020" key="blink" status=200
Now let’s make our service discoverable via Consul. Consul has simple HTTP API to register services that you can employ directly via “net/http” but we will use its Go library.
Consul Go library doesn’t have examples, BUT, it has tests! Tests are nice not only because it gives you confidence in your lib, approval for the sanity of your code structure and API and, finally, a set of usage examples. Here is an example from Consul API test suite for service registration and TTL checks.
Looking at these tests, we can tell that we interact with Consul by creating a
Client and then getting a handle for the particular endpoint like
/agent or /kv. For each endpoint, there is a corresponding Go type. Agent
endpoint is responsible for service registration and sending health checks. To
store an Agent handle we extend our Service type with a new pointer:
import (
consul "github.com/hashicorp/consul/api"
)
type Service struct {
Name string
TTL time.Duration
RedisClient redis.UniversalClient
ConsulAgent *consul.Agent
}
Next in the Service “constructor” we add the creation of Consul agent handle:
func New(addrs []string, ttl time.Duration) (*Service, error) {
...
c, err := consul.NewClient(consul.DefaultConfig())
if err != nil {
return nil, err
}
s.ConsulAgent = c.Agent()
Next, we use the agent to register our service:
serviceDef := &consul.AgentServiceRegistration{
Name: s.Name,
Check: &consul.AgentServiceCheck{
TTL: s.TTL.String(),
},
}
if err := s.ConsulAgent.ServiceRegister(serviceDef); err != nil {
return nil, err
}
The key thing here is the Check part where we tell Consul how it should check
our service. In our case, we say that we ourselves will send heartbeat-like
signals to Consul so that it will mark our service failed after TTL. Failed
service is not returned as part of DNS or HTTP API queries.
After service is registered we have to send a TTL check signal with Pass, Fail or Warn type. We have to send it periodically and in time to avoid service failure by TTL. We’ll do it in a separate goroutine:
go s.UpdateTTL(s.Check)
UpdateTTL method uses time.Ticker to periodically invoke the
actual update function:
func (s *Service) UpdateTTL(check func() (bool, error)) {
ticker := time.NewTicker(s.TTL / 2)
for range ticker.C {
s.update(check)
}
}
check argument is a function that returns a service status. Based on its
result we send either pass or fail check:
func (s *Service) update(check func() (bool, error)) {
ok, err := check()
if !ok {
log.Printf("err=\"Check failed\" msg=\"%s\"", err.Error())
if agentErr := s.ConsulAgent.FailTTL("service:"+s.Name, err.Error()); agentErr != nil {
log.Print(agentErr)
}
} else {
if agentErr := s.ConsulAgent.PassTTL("service:"+s.Name, ""); agentErr != nil {
log.Print(agentErr)
}
}
}
Check function that we pass to goroutine is the one we used earlier on creating service, it just returns bool status of Redis PING command.
And that’s it! This is how it all works together:
webkvTo see it in action you need to launch a Consul and Redis. You can launch Consul
with consul agent -dev or start a normal cluster. How to launch Redis depends
on your distro, in my Fedora it’s just systemctl start redis.
Now launch the webkv like this:
$ ./webkv -addrs localhost:6379 -port 8888
2017/12/14 19:00:29 Listening on :8888
Query the Consul for services:
$ dig +noall +answer @127.0.0.1 -p 8600 webkv.service.dc1.consul
webkv.service.dc1.consul. 0 IN A 127.0.0.1
$ curl localhost:8500/v1/health/service/webkv?passing
[
{
"Node": {
"ID": "a4618035-c73d-9e9e-2b83-24ece7c24f45",
"Node": "alien",
"Address": "127.0.0.1",
"Datacenter": "dc1",
"TaggedAddresses": {
"lan": "127.0.0.1",
"wan": "127.0.0.1"
},
"Meta": {
"consul-network-segment": ""
},
"CreateIndex": 5,
"ModifyIndex": 6
},
"Service": {
"ID": "webkv",
"Service": "webkv",
"Tags": [],
"Address": "",
"Port": 0,
"EnableTagOverride": false,
"CreateIndex": 15,
"ModifyIndex": 37
},
"Checks": [
{
"Node": "alien",
"CheckID": "serfHealth",
"Name": "Serf Health Status",
"Status": "passing",
"Notes": "",
"Output": "Agent alive and reachable",
"ServiceID": "",
"ServiceName": "",
"ServiceTags": [],
"Definition": {},
"CreateIndex": 5,
"ModifyIndex": 5
},
{
"Node": "alien",
"CheckID": "service:webkv",
"Name": "Service 'webkv' check",
"Status": "passing",
"Notes": "",
"Output": "",
"ServiceID": "webkv",
"ServiceName": "webkv",
"ServiceTags": [],
"Definition": {},
"CreateIndex": 15,
"ModifyIndex": 141
}
]
}
]
Now if we stop the Redis we’ll see the log messages
...
2017/12/14 19:29:19 err="Check failed" msg="EOF"
2017/12/14 19:29:27 err="Check failed" msg="dial tcp [::1]:6379: getsockopt: connection refused"
...
and that Consul doesn’t return our service:
$ dig +noall +answer @127.0.0.1 -p 8600 webkv.service.dc1.consul
$ # empty reply
$ curl localhost:8500/v1/health/service/webkv?passing
[]
Starting Redis again will make service healthy.
So, basically this is it – the basic Web service with Consul integration for service discovery and health checking. Check out the full source code at github.com/alexdzyoba/webkv. Next time we’ll add metrics export for monitoring our service with Prometheus.