[yedis] Even with redis_allow_reads_from_followers set to true, reads are forwarded to leaders

[kmuthukk]

Rough steps to repro.

1. Create a 3 node pseudo-distributed cluster on a single node using yb-ctl. We create the three nodes in three AZs (although this may or may not be relevant to reproducing this issue).

cd ~/yugabyte-1.0.2.0/
./bin/yb-ctl destroy

./bin/yb-ctl --replication_factor 3 create --placement_info mycloud.mydc.az1,mycloud.mydc.az2,mycloud.mydc.az3 --tserver_flags=redis_allow_reads_from_followers=true

./bin/yb-ctl setup_redis

2. Load 1000 keys using RedisKeyValue sample app:

java -jar ./yb-sample-apps.jar --workload RedisKeyValue --nodes 127.0.0.3:6379 --num_unique_keys 1000 --num_writes 1000 --nouuid --num_threads_read 0

3. Given that this is a 3-node, RF=3 cluster, all nodes should contain all the tablets. When running he read test with just one of the nodes in the --nodes list, one would expect all the reads to get served from the local node (because the gflag redis_allow_reads_from_followers is set to TRUE).

Run the sample app in read mode, and just hit one of the nodes:

% java -jar ./yb-sample-apps.jar --workload RedisKeyValue --nodes 127.0.0.3:6379 --max_written_key 999 --read_only --nouuid --num_threads_read 8
...
...
2018-05-30 12:59:09,319 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 19073.06 ops/sec (0.42 ms/op), 193230 total ops  |  Write: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Uptime: 10021 ms |
2018-05-30 12:59:14,324 [INFO|com.yugabyte.sample.common.metrics.MetricsTracker|MetricsTracker] Read: 20137.35 ops/sec (0.40 ms/op), 294015 total ops  |  Write: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Uptime: 15026 ms |
...

When we look at this yb-master UI view (http://127.0.0.1:7000/tablet-servers) which lists the IOPS per tablet server, while the load is running we see that the reads are going to nodes in other AZs as well, and not getting handled at the 127.0.0.3 which has all the tablets (either in leader or follower role).