Hadoop V2 - Caching and Cache Directives

In this blog I discuss on how to configure hdfs caching and few caching related commands.

All the steps are as hdfs user (member of supergroup)

Step 1 - Add Pool
hdfs cacheadmin -addPool Finance

[hdfs@namenode ~]$ hdfs cacheadmin -addPool Finance
Successfully added cache pool Finance.

Step 2 - Add Directive
This is the point where you add a path/file to the pool to have it cached.

hdfs cacheadmin -addDirective -path /user -pool Finance

[hdfs@namenode ~]$ hdfs cacheadmin -addDirective -path /user -pool Finance
Added cache directive 1

Step 3 - List available pools

hdfs cacheadmin -listPools
[hdfs@namenode ~]$ hdfs cacheadmin -listPools
Found 2 results.
NAME        OWNER  GROUP   MODE            LIMIT  MAXTTL
Finance     hdfs   hadoop  rwxr-xr-x   unlimited   never
Marketting  hdfs   hadoop  rwxr-xr-x   unlimited   never


Step 4 - Verify Stats of the pool
hdfs cacheadmin -listPools -stats Finance

[hdfs@namenode ~]$ hdfs cacheadmin -listPools -stats Finance
Found 1 result.
NAME     OWNER  GROUP   MODE            LIMIT  MAXTTL  BYTES_NEEDED  BYTES_CACHED  BYTES_OVERLIMIT  FILES_NEEDED  FILES_CACHED
Finance  hdfs   hadoop  rwxr-xr-x   unlimited   never          1573             0                0             1             0


Step 5  - Verify Cache Statistics
hdfs dfsadmin -report | grep -i -e Cache -e Name

[hdfs@namenode ~]$ hdfs dfsadmin -report | grep -i -e Cache -e Name
Name: 192.168.10.54:50010 (d1node.cluster.com)
Hostname: d1node.cluster.com
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Name: 192.168.10.55:50010 (d2node.cluster.com)
Hostname: d2node.cluster.com
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Name: 192.168.10.58:50010 (d3node.cluster.com)
Hostname: d3node.cluster.com
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%