Hadoop V2 - FSCK

FSCK is one of the key utilized and monitoring command for monitoring hdfs

FSCK
1. Similar to Linux fsck, finds out block corruptions and issue with the File System
2. Does changes only when told, and only checks by default
3. Is a metadata only operation i.e. only Namenode FSImage is processed.
4. can be run on complete FS or specific directory
5. Should be run as superuser only.

Options -
fsck /: Performs an HDFS file system check
fsck / -files: Displays files being checked
fsck / -files –blocks: Displays files and blocks
fsck / files –blocks –locations: Displays files, blocks and their locations
fsck / -files –blocks –locations –racks: Displays files, blocks,
locations and racks
fsck –locations: Shows locations for every block
fsck – move: Moves corrupted files to the /lost+found directory
fsck –delete: Deletes corrupt files
fsck –list-corruptfileblocks: Lists missing blocks and the files they belong to

Example - Running FSCK for a directory and display everything

hdfs fsck  /data/conf -files -blocks -locations -racks
.
.
.
/data/conf/yarn-env.sh 4760 bytes, 1 block(s):  OK
0. BP-1256862429-192.168.2.101-1523506309307:blk_1073741860_1036 len=4760 repl=3 [/default-rack/192.168.2.104:50010, /default-rack/192.168.2.105:50010, /default-rack/192.168.2.106:50010]

Status: HEALTHY
 Total size:    78390 B
 Total dirs:    1
 Total files:   31
 Total symlinks:                0
 Total blocks (validated):      30 (avg. block size 2613 B)
 Minimally replicated blocks:   30 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Mon Apr 16 02:01:20 EDT 2018 in 3 milliseconds


The filesystem under path '/data/conf' is HEALTHY