How to: Big Data & NoSQL
`bigdata/` and `kafka/` directories:

- `kafka_*.sh` - scripts to make [Kafka](http://kafka.apache.org/) CLI usage easier including auto-setting Kerberos to source TGT from environment and auto-populating broker and zookeeper addresses. These are auto-added to the `$PATH` when `.bashrc` is sourced. For something similar for [Solr](https://lucene.apache.org/solr/), see `solr_cli.pl` in the [DevOps Perl Tools](https://github.com/HariSekhon/DevOps-Perl-tools) repo.
- `zookeeper*.sh` - [Apache ZooKeeper](https://zookeeper.apache.org/) scripts:
  - `zookeeper_client.sh` - shortens `zookeeper-client` command by auto-populating the zookeeper quorum from the environment variable `$ZOOKEEPERS` or else parsing the zookeeper quorum from `/etc/**/*-site.xml` to make it faster and easier to connect
  - `zookeeper_shell.sh` - shortens Kafka's `zookeeper-shell` command by auto-populating the zookeeper quorum from the environment variable `$KAFKA_ZOOKEEPERS` and optionally `$KAFKA_ZOOKEEPER_ROOT` to make it faster and easier to connect
- `hive_*.sh` / `beeline*.sh` - [Apache Hive](https://hive.apache.org/) scripts:
  - `beeline.sh` - shortens `beeline` command to connect to [HiveServer2](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview) by auto-populating Kerberos and SSL settings, zookeepers for HiveServer2 HA discovery if the environment variable `$HIVE_HA` is set or using the `$HIVESERVER_HOST` environment variable so you can connect with no arguments (prompts for HiveServer2 address if you haven't set `$HIVESERVER_HOST` or `$HIVE_HA`)
    - `beeline_zk.sh` - same as above for [HiveServer2](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview) HA by auto-populating SSL and ZooKeeper service discovery settings (specify `$HIVE_ZOOKEEPERS` environment variable to override). Automatically called by `beeline.sh` if either `$HIVE_ZOOKEEPERS` or `$HIVE_HA` is set (the latter parses `hive-site.xml` for the ZooKeeper addresses)
  - `hive_foreach_table.sh` - executes a SQL query against every table, replacing `{db}` and `{table}` in each iteration eg. `select count(*) from {table}`
  - `hive_list_databases.sh` - list Hive databases, one per line, suitable for scripting pipelines
  - `hive_list_tables.sh` - list Hive tables, one per line, suitable for scripting pipelines
  - `hive_tables_metadata.sh` - lists a given DDL metadata field for each Hive table (to compare tables)
  - `hive_tables_location.sh` - lists the data location per Hive table (eg. compare external table locations)
  - `hive_tables_row_counts.sh` - lists the row count per Hive table
  - `hive_tables_column_counts.sh` - lists the column count per Hive table
- ` impala*.sh` - [Apache Impala](https://impala.apache.org/) scripts:
  - `impala_shell.sh` - shortens `impala-shell` command to connect to [Impala](https://impala.apache.org/) by parsing the Hadoop topology map and selecting a random datanode to connect to its Impalad, acting as a cheap CLI load balancer. For a real load balancer see [HAProxy config for Impala](https://github.com/HariSekhon/HAProxy-configs) (and many other Big Data & NoSQL technologies). Optional environment variables `$IMPALA_HOST` (eg. point to an explicit node or an HAProxy load balancer) and `IMPALA_SSL=1` (or use regular impala-shell `--ssl` argument pass through)
  - `impala_foreach_table.sh` - executes a SQL query against every table, replacing `{db}` and `{table}` in each iteration eg. `select count(*) from {table}`
  - `impala_list_databases.sh` - list Impala databases, one per line, suitable for scripting pipelines
  - `impala_list_tables.sh` - list Impala tables, one per line, suitable for scripting pipelines
  - `impala_tables_metadata.sh` - lists a given DDL metadata field for each Impala table (to compare tables)
  - `impala_tables_location.sh` - lists the data location per Impala table (eg. compare external table locations)
  - `impala_tables_row_counts.sh` - lists the row count per Impala table
  - `impala_tables_column_counts.sh` - lists the column count per Impala table
- `hdfs_*.sh` - Hadoop [HDFS](https://en.wikipedia.org/wiki/Apache_Hadoop#Hadoop_distributed_file_system) scripts:
  - `hdfs_checksum*.sh` - walks an HDFS directory tree and outputs HDFS native checksums (faster) or portable externally comparable CRC32, in serial or in parallel to save time
  - `hdfs_find_replication_factor_1.sh` / `hdfs_set_replication_factor_3.sh` - finds HDFS files with replication factor 1 / sets HDFS files with replication factor <=2 to replication factor 3 to repair replication safety and avoid no replica alarms during maintenance operations (see also Python API version in the [DevOps Python Tools](https://github.com/HariSekhon/DevOps-Python-tools) repo)
  - `hdfs_file_size.sh` / `hdfs_file_size_including_replicas.sh` - quickly differentiate HDFS files raw size vs total replicated size
  - `hadoop_random_node.sh` - picks a random Hadoop cluster worker node, like a cheap CLI load balancer, useful in s