Blog

Aug 24, 2017

Spark Cluster Configuration

In $SPARK_HOME/conf:

Configure spark-env.sh

cp spark-env.sh.template spark-env.sh and add following:

Text Only

PYSPARK_DRIVER_PYTHON=python3
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_MASTER_HOST=$Master IP
export SPARK_WORKER_PORT=20001

Aug 22, 2017

Invoke Script Functions and Methods via JavaScript Engine

List supported script engines

Java

ScriptEngineManager manager = new ScriptEngineManager();
for(ScriptEngineFactory factory : manager.getEngineFactories()){
    System.out.printf("Name: %s%n" +
                    "\tVersion: %s%n" +
                    "\tLanguage name: %s%n" +
                    "\tLanguage version: %s%n" +
                    "\tExtensions: %s%n" +
                    "\tMime types: %s%n" +
                    "\tNames: %s%n",
            factory.getEngineName(),
            factory.getEngineVersion(),
            factory.getLanguageName(),
            factory.getLanguageVersion(),
            factory.getExtensions(),
            factory.getMimeTypes(),
            factory.getNames());
}

Aug 4, 2017

Hadoop Cluster Configuration

Environment Configuration

User configuration

Create hadoop user sudo useradd -m hadoop -s /bin/bash
Set password for user 'hadoop' sudo passwd hadoop
Add hadoop user into sudoers sudo adduser hadoop sudo

You Need configure hadoop user on all related servers

Jul 25, 2017

Utility to debug Java https connection

If the certificate of the host is not valid (self signed) or didn't well configured in java keystore, then it will fail to connect.

Jul 24, 2017

Linux Sed Command

Basic Usage

replace a string in a file

sed -i "s/<string to be replaced>/<string will be replaced to>/g" $file

You also can escape special character with a backslash