2017

Docker Usage

Run a MySQL Container

docker run --name={name} -p {host port}:3306 -e MYSQL_ROOT_PASSWORD={mysql root password} -e MYSQL_ROOT_HOST=localhost -d mysql/mysql-server:{mysql version} --config-entry-key=value

• Check container logs docker logs {name}
• Restore from dump files cat {dump file} | docker exec -i {name} mysql -u{user} -p{pass} {database}
• Enter MySQL console docker exec -it {name} mysql -u{user} -p{pass}

Using HTTP/2 Client in Java 9

To use HTTP/2 in Java 9, it's required to declare the dependency of module jdk.incubator.httpclient. Here is the Java 9 module overview summary.

Assume the root package is com.jdk9.http.

• To use module jdk.incubator.httpclient module, create module-info.java file with content:

module com.jdk9.http {
    requires jdk.incubator.httpclient;
}

PostgreSQL backup & restore example

Backup

pg_dump --file ${backup file} --port "5432" --format=c --blobs --schema ${schema} ${database} -U ${user}

Restore

pg_restore -U ${user} --exit-on-error --verbose --dbname=${database} ${backup file}

PostgreSQL Basic Configurations After Installation

Allow all connections

Edit /etc/postgresql/${version}/main/pg_hba.conf, add host all all 0.0.0.0/0 md5 in section IPv4 local connections

Listen on all IP addresses

Edit /etc/postgresql/${version}/main/postgres.conf, in section Connection Settings, uncomment listen_addresses = 'localhost', change the localhost to *

Create a superuser

Run in shell, 1. su - postgres 2. createuser -P -s -e <username>, will prompt to input the password for the user

Spark Cluster Configuration

Following the Hadoop configurations.

In $SPARK_HOME/conf:

• Configure spark-env.sh
  

cp spark-env.sh.template spark-env.sh and add following:

PYSPARK_DRIVER_PYTHON=python3
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_MASTER_HOST=$Master IP
export SPARK_WORKER_PORT=20001

Invoke Script Functions and Methods via JavaScript Engine

List supported script engines

ScriptEngineManager manager = new ScriptEngineManager();
for(ScriptEngineFactory factory : manager.getEngineFactories()){
    System.out.printf("Name: %s%n" +
                    "\tVersion: %s%n" +
                    "\tLanguage name: %s%n" +
                    "\tLanguage version: %s%n" +
                    "\tExtensions: %s%n" +
                    "\tMime types: %s%n" +
                    "\tNames: %s%n",
            factory.getEngineName(),
            factory.getEngineVersion(),
            factory.getLanguageName(),
            factory.getLanguageVersion(),
            factory.getExtensions(),
            factory.getMimeTypes(),
            factory.getNames());
}

Hadoop Cluster Configuration

Environment Configuration

User configuration

• Create hadoop user sudo useradd -m hadoop -s /bin/bash
• Set password for user 'hadoop' sudo passwd hadoop
• Add hadoop user into sudoers sudo adduser hadoop sudo

You Need configure hadoop user on all related servers

Utility to debug Java https connection

If the certificate of the host is not valid (self signed) or didn't well configured in java keystore, then it will fail to connect.

Linux Sed Command

Basic Usage

• replace a string in a file

  sed -i "s/<string to be replaced>/<string will be replaced to>/g" $file

  You also can escape special character with a backslash