spark-submit 옵션 관련

Spark Submit의 옵션들에 대해서 알아보자.

$SPARK_HOME/bin/spark-submit --class "Main" \

    --master spark://acet.pe.kr:7077 \

    --executor-memory 4G \

    --total-executor-cores 25 \

    --conf spark.driver.memory=2G \

    --properties-file $CONF_NAME \

    --conf spark.driver.extraJavaOptions='-Xms1024m -Xmx2048m' \

    /home/acet/service/hahaha/good-dev-assembly-1.0.jar

---

--master  // 스파크 마스터의 URL을 적어준다.

The cluster manager to connect to. See the list of allowed master URL's.

--executor-memory  // executor process 마다 사용되어지는 메모리의 양이다.

Amount of memory to use per executor process (e.g. 2g, 8g).

--total-executor-cores // 전체 executor process 코어 수.

--conf spark.driver.memory // driver process가 사용되어지는 메모리 양.

Amount of memory to use for the driver process, i.e. where SparkContext is initialized. (e.g. 1g, 2g). 
Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-memory command line option or in your default properties file.

--properties-file  내가 원하는 파일의 properties를 사용할 수 있음.

--conf spark.driver.extraJavaOptions // 

A string of extra JVM options to pass to the driver. For instance, GC settings or other logging. 
Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file.

참고 URL

http://spark.apache.org/docs/latest/configuration.html#application-properties

http://spark.apache.org/docs/latest/submitting-applications.html