Job
data class JobPysparkConfigArgs(val archiveUris: Output<List<String>>? = null, val args: Output<List<String>>? = null, val fileUris: Output<List<String>>? = null, val jarFileUris: Output<List<String>>? = null, val loggingConfig: Output<JobPysparkConfigLoggingConfigArgs>? = null, val mainPythonFileUri: Output<String>, val properties: Output<Map<String, String>>? = null, val pythonFileUris: Output<List<String>>? = null) : ConvertibleToJava<JobPysparkConfigArgs>
Constructors
Link copied to clipboard
constructor(archiveUris: Output<List<String>>? = null, args: Output<List<String>>? = null, fileUris: Output<List<String>>? = null, jarFileUris: Output<List<String>>? = null, loggingConfig: Output<JobPysparkConfigLoggingConfigArgs>? = null, mainPythonFileUri: Output<String>, properties: Output<Map<String, String>>? = null, pythonFileUris: Output<List<String>>? = null)
Properties
Link copied to clipboard
HCFS URIs of archives to be extracted in the working directory of .jar, .tar, .tar.gz, .tgz, and .zip.
Link copied to clipboard
HCFS URIs of jar files to add to the CLASSPATHs of the Python driver and tasks.
Link copied to clipboard
Link copied to clipboard
The HCFS URI of the main Python file to use as the driver. Must be a .py file.
Link copied to clipboard
A mapping of property names to values, used to configure PySpark. Properties that conflict with values set by the Cloud Dataproc API may be overwritten. Can include properties set in /etc/spark/conf/spark-defaults.conf and classes in user code.
Link copied to clipboard
HCFS file URIs of Python files to pass to the PySpark framework. Supported file types: .py, .egg, and .zip.