pvdeg.geospatial.start_dask

pvdeg.geospatial.start_dask(hpc=None)

Start a dask cluster for parallel processing.

Parameters:

hpc (dict) – Dictionary containing dask hpc settings (see examples below).

Examples

Local cluster:

hpc = {'manager': 'local',
       'n_workers': 1,
       'threads_per_worker': 8,
       'memory_limit': '10GB'}

SLURM cluster:

kestrel = {
    'manager': 'slurm',
    'n_jobs': 1,  # Max number of nodes used for parallel processing
    'cores': 104,
    'memory': '246GB',
    'account': 'pvsoiling',
    'walltime': '4:00:00',
    'processes': 52,
    'local_directory': '/tmp/scratch',
    'job_extra_directives': ['-o ./logs/slurm-%j.out'],
    'death_timeout': 600,}

Returns:

client (dask.distributed.Client) – Dask client object.