Compute Jobs

A job consists of a name and JavaScript which defines the input and stage definitions, and various other metadata for executing the job.

Creating Jobs

Select the "Jobs" link in the menu.
Click the "NEW JOB" button to create a new job.
Enter the name and JS for the job.
Click the "SAVE" button to save the job.

The basic JS template for a job is the following:

// create a new `JobExectution`
const execution = new JobExecution();

// defina a fn that the coordinator will execute. This fn should returns a promise 
// that resolves to an array. The array will be split up among the worker nodes
// and processed in parallel.
execution.setInputFn(async function input() {
});

// add a new map stage
execution.addStage(new MapStage(async function map(x) {
}));

// add a new reduce stage
execution.addStage(new ReduceStage(async function map(keyToValues) {
}));

// start the JobExecution
execution.start(jobRunner);

Job Execution Metadata

Metadata for a job execution gives PeerMR information needed to run the job. The metadata includes:

storageType: One of: 'pmrfs', 'gcs', or 's3'. Defines where to store for intermediate and final outputs.
workerCount: How many workers will run the job. Default worker count is 4. The maximum worker count is 8192.
groupCount: How many groups to run at a time. Default is 1. Groups can be used for A/B testing with up to 26 variations per job.
scripts: Third-party scripts required by the stages to run.
wasmBinaries: URLs of WebAssembly binaries required by the stages to run.
privateMode: If true, the job will run in private mode. Private mode is useful for debugging and testing jobs as well as keeping data private to only workers you've connected to your coordinator account.
gpu: If true, the job will only run on web browsers with WebGPU support.
gpuRequirements: a map of GPU requirements for the job. The map includes the following three keys:
- info: // string to string map of WebGPU adapter info required for the job
- features: // string to boolean map of WebGPU adapter features required for the job
- limits: // string to number map of WebGPU adapter minimums required for the job
- for example, to require an Apple M3 Max GPU that supports ASTC texture compression and a minimum max buffer size of 4294967296 bytes:
```
  {
    info: {
      'vendor': 'apple',
      'description': 'Apple M3 Max'
    },
    features: {
      'texture-compression-astc': true
    },
    limits: {
      'maxBufferSize': 4294967296
    }
  }
```
  For more information please refer to:
- https://developer.mozilla.org/en-US/docs/Web/API/GPUAdapterInfo
- https://developer.mozilla.org/en-US/docs/Web/API/GPUAdapter/features
- https://developer.mozilla.org/en-US/docs/Web/API/GPUAdapter/limits
saveIntermediateOutput: If true, intermediate outputs for PMRFS jobs will not be deleted after a job runs. Defaults to true. Is ignored for GCS and S3 jobs whose intermediate outputs are never deleted by PeerMR.

Submitting Jobs

Click the "RUN" button. This opens a new dialog where you can confirm the job execution metadata prior to running the job.
Confirm the job execution metadata and click the "RUN" button to submit the job.

After confirming a job run, the job is put in a globally shared job queue for processing. Once the job is at the front of the queue, it is selected for possible execution. If there are enough workers that match the job's requirements, the job will be executed. If not, the job will stay at the front of the queue. The browser that the job was queued the job is the coordinator for the job. The coordinator must stay connected to PeerMR for the job to run. If the coordinator is not connected, the job will not be started.

Private Mode

If the job execution's privateMode flag is true then only browsers that are connected to PeerMR with your coordinator account are eligible to be selected as workers. Only WebRTC relay traffic is billed in private mode, and no billing will be incurred if all of your workers are able to connect to each other without a relay server. Private mode is useful for testing your jobs before running them on the larger pool of public workers. Once you are confident that your job is working as expected, you can disable private mode and run the job in public mode. If privateMode is false then any browser connected to PeerMR is eligible to be selected as a worker provided it meets the job's gpu requirements (if any).

GPU Requirements

If the job execution's gpu flag is true then only browsers with WebGPU support are eligible to be selected as workers. Use gpuRequirements to specify the minimum WebGPU adapter requirements if your job needs certain levels of performance from GPU workers. If you do not specify gpu then PeerMR may still schedule your job on browsers with WebGPU support, but it is not guaranteed. You can check for WebGPU support at runtime in your worker map and reduce functions.

PeerMR