/
Common Errors running Python and Spark in Parallel computing

Common Errors running Python and Spark in Parallel computing

Error 1:

Permission denied: /umbc/.../conda.sh conda: command not found

Cause: Your script tries to use a conda environment, but:

  • You don’t have permission to source it.

  • conda isn’t available in the path.

Solution: Use module load anaconda3 (or correct module). Avoid sourcing conda.sh directly unless you have access. Or use Python via module without conda.

Error 2:

torchrun: command not found

Cause:
torchrun not in environment path.

Solution: Activate correct Python environment (with PyTorch):

module load anaconda3 source activate torch_env # or correct env

Or install PyTorch if missing. Check or online or this page about how to set up pyVenv virtual environments

PyTorch Distributed Data Parallel (DDP) Job Warning:

Setting OMP_NUM_THREADS to 1 to avoid overload...

Cause:
Default PyTorch DDP behavior to prevent CPU overload.

Solution:
No action required unless you're optimizing performance.
If needed, set manually:

export OMP_NUM_THREADS=2 # or another number based on cores

Job Pending Issue:

Job's account not permitted to use this partition
(high_mem allows is777sp25 not Li_Minwa)

Cause:
Trying to run on a partition not allowed by your group/account.

Solution: Use a permitted partition for example:

--partition=cpu2021 --account=is777sp25

You can either check with: sacctmgr show assoc user=$USER format=account,partition

[user1@chip ~]$ sacctmgr show assoc user=$USER format=account,partition Account Partition ---------- ---------- is789sp25 pi_jianwu

Or using sinfo to see:

[user1@chip ~]$ sinfo CLUSTER: chip-cpu PARTITION AVAIL TIMELIMIT NODES STATE NODELIST 2024 up infinite 4 mix c24-[10,14,39,52] 2024 up infinite 14 alloc c24-[11-12,29-31,33-38,40-41,51] 2024 up infinite 34 idle c24-[01-09,13,15-28,32,42-50] 2021 up infinite 10 mix c21-[01-02,06-10,13-14,18] 2021 up infinite 8 alloc c21-[03-05,11-12,15-17] 2018 up infinite 17 mix c18-[03,06,13-27] 2018 up infinite 3 alloc c18-[04-05,07] 2018 up infinite 22 idle c18-[01-02,08-12,28-42] general up infinite 27 mix c18-[03,06,13-27],c21-[01-02,06-10,13-14,18] general up infinite 11 alloc c18-[04-05,07],c21-[03-05,11-12,15-17] general up infinite 22 idle c18-[01-02,08-12,28-42] CLUSTER: chip-gpu PARTITION AVAIL TIMELIMIT NODES STATE NODELIST gpu* up infinite 1 resv g24-09 gpu* up infinite 19 mix g20-[05-11,13],g24-[01-08,10-12] gpu* up infinite 5 idle g20-[01-04,12]

Which partition is allowed and available to use, from that you can use the available one.

Node Not Available:

srun: Required node not available (down, drained or reserved)

Cause: Requested node or partition is not currently available.

Solution: Try again later or Request a different partition.

Issue 1: srun command error – missing --qos

srun --mem 5120 --partition=high_mem -N1 --time 120 --pty --account=is777sp25 $SHELL
srun: error: Missing valid: '--qos/-q' srun: error:     You must specify a valid QOS for your job.

To solve this issue, add the --qos flag with a valid value when using srun

srun --mem 5120 --partition=high_mem --qos=high --account=is777sp25 -N1 --time=120 --pty $SHELL

srun: error: See this webpage for more details: How to Run Programs on taki

Issue 2: Empty output in Spark Streaming job: While running python file:

spark-submit ./file_dstream_myfile.py localhost 9998

If you get the output files after running the command, and all the (part-00000) are empty, even though you typed some input using:

nc -lk 9998

The possible errors, cause and solution is down below:

Cause

Explanation

Fix

Cause

Explanation

Fix

  1. Port mismatch

Your script may listen to a different port than 9998

Ensure both nc and your Python script use port 9998

  1. No running nc server

You may have started spark-submit before nc -lk 9998

Start nc -lk 9998 first, then run spark-submit

  1. Not enough time

You may have stopped the job too early before it processed any data

Wait 30–60 seconds after typing input in nc

  1. Wrong output path in script

The output directory may be getting overwritten each time or written somewhere else

Check your script to ensure output path is not getting deleted/reused.

Make sure file path is within the directory.

  1. Buffered data not flushed

Spark streaming uses micro-batches

Input may not be enough to trigger a batch. Try repeating input a few times:

(hello\nhello\nworld\n) This is a test Hello world!