Myujth

Edit, before trying anything, make sure you install Snakemake with:

conda install -c bioconda -c conda-forge snakemake

As advertised here: snakemake.readthedocs.io. Don't install it as advertised here: anaconda.org/bioconda/snakemake, you will end up with a very old version (the -c conda-forge is important!)

Original post =>

I have been wrestling with Snakemake today. My problem is that my STAR rule gives me an error:

/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/activate-binutils_linux-64.sh: line 67: HOST: unbound variable
Error in job star_map while creating output file /rst1/2017-0205_illuminaseq/scratch/swo-406/preprocessing/180413_NB501997_0054_AHTFJ3BGX3/0054_P2018SEQE15S4_S14.Aligned.out.bam.
RuleException:
CalledProcessError in line 50 of /home/nlv24077/experiments/experiments/swo-406/scripts/Snakefile.snakefile:
Command '
 source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
 STAR --runThreadN 8 --genomeDir /rst1/2017-0205_illuminaseq/scratch/swo-390/STAR_references/human --readFilesIn /rst1/2017-0205_illuminaseq/scratch/swo-406/fastq/180413_NB501997_0054_AHTFJ3BGX3/0054_P2018SEQE15S4_S14_R1_001.fastq.gz /rst1/2017-0205_illuminaseq/scratch/swo-406/fastq/180413_NB501997_0054_AHTFJ3BGX3/0054_P2018SEQE15S4_S14_R2_001.fastq.gz --outSAMtype BAM Unsorted --readFilesCommand zcat --outFileNamePrefix /rst1/2017-0205_illuminaseq/scratch/swo-406/preprocessing/180413_NB501997_0054_AHTFJ3BGX3/0054_P2018SEQE15S4_S14.
 ' returned non-zero exit status 1.
 File "/home/nlv24077/experiments/experiments/swo-406/scripts/Snakefile.snakefile", line 50, in __rule_star_map
 File "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Exiting because a job execution failed. Look above for error message

However, when I just copy that script/command into a terminal, it works.

This is my snakefile:

import os
from glob import glob
#from snakemake.utils import validate

configfile: 'config.yaml'
#validate(config, "config.schema.yaml")

# Set the working directory
workdir: config['workdir']

experiment_name = 'swo-406'
scratch_data_base_dir="/rst1/2017-0205_illuminaseq/scratch"
scratch_data_dir = os.path.join(scratch_data_base_dir, experiment_name)

seqrun = '180413_NB501997_0054_AHTFJ3BGX3'
fastq_dir = os.path.join(scratch_data_dir, 'fastq', seqrun)
preprocessing_dir = os.path.join(scratch_data_dir, 'preprocessing', seqrun)
quantification_dir = os.path.join(scratch_data_dir, 'quantification', seqrun)
if not os.path.isdir(preprocessing_dir):
 os.makedirs(preprocessing_dir)

#ref_base_dir = config[ref_base_dir]
ref_genome = os.path.join(config['ref_base_dir'], config['ref_genome'])
star_ref_dir = config['star_ref_dir']

## Rsem settings
rsem_ref_dir = os.path.join(scratch_data_base_dir, 'swo-387', 'RSEM_references')
rsem_ref_base = os.path.join(rsem_ref_dir, 'Homo_sapiens.GRCh38')

log = os.path.join(preprocessing_dir, 'log.txt')

SAMPLES = set([os.path.basename(fastq_file.replace('_R1_001.fastq.gz', '').replace('_R2_001.fastq.gz', ''))
 for fastq_file in glob(os.path.join(fastq_dir, '*_R*_001.fastq.gz'))
 if not 'Undetermined' in fastq_file])

#star_output_prefix = os.path.join(preprocessing_dir, 'sample.')

# Rule all is a pseudo-rule that tells snakemake what final files to generate.
rule all:
 input:
 expand(os.path.join(quantification_dir, 'sample'), sample=SAMPLES)

rule star_map:
 input:
 os.path.join(fastq_dir, 'sample_R1_001.fastq.gz'),
 os.path.join(fastq_dir, 'sample_R2_001.fastq.gz'),
 output:
 os.path.join(preprocessing_dir, 'sample.Aligned.out.bam')
 shell:
 """
 source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
 STAR 
 --runThreadN 8 
 --genomeDir star_ref_dir 
 --readFilesIn input 
 --outSAMtype BAM Unsorted 
 --readFilesCommand zcat 
 --outFileNamePrefix preprocessing_dir/wildcards.sample.
 """

rule samtools_sort:
 input:
 os.path.join(preprocessing_dir, 'sample.Aligned.out.bam')
 output:
 os.path.join(preprocessing_dir, 'sample.Aligned.out.sorted.bam')
 shell:
 """
 source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
 samtools sort -T wildcards.sample -O bam input > output
 """


rule rsem_quantify:
 input:
 os.path.join(preprocessing_dir, 'sample.Aligned.out.sorted.bam')
 output:
 os.path.join(quantification_dir, 'sample')
 shell:
 """
 source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
 rsem-calculate-expression 
 --paired-end 
 --bam 
 --num-threads 8 
 --strandedness reverse 
 rsem_ref_base 
 output
 """

Can anyone spot the error?
By the way, I have to comment out

validate(config, "config.schema.yaml")

Because my snakemake.utils does not seem to have a "validate":

(/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake) 16:40 nlv24077@kavia /rst1/2017-0205_illuminaseq/scratch/swo-406 > python3
Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 19:16:44) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from snakemake.utils import validate
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: cannot import name 'validate'
>>> 

Highest regards,

Freek.

Could you remove all the source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemakecommands from your shell portions of the different rules in the Snakefile and activate the environment either:

1. run command source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake before you actually run snakemake on that Snakefile (you can even add a version of snakemake that has validate to this environment). So you can run source activate /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake and then run snakemake.




2. Create a conda environment file matching that environment and add the conda : path/to/created/env/fileparameter in your rules requiring the environment. Then run snakemake with the --use-conda flag

Since you are using the same environment for all your rules, it is better to use option 1 as option 2 is much slower and will make it unecessarily rule specific.

I can reproduce your error with this example Snakefile:

rule test_activate:
 output : "test.txt"
 shell: "source activate NGS && conda list > output"

I get the same unbound variable error but for a different variable as my environment is different. This is an explanation of what might be going on:

Virtualenv activate script won't run in bash script with set -euo

in the sense that once you run it through snakemake vs terminal some variables become unbound which is treated as error.