SLURMGraph fails

[mfalkiewicz]

Summary

Attempt to submit jobs with SLURMGraph fails

Actual behavior

-bash-4.2$ ./run_fmriprep
180103-15:26:37,49 workflow IMPORTANT:

Running fMRIPREP version 1.0.2:
BIDS dataset path: /homea/inm7/mfalkiew/data/NaKo/BIDS.
Participant list: ['100005'].
Run identifier: 20180103-152636_9b5fa9df-2e8f-4c67-99fb-d3d9f203d1df.

180103-15:26:37,938 workflow IMPORTANT:
Creating bold processing workflow for "/homea/inm7/mfalkiew/data/NaKo/BIDS/sub-100005/func/sub-100005_task-rest_bold.nii.gz" (0.03 GB / 180 TRs).
180103-15:26:38,214 workflow IMPORTANT:
Slice-timing correction will be included.
180103-15:26:38,604 workflow WARNING:
No fieldmaps found or they were ignored, building base workflow for dataset /homea/inm7/mfalkiew/data/NaKo/BIDS/sub-100005/func/sub-100005_task-rest_bold.nii.gz.
180103-15:26:38,773 workflow IMPORTANT:
Creating BOLD surface-sampling workflow.
Traceback (most recent call last):
File "/homea/inm7/mfalkiew/.linuxbrew/bin/fmriprep", line 11, in
sys.exit(main())
File "/homea/inm7/mfalkiew/.linuxbrew/opt/python3/lib/python3.6/site-packages/fmriprep/cli/run.py", line 270, in main
fmriprep_wf.run(**plugin_settings)
File "/homea/inm7/mfalkiew/.local/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/workflows.py", line 574, in run
runner.run(execgraph, updatehash=updatehash, config=self.config)
File "/homea/inm7/mfalkiew/.local/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 530, in run
graph.predecessors(node)]
File "/homea/inm7/mfalkiew/.local/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/base.py", line 529, in
dependencies[idx] = [nodes.index(prevnode) for prevnode in
AttributeError: 'generator' object has no attribute 'index'

Expected behavior

Pipeline submitted

How to replicate the behavior

Submit jobs with SLURMGraph plugin

Script/Workflow details

Vanilla fmriprep 1.0.2

Platform details:

-bash-4.2$ python3 -c "import nipype; print(nipype.get_info()); print(nipype.version)"
{'pkg_path': '/homea/inm7/mfalkiew/.local/lib/python3.6/site-packages/nipype', 'commit_source': 'installation', 'commit_hash': 'bdb7afc', 'nipype_version': '0.14.0', 'sys_version': '3.6.4 (default, Jan 2 2018, 15:26:07) \n[GCC 5.5.0]', 'sys_executable': '/homea/inm7/mfalkiew/.linuxbrew/opt/python3/bin/python3.6', 'sys_platform': 'linux', 'numpy_version': '1.13.3', 'scipy_version': '1.0.0', 'networkx_version': '2.0', 'nibabel_version': '2.2.1', 'traits_version': '4.6.0'}
0.14.0

[mfalkiewicz]

Ok, I managed to hack my way through this. It seems that this error emerges because data structures that used to be lists in P2 are generators in P3. Following hacks solved the problem:

nodes = nx.topological_sort(graph)
changed to:
nodes = list(nx.topological_sort(graph))

and

dependencies[idx] = [nodes.index(prevnode) for prevnode in graph.predecessors(node)]
to
dependencies[idx] = [nodes.index(prevnode) for prevnode in list(graph.predecessors(node))]

However, I do realize that this is just a hack. Does anyone know a proper P3-way of looking at indices in generators?

[chrisgorgo]

I did some digging and it seems that casting to list is the way to go. Could you submit a patch with those changes?

[mfalkiewicz]

@chrisfilo: Sure I can, but I am not sure if this won't break the P2 compatibility. For example, if nx.topological_sort(graph) returns a list [a, b, c], then the hacks I proposed will create [[a,b,c]]. Is that the desired behavior? If so, I will submit a relevant patch.

[mgxd]

@mfalkiewicz feel free to submit the PR - if these changes causes issues with py2 we have tests in place to catch them.