NF: AFNI 3dNetCorr as afni.NetCorr #3263

aghayoor · 2020-10-26T18:19:42Z

This is a WIP to afni NetCorr to nipype

aghayoor · 2020-10-26T18:37:06Z

@effigies : Hi Chris, this is WIP commit to add afni NetCorr to nipype.

Everything works well when I call the interface like this:

ncorr = afni.NetCorr()
ncorr.inputs.in_file = '<workflowDir>/bandpass/bold_trimmed_wmcsfreg_bp.nii'
ncorr.inputs.mask = mniBrainMask
ncorr.inputs.in_rois = mniLabels
ncorr.inputs.ts_wb_corr = True
ncorr.inputs.ts_wb_Z = True
ncorr.inputs.fish_z = True
ncorr.inputs.nifti = True
ncorr.inputs.out_file = '<workflowDir>/correlation/sub0.tp1.ncorr'
ncorr.run()

However, when I wrap this interface in a MapNode, the workflow runs with no error, but it does not write the outputs.

ncorr = pe.Node(interface=afni.NetCorr(), name='correlation')
ncorr.inputs.ts_out = True
ncorr.inputs.ts_label = True
ncorr.inputs.ts_wb_strlabel = True
ncorr.inputs.nifti = True
ncorr.inputs.ts_wb_corr = True
ncorr.inputs.ts_wb_Z = True
ncorr.inputs.fish_z = True
ncorr.inputs.out_file = 'sub0.tp1.ncorr'
workflow.connect(bandpass, 'out_file', ncorr, 'in_file')
workflow.connect(inputsSpec, 'inputMNILables', ncorr, 'in_rois')
workflow.connect(inputsSpec, 'inputMNIBrainMask', ncorr, 'mask')

Do you have any advice on this?

effigies · 2020-10-26T18:38:49Z

The outputs should be in the workflow's working directory... Can you show the tree of that directory?

aghayoor · 2020-10-26T18:53:23Z

The outputs should be in the workflow's working directory... Can you show the tree of that directory?

Here is the tree:

I expected to see the output .netcc file and correlation maps in the correlation directory.

effigies · 2020-10-26T19:47:16Z

This may be a case where unused outputs are not saved. Try using the configuration option remove_unnecessary_outputs. See https://miykael.github.io/nipype_tutorial/notebooks/basic_execution_configuration.html.

aghayoor · 2020-10-26T21:31:28Z

This may be a case where unused outputs are not saved. Try using the configuration option remove_unnecessary_outputs. See https://miykael.github.io/nipype_tutorial/notebooks/basic_execution_configuration.html.

Well that was the case here. Thank you!
Is it possible to avoid deleting those files without changing the configuration?

effigies · 2020-10-26T23:01:39Z

If the output is used as the input to another node, then it won't be deleted. While you're debugging a workflow, it's useful to have on, and then go back to the default when you're in production.

aghayoor · 2020-10-27T01:24:38Z

@effigies : Thanks! Can you please review and merge this pull request?

nipype/interfaces/afni/preprocess.py

effigies · 2020-10-27T12:10:09Z

nipype/interfaces/afni/preprocess.py

+        if not isdefined(self.inputs.out_file):
+            prefix = self._gen_fname(self.inputs.in_file, suffix="_netcorr")
+        else:
+            prefix = self.inputs.out_file
+
+        # All outputs should be in the same directory as the prefix
+        out_dir = os.path.dirname(os.path.abspath(prefix))
+
+        outputs["out_matrix"] = (
+            fname_presuffix(prefix, suffix="_000", use_ext=False, newpath=out_dir) + ".netcc"
+        )


What is the output file called? I would expect, if I set netcorr.inputs.out_file, the output file is called exactly that.

@effigies : the netcorr.inputs.out_file defines the prefix for the output.
Afni 3dNetCorr doesn't have actual output options, but it uses the prefix to write out the correlation matrix and correlation maps to the output directory.
This added function finds and returns the convention filename for the output correlation matrix.

I am also interested to return the correlation maps return to a subdirectory in the output directory. The number of these correlation maps depends on the number of regions in the input ROI label map. Do you have any idea how I can those to the output as well?

@effigies : Thanks for the approval.
also, do you have any idea on this?
Please let me know if my question needs more clarification

Ah, sorry, missed this. Assuming these files exist, you can use glob to get them. See an example here:

nipype/nipype/interfaces/fsl/model.py

Lines 459 to 490 in e9217c2

class FEAT(FSLCommand):

"""Uses FSL feat to calculate first level stats

"""

_cmd = "feat"

input_spec = FEATInputSpec

output_spec = FEATOutputSpec

def _list_outputs(self):

outputs = self._outputs().get()

is_ica = False

outputs["feat_dir"] = None

with open(self.inputs.fsf_file, "rt") as fp:

text = fp.read()

if "set fmri(inmelodic) 1" in text:

is_ica = True

for line in text.split("\n"):

if line.find("set fmri(outputdir)") > -1:

try:

outputdir_spec = line.split('"')[-2]

if os.path.exists(outputdir_spec):

outputs["feat_dir"] = outputdir_spec

except:

pass

if not outputs["feat_dir"]:

if is_ica:

outputs["feat_dir"] = glob(os.path.join(os.getcwd(), "*ica"))[0]

else:

outputs["feat_dir"] = glob(os.path.join(os.getcwd(), "*feat"))[0]

print("Outputs from FEATmodel:", outputs)

return outputs

Also, can you run make specs and commit the new file?

Thanks! That was very helpful.

effigies · 2020-11-05T15:28:46Z

@aghayoor Just a reminder that I left a review when you have time.

effigies · 2020-11-21T15:19:22Z

Hi @aghayoor, any chance of finishing this one up before a release?

aghayoor · 2020-11-22T09:40:56Z

Hi @effigies , I was out of office for about two weeks. Thanks for your comments. I will try to address all of your comments by the end of this week.

Afni 3dNetCorr is added to nipype. For more details, please see: https://afni.nimh.nih.gov/pub/dist/doc/program_help/3dNetCorr.html

aghayoor · 2021-03-02T17:45:39Z

@effigies Hi, I addressed your comments. Could please review this new PR?

nipype/interfaces/afni/preprocess.py

aghayoor · 2021-03-02T19:57:25Z

@effigies : addressed your comment. Thanks for catching that.

nipype/interfaces/afni/preprocess.py

Co-authored-by: Chris Markiewicz <effigies@gmail.com>

codecov · 2021-03-02T21:07:11Z

Codecov Report

Merging #3263 (c00dd2f) into master (47fe00b) will increase coverage by 0.00%.
The diff coverage is 70.27%.

@@           Coverage Diff           @@
##           master    #3263   +/-   ##
=======================================
  Coverage   64.70%   64.70%           
=======================================
  Files         302      302           
  Lines       39869    39906   +37     
  Branches     5288     5290    +2     
=======================================
+ Hits        25796    25822   +26     
- Misses      12984    12995   +11     
  Partials     1089     1089

Flag	Coverage Δ
unittests	`64.70% <70.27%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
nipype/interfaces/afni/__init__.py	`100.00% <ø> (ø)`
nipype/interfaces/afni/preprocess.py	`81.33% <70.27%> (-0.47%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47fe00b...c00dd2f. Read the comment docs.

This file was automatically generated by running "make specs"

aghayoor · 2021-03-04T02:32:36Z

Hi @effigies , I added correlation maps to the output of this module and I also committed the output file of make specs. Please review and let me know if you have any comments.

aghayoor · 2021-03-12T20:05:55Z

Hi @effigies , just a reminder that I updated the PR. Please review when you have time. Thanks!

effigies · 2021-03-12T20:32:49Z

Thanks for the bump!

aghayoor force-pushed the afni_corrnet branch from 76704f0 to c013164 Compare October 26, 2020 22:46

aghayoor force-pushed the afni_corrnet branch from c013164 to 539d2a6 Compare October 27, 2020 01:24

effigies reviewed Oct 27, 2020

View reviewed changes

effigies changed the title ~~WIP: Afni NetCorr~~ NF: AFNI 3dNetCorr as afni.NetCorr Oct 27, 2020

ENH: added afni NetCorr

a8ee022

Afni 3dNetCorr is added to nipype. For more details, please see: https://afni.nimh.nih.gov/pub/dist/doc/program_help/3dNetCorr.html

aghayoor force-pushed the afni_corrnet branch from 539d2a6 to 0f94b22 Compare March 2, 2021 17:36

effigies reviewed Mar 2, 2021

View reviewed changes

nipype/interfaces/afni/preprocess.py Outdated Show resolved Hide resolved

ENH: modified afni NetCorr development

6da8ee8

aghayoor force-pushed the afni_corrnet branch from bba2d36 to 6da8ee8 Compare March 2, 2021 19:55

effigies reviewed Mar 2, 2021

View reviewed changes

nipype/interfaces/afni/preprocess.py Outdated Show resolved Hide resolved

ENH: AFNI NetCorr sorted commandline

380d798

Co-authored-by: Chris Markiewicz <effigies@gmail.com>

effigies approved these changes Mar 2, 2021

View reviewed changes

aghayoor added 2 commits March 3, 2021 21:24

ENH: added the correlation maps to the output afni NetCorr

bfcfa94

ENH: added test_auto_NetCorr

c00dd2f

This file was automatically generated by running "make specs"

effigies merged commit 10388b0 into nipy:master Mar 12, 2021

effigies added this to the 1.6.1 milestone May 1, 2021

effigies mentioned this pull request Nov 11, 2024

ts_Z_corr not defined #3696

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NF: AFNI 3dNetCorr as afni.NetCorr #3263

NF: AFNI 3dNetCorr as afni.NetCorr #3263

aghayoor commented Oct 26, 2020

aghayoor commented Oct 26, 2020

effigies commented Oct 26, 2020

aghayoor commented Oct 26, 2020

effigies commented Oct 26, 2020

aghayoor commented Oct 26, 2020

effigies commented Oct 26, 2020

aghayoor commented Oct 27, 2020

effigies Oct 27, 2020

aghayoor Mar 2, 2021

aghayoor Mar 2, 2021

effigies Mar 2, 2021

aghayoor Mar 4, 2021

effigies commented Nov 5, 2020

effigies commented Nov 21, 2020

aghayoor commented Nov 22, 2020

aghayoor commented Mar 2, 2021

aghayoor commented Mar 2, 2021

codecov bot commented Mar 2, 2021 •

edited

Loading

aghayoor commented Mar 4, 2021

aghayoor commented Mar 12, 2021

effigies commented Mar 12, 2021

	class FEAT(FSLCommand):
	"""Uses FSL feat to calculate first level stats
	"""

	_cmd = "feat"
	input_spec = FEATInputSpec
	output_spec = FEATOutputSpec

	def _list_outputs(self):
	outputs = self._outputs().get()
	is_ica = False
	outputs["feat_dir"] = None
	with open(self.inputs.fsf_file, "rt") as fp:
	text = fp.read()
	if "set fmri(inmelodic) 1" in text:
	is_ica = True
	for line in text.split("\n"):
	if line.find("set fmri(outputdir)") > -1:
	try:
	outputdir_spec = line.split('"')[-2]
	if os.path.exists(outputdir_spec):
	outputs["feat_dir"] = outputdir_spec

	except:
	pass
	if not outputs["feat_dir"]:
	if is_ica:
	outputs["feat_dir"] = glob(os.path.join(os.getcwd(), "*ica"))[0]
	else:
	outputs["feat_dir"] = glob(os.path.join(os.getcwd(), "*feat"))[0]
	print("Outputs from FEATmodel:", outputs)
	return outputs

NF: AFNI 3dNetCorr as afni.NetCorr #3263

NF: AFNI 3dNetCorr as afni.NetCorr #3263

Conversation

aghayoor commented Oct 26, 2020

aghayoor commented Oct 26, 2020

effigies commented Oct 26, 2020

aghayoor commented Oct 26, 2020

effigies commented Oct 26, 2020

aghayoor commented Oct 26, 2020

effigies commented Oct 26, 2020

aghayoor commented Oct 27, 2020

effigies Oct 27, 2020

Choose a reason for hiding this comment

aghayoor Mar 2, 2021

Choose a reason for hiding this comment

aghayoor Mar 2, 2021

Choose a reason for hiding this comment

effigies Mar 2, 2021

Choose a reason for hiding this comment

aghayoor Mar 4, 2021

Choose a reason for hiding this comment

effigies commented Nov 5, 2020

effigies commented Nov 21, 2020

aghayoor commented Nov 22, 2020

aghayoor commented Mar 2, 2021

aghayoor commented Mar 2, 2021

codecov bot commented Mar 2, 2021 • edited Loading

Codecov Report

aghayoor commented Mar 4, 2021

aghayoor commented Mar 12, 2021

effigies commented Mar 12, 2021

codecov bot commented Mar 2, 2021 •

edited

Loading