calling aggregate_channels returned AttributeError

[chiyu1203]

Dear Spikeinterface community
I was hoping to correct drift/motion and run spike sorting shank by shank for our 4-shank probe. I followed this tutorial and was able to run those analysis. However, when I called the function to aggregate those object together, I bumped into this error.

Traceback (most recent call last):
File "", line 1, in
File "c:\Users\einat\anaconda3\envs\spike_interface\lib\site-packages\spikeinterface\core\channelsaggregationrecording.py", line 217, in aggregate_channels
return ChannelsAggregationRecording(recording_list, renamed_channel_ids)
File "c:\Users\einat\anaconda3\envs\spike_interface\lib\site-packages\spikeinterface\core\channelsaggregationrecording.py", line 20, in init
self._perform_consistency_checks()
File "c:\Users\einat\anaconda3\envs\spike_interface\lib\site-packages\spikeinterface\core\channelsaggregationrecording.py", line 104, in _perform_consistency_checks
sampling_frequencies = [rec.get_sampling_frequency() for rec in self.recordings]
File "c:\Users\einat\anaconda3\envs\spike_interface\lib\site-packages\spikeinterface\core\channelsaggregationrecording.py", line 104, in
sampling_frequencies = [rec.get_sampling_frequency() for rec in self.recordings]
AttributeError: 'numpy.int64' object has no attribute 'get_sampling_frequency'

I have tried to change the class type into float32 but still received the same error. Does anyone know how to fix this?

Below is the code I used:

            raw_rec = se.read_openephys(oe_folder, load_sync_timestamps=True)
            fs = raw_rec.get_sampling_frequency()
            stacked_probes = pi.read_probeinterface("H10_stacked_probes.json")
            probe = stacked_probes.probes[0]
            raw_rec = raw_rec.set_probe(probe,group_mode='by_shank')
            probe_rec = raw_rec.get_probe()
            probe_rec.to_dataframe(complete=True).loc[
            raw_rec = spre.astype(raw_rec, "float32")
            raw_rec_dict = raw_rec.split_by(property='group', outputs='dict')
            recording_corrected = si.aggregate_channels(raw_rec_dict)

And here is what I have in the raw_rec_dict

{0: ChannelSliceRecording: 32 channels - 30.0kHz - 1 segments - 25,237,540 samples
...(14.02 minutes) - float32 dtype - 3.01 GiB, 1: ChannelSliceRecording: 32 channels - 30.0kHz - 1 segments - 25,237,540 samples
...(14.02 minutes) - float32 dtype - 3.01 GiB, 2: ChannelSliceRecording: 32 channels - 30.0kHz - 1 segments - 25,237,540 samples
...(14.02 minutes) - float32 dtype - 3.01 GiB, 3: ChannelSliceRecording: 32 channels - 30.0kHz - 1 segments - 25,237,540 samples
...(14.02 minutes) - float32 dtype - 3.01 GiB}

Operation System: Windows 11, spikeinterface 0.102.1 and 0.101.2 under python 3.10
In case it is my probe being too complicated. I have attached the probe info here

H10_stacked_probes.json

[chrishalcrow]

Hi @chiyu1203 I think I know what's going on. We're currently updating how processing stuff by group is done. So we've just updated the aggregate_channels function so that it accepts dictionaries. This is a brand new feature (it got added 2 weeks ago!), so is only available on the dev version of spikeinterface (what you get when you get spikeinterface using git clone).

The docs you're reading are for the dev verison of spikeinterface but your spikeinterface version is the "stable" version so you should read those docs (https://spikeinterface.readthedocs.io/en/stable/how_to/process_by_channel_group.html, NOT https://spikeinterface.readthedocs.io/en/latest/how_to/process_by_channel_group.html).

So you can either download the latest spikeinterface version (which will allow you to pass a dictionary of recordings to aggregate channels) or read the docs from the the stable spikeinterface version and pass a list of recordings to aggregate channels.

Hope that makes sense - let me know if it works. Sorry that you've got caught up in a dev-versus-stable conflict - they are very annoying, and this problem will vanish once do another release (hopefully very soon!).

[chiyu1203]

@chrishalcrow Thank you so much for the reply! I am really happy about this new feature. I have installed the dev verison of spikeinterface and can confirm that this fixed my problem.
Just quick follow-up questions before we close this issue.
Besides the functions listed up in the dev version of the docs ( phase_shift, common_reference and bandpass_filter ), I had no problem calling whiten with the dictionary of the recording.
However, it does not seem to be the case for spre.correct_motion, as calling that High-level API with the dictionary resulted in this error.

Exception has occurred: AttributeError
'dict' object has no attribute 'sampling_frequency'
File "C:\Users\neuroLaptop\Documents\GitHub\spikeinterface\src\spikeinterface\preprocessing\motion.py", line 365, in correct_motion
sampling_frequency=recording.sampling_frequency,
File "C:\Users\neuroLaptop\Documents\GitHub\ephys\estimate_drift_motion.py", line 55, in AP_band_drift_estimation
recording_corrected, , motion_info = spre.correct_motion(
File "C:\Users\neuroLaptop\Documents\GitHub\ephys\raw2si.py", line 272, in raw2si
recording_corrected,=AP_band_drift_estimation(4,rec_for_sorting,oe_folder,analysis_methods,win_um,job_kwargs)
File "C:\Users\neuroLaptop\Documents\GitHub\ephys\raw2si.py", line 393, in
raw2si(thisDir, json_file)
AttributeError: 'dict' object has no attribute 'sampling_frequency'

I also bumped into a similar error when using save(), which returned

Exception has occurred: AttributeError
'dict' object has no attribute 'save'
File "C:\Users\neuroLaptop\Documents\GitHub\ephys\raw2si.py", line 222, in raw2si
recording_saved = rec_of_interest.save(
File "C:\Users\neuroLaptop\Documents\GitHub\ephys\raw2si.py", line 394, in
raw2si(thisDir, json_file)
AttributeError: 'dict' object has no attribute 'save'

In this case, would you recommend that I use Manual splitting (loop over all preprocessed recordings) to run correct_motion and then save the output as a new dictionary, which can be used in whiten and run_sorter_by_property? And then when I want to save the result, I call aggregate_channels for the resulting dictionary first?

[chrishalcrow]

Haha @chiyu1203 you're finding all the edge cases I'm currently trying to fix!!! There are two preprocessing steps which aren't implemented like the others: motion correction and bad channel removal. We're in the middle of turning these into classes (for bad channel removal: #3685) and then you'll be able to pass dictionaries of recordings to them! We also need to add a save function for dicts of recordings.

For now, you need to do a manual splitting to correct motion, then aggregate at the end. I'll ping you once the new features are added!

I'm tagging @alejoe91 @samuelgarcia so they see that there's high demand (well, some demand) for these features!

[chiyu1203]

Thanks, then I will let you close this issue! Yes, on one hand, I was hoping to do spike sorting for multi-shank probes just based on the position of channels on the probe map, without splitting channels shank by shank. On the other hand, thinking about that the physical distance between channels, across shanks is not as accurate as within shanks, it is a good idea to analyze the data shank by shank. These new features will certainly make things much easier and encourage researchers to take this approach!

[chiyu1203]

@chrishalcrow Your might have been working on this already, but just in case.

I use Manual splitting (loop over all preprocessed recordings) to run correct motion, apply whitening and spike-sorting
Then I save the output as a new dictionary and call the function aggregate_channels to merge them.
However, this returned the following error:

Exception has occurred: AttributeError
'KiloSortSortingExtractor' object has no attribute 'get_num_channels'
File "C:\Users\einat\Documents\GitHub\spikeinterface\src\spikeinterface\core\channelsaggregationrecording.py", line 35, in init
recording.set_property("group", [group_id] * recording.get_num_channels())
File "C:\Users\einat\Documents\GitHub\spikeinterface\src\spikeinterface\core\channelsaggregationrecording.py", line 269, in aggregate_channels
return ChannelsAggregationRecording(recording_list_or_dict, renamed_channel_ids)
File "C:\Users\einat\Documents\GitHub\ephys\raw2si.py", line 319, in raw2si
sorting_spikes=si.aggregate_channels(recording_corrected_dict)
File "C:\Users\einat\Documents\GitHub\ephys\raw2si.py", line 454, in
raw2si(thisDir, json_file)
AttributeError: 'KiloSortSortingExtractor' object has no attribute 'get_num_channels'

Is this related to the sorter or the function aggregate_channels?

(Actually, I tried the run_sorter_by_property first when applying spike-sorting shank by shank as we discussed earlier this thread but that returned the following TypeError)

TypeError: ChannelsAggregationRecording.init() got an unexpected keyword argument 'recording_list'

[chrishalcrow]

Hi @chiyu1203 , that error looks like you're trying to apply aggregate_channels (which aggregates recordings) to a sorting object. You need to apply aggregate_units if you want to combine sorting objects.

[chiyu1203]

Hi @chrishalcrow, thank you for the quick response! I called that function but it returned this error.

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\einat\Documents\GitHub\spikeinterface\src\spikeinterface\core\unitsaggregationsorting.py", line 30, in init
num_all_units = sum([sort.get_num_units() for sort in sorting_list])
File "C:\Users\einat\Documents\GitHub\spikeinterface\src\spikeinterface\core\unitsaggregationsorting.py", line 30, in
num_all_units = sum([sort.get_num_units() for sort in sorting_list])
AttributeError: 'int' object has no attribute 'get_num_units'

I guessed that is because the function does not accept dict?! Then I played around the code inside run_sorter_by_property and manged to aggreate the units and save them (not sure if I missed something).

But below is my code. Feel free to let me know if you have any thought (Spikeinterface version 0.102.2 built from source)

recording_f = raw_rec.set_probe(probe,group_mode='by_shank')
recording_saved = recording_f.split_by(property='group', outputs='dict')

recording_corrected_dict = {}
unit_groups = []
for group, sub_recording in recording_saved.items():
    recording_corrected,_=AP_band_drift_estimation(group,sub_recording,oe_folder,analysis_methods,win_um,job_kwargs)
    rec_for_sorting = spre.whiten(
        recording=recording_corrected,
        mode="local",
        radius_um=25 * 2,
        dtype=float,
    )
    sorting_spikes = ss.run_sorter(
        sorter_name=this_sorter,
        recording=rec_for_sorting,
        remove_existing_folder=True,
        output_folder=oe_folder / f"result_folder_name{group}",
        verbose=True,
        **sorter_params,
    )
    recording_corrected_dict[group]=sorting_spikes
    num_units =sorting_spikes.get_unit_ids().size
    unit_groups.extend([group] * num_units)
unit_groups=np.array(unit_groups)
aggregate_sorting = spre.aggregate_units(list(recording_corrected_dict.values()))
aggregate_sorting.set_property(key='group', values=unit_groups)
aggregate_sorting.register_recording(recording_f)
aggregate_sorting.save(folder=oe_folder / sorting_folder_name, overwrite=True)

[chiyu1203]

Sorry for keeping posting my problems again. But after I ran the abovementioned code to aggregate units, I tried to create an sorting_analyzer with units and channels being aggregated by their respective functions (spre.aggregate_units, si.aggregate_channels). With this function

sorting_analyzer = si.create_sorting_analyzer(
            sorting=sorting_spikes,
            recording=recording_saved,
            sparse=True,  # default
            format="memory",  # default
        )

I bumped into this error

Exception has occurred: TypeError (note: full exception trace is shown but execution is paused at: )
ChannelsAggregationRecording.init() got an unexpected keyword argument 'recording_list'
File "C:\Users\neuroPC\Documents\GitHub\spikeinterface\src\spikeinterface\core\base.py", line 1127, in _load_extractor_from_dict
extractor = extractor_class(**new_kwargs)
File "C:\Users\neuroPC\Documents\GitHub\spikeinterface\src\spikeinterface\core\base.py", line 571, in from_dict
extractor = _load_extractor_from_dict(dictionary)
File "", line 1, in (Current frame)
TypeError: ChannelsAggregationRecording.init() got an unexpected keyword argument 'recording_list'

Is there any way to circumvent this issue? (spikeinterface-0.102.3, built from source)

[chrishalcrow]

Hi @chiyu1203 , we merged a bug fix last night (#3829) so hopefully a new install from source should be bug free (or just git pull in your spikeinterface directory, if you installed using the -e flag)! Let me know if that works

[chiyu1203]

Hi @chrishalcrow, thanks for quick response and the amazing merge! It worked! Also it seems that the new merge also fixed some kind of issues with .save() (it still does not accept .dict so I called si.aggregate_channels(rec_of_interest) first before saving the recording)!!

[chrishalcrow]

That's great! Hoping to add dicts to save/sort very soon! Will let you know :)

[chiyu1203]

Thanks! Take your time! Actually I just realised that using run_sorter_by_property will return an Object that .save() can process already so I will just start to use run_sorter_by_property from now

[chiyu1203]

just another quick note: it seems that .frame_slice() does not take dict either...

[chrishalcrow]

just another quick note: it seems that .frame_slice() does not take dict either...

Thanks very much - I've made a note :)

This is one a bit trickier because it's a method (i.e. do you recording.frame_slice(...) rather than common_reference(recording)). And I think the order you do this (frame slice then split by group VS split by group then frame slice) will never matter. Do you have an example where you do a frame slice based on output from the preprocessing steps?

For now: I recommend doing a frame_slice at the very start of your preprocessing. So something like:

rec = si.load(...)
sliced_rec = rec.frame_slice(...)
dict_of_recs = sliced_rec.split_by('group')
# now do preprocessing steps

[chiyu1203]

just another quick note: it seems that .frame_slice() does not take dict either...

Thanks very much - I've made a note :)

This is one a bit trickier because it's a method (i.e. do you recording.frame_slice(...) rather than common_reference(recording)). And I think the order you do this (frame slice then split by group VS split by group then frame slice) will never matter. Do you have an example where you do a frame slice based on output from the preprocessing steps?

For now: I recommend doing a frame_slice at the very start of your preprocessing. So something like:

rec = si.load(...)
sliced_rec = rec.frame_slice(...)
dict_of_recs = sliced_rec.split_by('group')

That's exactly what happened! I built up the analysis pipeline based on the spiketutorial jupyter notebook last year, where frames are only sliced after filter+detect_bad_channels+common_reference (before saving preprocessed data).
But I dont use this function .frame_slice often, so I am happy just to split them manually as below or like what you suggested to do : slicing frames right after loading the raw data

            recording_cmr = spre.common_reference(
                recordings_dict, reference="global", operator="average"
            )
        if "recording_cmr" in locals():
            rec_of_interest = recording_cmr
        else:
            rec_of_interest = recording_saved
            rec_of_interest.annotate(
                is_filtered=True
            )  # needed to add this somehow because when loading a preprocessed data saved in the past, that data would not be labeled as filtered data
        # Slice the recording if needed
        if tmin_tmax[1]>0 and tmin_tmax[1]>tmin_tmax[0]:
            start_sec = tmin_tmax[0]
            end_sec = tmin_tmax[1]
            if type(rec_of_interest)==dict:
                tmp = {}
                for group, sub_recording in rec_of_interest.items():
                    tmp[group] = sub_recording.frame_slice(start_frame=start_sec * fs, end_frame=end_sec * fs)
                rec_of_interest=tmp
            else:
                rec_of_interest = rec_of_interest.frame_slice(start_frame=start_sec * fs, end_frame=end_sec * fs)
        elif tmin_tmax[1]<0:
            print("tmax <0 means to analyse the entire recording")
        else:
            ValueError("tmax needs to be bigger than tmin to select certain section of the recording")