Improve downloads of sharded variants

[DN6]

What does this PR do?

Our current check for variants in variant_compatible_sibling downloads extra non variant files. See: #9253 (comment)

This is because we check if a non-variant filename has an equivalent variant filename. e.g. transformers/model-0003-0005.safetensors has transformers/model-fp16-0003-0005.safetensors in its variants. If it doesn't, we add the file to the list of allowed patterns to download.

This type of check doesn't work for sharded checkpoints since variants of smaller dtypes such as fp16 will always have a fewer number of shards. Therefore the non-variant shard file have no equivalent variant shard file.

This PR:
Changes the check to see if a variant exists for a pipeline component (regardless of whether it's sharded or not). If a variant exists for the component, we don't attempt to add any more files related to that component.

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[DN6]

Oh we should also account for model repos without component folders. Updating.

[yiyixuxu]

Suggested change

	if not len(filename.split("/")) == 2:
	continue
	component, component_filename = filename.split("/")
	components_with_variant.add(component)
	components_with_variant.add(filename.split("/")[0])

I think we should match the logic to find "component" between variant_filenames and non_variant_filenames

[yiyixuxu]

ohh I realize you actually just updated this to account for files that are not put inside any subfolders
maybe make this a in-line function find_component or something and do the same the for f in non_variant_filenames too

[sayakpaul]

Will this work in case we're loading from a non-subfolder location?

[DN6]

@yiyixuxu Made it a bit more robust by using the existing regexes and sharded variant index files.

Now we

1. First check if a variant version of a non-variant file exists in variant files names. If this is the case skip adding the file
2. If a variant version of a non-variant file does not exist. First check if its in a component folder and check if a sharded variant index file exists in that folder. If it isn't a component folder check the main dir. If a sharded variant index file is present then skip
3. Only add a file if none of the above is True.

[yiyixuxu]

awesome!! thank you

[sayakpaul]

@DN6 I think it'd be nice to also add a test for this improvement. Great work!