Sagemaker sdk does not support un-versioned docker API's (Sagemaker Studio)

[alvilssture]

Describe the bug
I'm encountering an issue while attempting to build a local model deployment using the ModelBuilder class in SageMaker Studio. When I attempt to use ModelBuilder, I get the following error:
DockerException: Error while fetching server API version: Expecting value: line 1 column 1 (char 0) The problem seems to stem from the fact that the Docker version in SageMaker Studio is not recognized, which causes this error when trying to interact with Docker as part of the local deployment process. This impacts the usage of the ModelBuilder class, as it is unable to retrieve necessary Docker API information.
Better explained issue with mlflow:
aws/amazon-sagemaker-feedback#161.

By changing directly in sdk docker.from_env() to docker.from_env(version='1.44') solves this issue, but then there is another issue with local mdoe:

Forbidden ("Forbidden. Reason: 
[ContainerCreate] SageMaker Studio Docker Usage only allows 'sagemaker' as network input")

To reproduce
In sagemaker studio:

model_name = custom
model_version = 1
MLFLOW_TRACKING_ARN = "your tracking uri"
execution_role = get_execution_role() if get_execution_role() is not None else "your-role-arn"

model_builder = ModelBuilder(
    mode=Mode.LOCAL_CONTAINER,
    schema_builder=model_schema,
    role_arn=execution_role,
    model_metadata={
        "MLFLOW_MODEL_PATH": f"models:/{model_name}/{model_version}",
        "MLFLOW_TRACKING_ARN" : '/home/sagemaker-user/projects/clvm/mlruns'
    })

Expected behavior
Are able to deploy a model locally/on cloud.

Screenshots or logs

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/client.py:223   │
│ in _retrieve_server_version                                                                      │
│                                                                                                  │
│   220 │                                                                                          │
│   221 │   def _retrieve_server_version(self):                                                    │
│   222 │   │   try:                                                                               │
│ ❱ 223 │   │   │   return self.version(api_version=False)["ApiVersion"]                           │
│   224 │   │   except KeyError as ke:                                                             │
│   225 │   │   │   raise DockerException(                                                         │
│   226 │   │   │   │   'Invalid response from docker daemon: key "ApiVersion"'                    │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/daemon.py:181   │
│ in version                                                                                       │
│                                                                                                  │
│   178 │   │   │   │   If the server returns an error.                                            │
│   179 │   │   """                                                                                │
│   180 │   │   url = self._url("/version", versioned_api=api_version)                             │
│ ❱ 181 │   │   return self._result(self._get(url), json=True)                                     │
│   182                                                                                            │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/client.py:284   │
│ in _result                                                                                       │
│                                                                                                  │
│   281 │   │   self._raise_for_status(response)                                                   │
│   282 │   │                                                                                      │
│   283 │   │   if json:                                                                           │
│ ❱ 284 │   │   │   return response.json()                                                         │
│   285 │   │   if binary:                                                                         │
│   286 │   │   │   return response.content                                                        │
│   287 │   │   return response.text                                                               │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/requests/models.py:978 in  │
│ json                                                                                             │
│                                                                                                  │
│    975 │   │   except JSONDecodeError as e:                                                      │
│    976 │   │   │   # Catch JSON-related errors and raise as requests.JSONDecodeError             │
│    977 │   │   │   # This aliases json.JSONDecodeError and simplejson.JSONDecodeError            │
│ ❱  978 │   │   │   raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)                            │
│    979 │                                                                                         │
│    980 │   @property                                                                             │
│    981 │   def links(self):                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:1                                                                                    │
│                                                                                                  │
│ ❱ 1 predictor = model.deploy(                                                                    │
│   2 )                                                                                            │
│   3                                                                                              │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/utils/tele │
│ metry_logger.py:116 in wrapper                                                                   │
│                                                                                                  │
│   113 │   │   │   │   failure_reason = str(e)                                                    │
│   114 │   │   │   │   failure_type = e.__class__.__name__                                        │
│   115 │   │   │   except Exception as e:  # pylint: disable=W0703                                │
│ ❱ 116 │   │   │   │   raise e                                                                    │
│   117 │   │   │                                                                                  │
│   118 │   │   │   stop_timer = perf_counter()                                                    │
│   119 │   │   │   elapsed = stop_timer - start_timer                                             │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/utils/tele │
│ metry_logger.py:104 in wrapper                                                                   │
│                                                                                                  │
│   101 │   │   │                                                                                  │
│   102 │   │   │   start_timer = perf_counter()                                                   │
│   103 │   │   │   try:                                                                           │
│ ❱ 104 │   │   │   │   response = func(self, *args, **kwargs)                                     │
│   105 │   │   │   except (                                                                       │
│   106 │   │   │   │   ModelBuilderException,                                                     │
│   107 │   │   │   │   exceptions.CapacityError,                                                  │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/builder/mo │
│ del_builder.py:600 in _model_builder_deploy_wrapper                                              │
│                                                                                                  │
│    597 │   │   │   │   deserializer=deserializer,                                                │
│    598 │   │   │   )                                                                             │
│    599 │   │   │                                                                                 │
│ ❱  600 │   │   │   self.modes[str(Mode.LOCAL_CONTAINER)].create_server(                          │
│    601 │   │   │   │   self.image_uri, container_timeout_in_second, self.secret_key, predictor   │
│    602 │   │   │   )                                                                             │
│    603 │   │   │   return predictor                                                              │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/mode/local │
│ _container_mode.py:100 in create_server                                                          │
│                                                                                                  │
│    97 │   ):                                                                                     │
│    98 │   │   """Placeholder docstring"""                                                        │
│    99 │   │                                                                                      │
│ ❱ 100 │   │   self._pull_image(image=image)                                                      │
│   101 │   │                                                                                      │
│   102 │   │   self.destroy_server()                                                              │
│   103                                                                                            │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/sagemaker/serve/mode/local │
│ _container_mode.py:232 in _pull_image                                                            │
│                                                                                                  │
│   229 │   │   except subprocess.CalledProcessError as e:                                         │
│   230 │   │   │   logger.warning("Unable to login to ecr: %s", e)                                │
│   231 │   │                                                                                      │
│ ❱ 232 │   │   self.client = docker.from_env()                                                    │
│   233 │   │   try:                                                                               │
│   234 │   │   │   logger.info("Pulling image %s from repository...", image)                      │
│   235 │   │   │   self.client.images.pull(image)                                                 │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/client.py:94 in     │
│ from_env                                                                                         │
│                                                                                                  │
│    91 │   │   max_pool_size = kwargs.pop('max_pool_size', DEFAULT_MAX_POOL_SIZE)                 │
│    92 │   │   version = kwargs.pop('version', None)                                              │
│    93 │   │   use_ssh_client = kwargs.pop('use_ssh_client', False)                               │
│ ❱  94 │   │   return cls(                                                                        │
│    95 │   │   │   timeout=timeout,                                                               │
│    96 │   │   │   max_pool_size=max_pool_size,                                                   │
│    97 │   │   │   version=version,                                                               │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/client.py:45 in     │
│ __init__                                                                                         │
│                                                                                                  │
│    42 │   │   │   to save in the pool.                                                           │
│    43 │   """                                                                                    │
│    44 │   def __init__(self, *args, **kwargs):                                                   │
│ ❱  45 │   │   self.api = APIClient(*args, **kwargs)                                              │
│    46 │                                                                                          │
│    47 │   @classmethod                                                                           │
│    48 │   def from_env(cls, **kwargs):                                                           │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/client.py:207   │
│ in __init__                                                                                      │
│                                                                                                  │
│   204 │   │   │   │   │   │   │   │   version,                                                   │
│   205 │   │   │   │   │   │   │   │   str                                                        │
│   206 │   │   │   │   │   │   │   │   ) and version.lower() == 'auto'):                          │
│ ❱ 207 │   │   │   self._version = self._retrieve_server_version()                                │
│   208 │   │   else:                                                                              │
│   209 │   │   │   self._version = version                                                        │
│   210 │   │   if not isinstance(self._version, str):                                             │
│                                                                                                  │
│ /home/sagemaker-user/projects/clvm/.venv/lib/python3.11/site-packages/docker/api/client.py:230   │
│ in _retrieve_server_version                                                                      │
│                                                                                                  │
│   227 │   │   │   │   ' is missing.'                                                             │
│   228 │   │   │   ) from ke                                                                      │
│   229 │   │   except Exception as e:                                                             │
│ ❱ 230 │   │   │   raise DockerException(                                                         │
│   231 │   │   │   │   f'Error while fetching server API version: {e}'                            │
│   232 │   │   │   ) from e                                                                       │
│   233                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
DockerException: Error while fetching server API version: Expecting value: line 1 column 1 (char 0)

```**System information**
A description of your system. Please provide:
- sagemaker==2.242.2
- mlflow==2.21.0
- python 3.11.11
- cpu
- No custom docker image
- SageMaker Distribution 3.0.0 image in sagemaker studio