Contribution
Unit 42 published the bucket-squatting attack, and Google shipped the permanent ownership-check fix for auto-generated staging buckets in Vertex AI SDK for Python v1.148.0. This post turns those public findings into an operator-side detection strategy: identify use of the affected local-artefact staging path, resolve who owned the exact default bucket, prove whether the model reached an endpoint, then look for data access by the serving identity. The value is not a new exploit claim. It is the decision path defenders need when a model upload used old tooling and a bucket whose owner they cannot prove.
The pattern
Unit 42's Pickle in the Middle research exposes a common release-control blind spot: teams that verify container provenance can still treat model artefacts as passive data. The vulnerability sat in the Google Cloud Vertex AI SDK for Python. In versions 1.139.0 and 1.140.0, the latest releases Unit 42 tested, a model upload without an explicit staging bucket built a deterministic bucket name in the form {project}-vertex-staging-{location}. The SDK checked whether that bucket existed, but not whether it belonged to the caller's project.
That missing ownership check created the opening. An attacker who knew the victim project ID and target region could create the expected bucket first in an attacker-controlled Google Cloud project, then grant allAuthenticatedUsers enough access for the victim SDK to create objects and the Vertex AI service agent to read them. The victim ran normal SDK code. The SDK staged the model artefacts into the attacker's bucket. Unit 42 showed that a Cloud Function trigger on google.storage.object.finalize could replace model.joblib before the service agent read it. Their proof of concept measured a roughly 2.5 second hand-off window: upload at T+0 ms, function detection at T+804 ms, replacement at T+1,433 ms and service-agent read at T+2,460 ms. The poisoned model then executed code when the serving container loaded the pickle or joblib object.
Google's v1.148.0 release records the permanent fix as a bug fix for bucket ownership verification in Model.upload(). Unit 42 also notes an earlier v1.144.0 mitigation that randomised the generated bucket name. The v1.148.0 commit adds _verify_bucket_ownership, reloads bucket metadata, compares the bucket's project number with the expected project number, and raises a ValueError if an auto-generated bucket exists in the wrong project. The check runs only when the SDK generates the default staging bucket. An explicitly passed staging_bucket, or one configured through aiplatform.init(), bypasses this SDK ownership check and therefore still needs operator-side ownership validation.
The model artefact is the bridge between developer tooling and runtime identity. Google Cloud's model import documentation states that model artefacts live in Cloud Storage, the bucket region must match the endpoint location, and the imported model can then be deployed to an endpoint for inference. The docs also call out the model.joblib shape for scikit-learn and XGBoost prebuilt containers. Unit 42's exploit used that same model-loading path. The attacker did not need console access to the victim project. The attacker needed the upload to land in the wrong bucket, then needed the runtime to load attacker-controlled bytes.
The useful pattern is not only 'upgrade the SDK'. It is 'model upload is a release event with a storage owner, a service agent and a runtime token'. If any part of that chain points outside the expected project, treat the deployment as suspect, especially when the model uses a prebuilt container path that deserialises pickle or joblib material at load time.
Why it matters to cloud defenders
Vertex AI is a cloud control-plane surface, but many organisations still monitor model deployment like data-science plumbing. That is a gap. A model upload can create a new executable path in a managed serving environment. A model deployment can cause a service agent or serving identity to read objects, start containers, query metadata, and reach other Google Cloud resources. If the artefact came from a bucket the project does not own, the deployment is not only a bad release. It is a possible credential and data exposure event.
The Unit 42 proof of concept makes the cloud impact concrete. Their payload queried the Google metadata service from the serving container, collected service-account details and environment data, and sent material to an attacker-controlled endpoint. They also reported that the token in their test had cloud-platform scope and could read information beyond the single deployment, including model artefacts from other deployments, BigQuery dataset names and access-control lists, and tenant-project logging details. The exact permissions depend on the managed service context and Google-side boundaries, but defenders should not wait for a perfect blast-radius model before triaging.
The hard part is that no single log source tells the full story. Package inventory can say whether a developer or pipeline used an affected google-cloud-aiplatform version. Cloud Asset Inventory can resolve buckets inside projects, folders or organisations the defender can inspect. Cloud Storage logs can show object creation or access, but Data Access logs are disabled by default for many Google Cloud services, and Cloud Audit Logs do not track access to public objects. Vertex AI Admin Activity logs record models.upload and endpoints.deployModel. Target-service audit logs can record later API calls attributed to the serving identity. Network telemetry can show an unexpected callback where serving egress is visible. None of those is enough alone.
Cloud defenders also need to handle ownership boundaries cleanly. The suspicious bucket may not be inside the victim organisation, so the victim will not have its logs or Cloud Asset Inventory record. A miss in organisation inventory is therefore unresolved ownership, not proof that an attacker owns the bucket. Use bucket metadata to compare project numbers when permissions allow, and use Cloud Asset Inventory or IAM-policy inspection only for resources inside an accessible scope. The response should rely on evidence the defender controls: package version, local-versus-gs:// artefact input, the vertex_ai_auto_staging/ path, project ID, region, model upload time, endpoint deploy time, service-account activity and outbound traffic.
ATT&CK mapping
The model replacement maps most directly to T1565.001, Stored Data Manipulation. The attacker changed data at rest in the staging bucket so a later deployment produced attacker-chosen execution. ATT&CK currently lists Linux, Windows and macOS for this sub-technique rather than cloud or container platforms, so this is a behavioural mapping to stored-file manipulation, not a claim of cloud-platform coverage in ATT&CK.
The credential collection stage maps to T1552.005, Cloud Instance Metadata API. Unit 42's payload queried the metadata service from the managed serving container for service-account material and runtime context. That is the same defensive question as any metadata-token event: why did this workload reach the metadata endpoint, what identity did it receive, and what did that identity do next? In this case the workload is not a web server or a VM shell. It is a model-serving container that loaded a staged artefact.
The payoff maps to T1528, Steal Application Access Token. A service-account access token gives the attacker a second plane after the model code runs. The token may expire quickly, but short-lived does not mean harmless. If it can read model artefacts, list datasets, inspect access-control lists or call other APIs during its lifetime, the incident has moved from artefact poisoning to cloud identity abuse.
Several adjacent ideas stay in prose. The storage-bucket name collision resembles resource hijacking and namespace squatting. The legitimate SDK creates a supply-chain-like trust boundary, but T1195.001 requires manipulation of dependencies or development tools before receipt by a consumer, which did not happen here. The listed techniques stay with observed behaviour: stored model manipulation, metadata-token collection and token theft.
Detection guidance
Start with asset and package inventory. Find developer workstations, CI images, notebooks, training containers and release runners that used google-cloud-aiplatform 1.139.0 or 1.140.0, the versions Unit 42 tested as vulnerable. Source history shows the predictable name through v1.143.0, a UUID-salted name from v1.144.0 through v1.147.0, and ownership verification from v1.148.0. Upgrade every earlier runtime, but do not assign them equal historical exposure. The demonstrated path also requires Model.upload() to receive a local artefact path: an artifact_uri already beginning with gs:// does not enter stage_local_data_in_gcs(). For the predictable-name cohort, look for local input, no explicit staging_bucket, and absence of the exact {project}-vertex-staging-{location} bucket in the victim project. Evidence of a vertex_ai_auto_staging/ object path is a stronger retrospective indicator than package version alone.
Next, build a bucket-ownership check around model upload and import events. For each Vertex AI model upload or imported model artefact, extract the gs:// URI, bucket name, project, region, caller and time where those fields are present. Compare the bucket's project number with the expected project through storage metadata, then check an allowlist or Cloud Asset Inventory for approved shared artefact projects. Alert when the predictable default pattern resolves outside that set or cannot be resolved. Inspect broad principals such as allAuthenticatedUsers only on buckets whose IAM policy the defender is authorised to read. An external bucket can expose metadata without exposing its allow policy, so ownership mismatch is the reliable cross-project signal.
Then join to deployment. A suspicious bucket becomes urgent when its artefact is deployed to an endpoint. Track models.upload, endpoints.create and endpoints.deployModel, and keep the model resource name, endpoint ID, service account, caller identity and artefact URI together where the audit record provides them. Vertex copies an artifactUri directory into a managed bucket when it creates a Model, and that managed copy cannot be updated. This makes Model creation the integrity boundary: focus object-generation and replacement analysis immediately before models.upload, then follow deployment of that specific Model resource. If it used a prebuilt container that loads pickle or joblib material, raise severity because the payload can execute at load time.
The runtime detector is a serving-identity activity join, and it is post-compromise evidence. Metadata access alone can be normal for a workload using attached credentials, so pair it with an unexpected callback or unusual target-service API use after first deployment. Unit 42 observed the principal custom-online-prediction@<tenant-project>.iam.gserviceaccount.com; use that as a proof-of-concept hunt seed, not a universal identity name. Review Cloud Storage reads outside the deployed model's bucket, Secret Manager access, BigQuery dataset listing, logging reads and other unusual calls attributed to the serving principal. The demonstrated payload obtained its token from metadata, so it did not generate an IAM Credentials GenerateAccessToken event. IAM Credentials logs become relevant only if the attacker later mints another token or impersonates another service account.
Use the chain as a decision tree. A predictable-name SDK, local artefact input and automatic staging establish exposure. Add project-number mismatch and the upload is suspect; unresolved ownership is a high-risk containment trigger, but not proof of poisoning. Add deployment of the resulting Model and the suspect artefact reached runtime. Add object replacement, an unexpected callback or unusual data reads by the serving identity and the case becomes a probable active compromise. Tune false positives with an explicit allowlist of approved shared artefact projects, signed release windows, service-account allowlists and model owners. Do not suppress a confirmed ownership mismatch merely because deployment succeeded.
What to do now
Upgrade every Vertex AI SDK for Python runtime that can upload models to v1.148.0 or later. That includes developer notebooks, CI images, training containers, scheduled training jobs and shared data-science base images. Do not stop at production services, because the vulnerable call may run from a release notebook that never appears in a normal server inventory.
Create explicit staging buckets for model upload, and make bucket ownership part of the release checklist. The bucket should live in the expected project or an approved shared artefact project, match the Vertex AI region, use narrow IAM, and avoid broad principals. Remember that v1.148.0 does not verify an explicitly configured bucket for you. Resolve its project number during provisioning and again before release. If a project has never used Vertex AI in a region, create and validate the staging bucket before the first model upload rather than letting tooling choose a default path.
Add a preventive boundary where the upload environment supports it. VPC Service Controls can stop clients inside a protected perimeter from accessing unauthorised, potentially public Cloud Storage resources outside that perimeter. Test the policy in dry-run mode before enforcement because approved cross-project artefact stores, notebooks and CI runners may need explicit egress rules.
Enable the logs needed for the join. Admin Activity logs, including Vertex AI upload and deploy operations, are retained for 400 days in _Required. Data Access logs are disabled by default for most services and _Default retains them for 30 days unless the bucket retention is changed. Set retention long enough to cover the organisation's model release and incident-discovery cycle, route Cloud Storage and Vertex AI audit logs into the same query surface, and keep network or egress telemetry where available to spot callbacks from serving workloads.
During response, treat unresolved ownership as a containment trigger and a confirmed non-owned default bucket as a suspected poisoning event after checking approved shared artefact projects. Preserve the source and managed artefact URIs, Model resource name, endpoint deployment timeline, caller identity, SDK version, service-account activity and egress evidence. Halt deployment if the Model has not reached an endpoint. If it has reached runtime, create a new Model from a trusted bucket and redeploy it, revoke or rotate any keys and secrets the serving workload could read, and let the stolen short-lived OAuth token expire while blocking related activity where possible. Review Cloud Storage, BigQuery, Secret Manager and logging access by the observed serving identity.
Model artefacts are executable release material. They deserve the same ownership checks, logging joins and token-response practice as container images and CI runners. Otherwise the weakest link in the machine-learning pipeline may be a bucket name that looked harmless until somebody else created it first.