[SOLVED] Google Cloud Storage bucket - signed URL with directory access

Issue

I'm using terraform to create a GCP bucket and signed url, then I'am trying to upload files to this bucket. The process is kind of working but the files uploaded (via curl) are not put inside a desired directory and reather as a directory.

Here is my code:

data "google_storage_object_signed_url" "get_url" {
  bucket       = "my_bucket"
  path         = "some_directory/"
  http_method  = "PUT"
  duration     = "24h"
  credentials  = file("keys.json")
}

and curl:

curl -X PUT --upload-file some_file.txt $SIGNED_URL

If I use path = "some_directory" ("/" removed) the file uploaded with curl is put as :some_directory" yet its a file, when I use it with "/" the file seem to upload correctly yet it is not visible in that directory.

What am I missing here? Do I have to create s signed_url for every file or can I somehow access the whole directory with it?

Solution

Answering as a Community Wiki, so that it may help others too.

As @John Hanley suggested in the comment,

Google Cloud Storage does not support directories. The path separator / is just an object name character. Software emulates directories using prefixes and delimiters. For example the object name my_files/object_name, the prefix is my_files/. A prefix means to search for all object names that start with those characters. If you append a / to a name, it will be treated as a prefix and basically become invisible with most software. If you use the Cloud Storage JSON API, you will see the object.

Accessing the whole directory with a signed URL is not possible because one cannot create a signed URL for a directory. It's worth noting that signed URLs are usually used for single objects (files) rather than directories. So, You must create a signed URL for each object.

Answered By - Sandeep Vokkareni

Answer Checked By - Marie Seifert (WPSolving Admin)