🪴 Zero's Garden

Amazon S3

2 min read

Partitioned prefixes

The purpose of the prefix and delimiter parameters is to help you organize and then browse your keys hierarchically. To do this, first pick a delimiter for your bucket, such as slash (/), that doesn’t occur in any of your anticipated key names. You can use another character as a delimiter. There is nothing unique about the slash (/) character, but it is a very common prefix delimiter.

Note that delimiters do not affect prefix partitioning, so you don’t have to add a “folder structure” to get partitioned Amazon S3 prefixes. A prefix is just a string of characters at the beginning of the object key name, and it can end at any position in the key name string.

When a specific partition runs cold, the partitioning process would need to start over when request rates are high again. Thus, to fully benefit from S3’s partitioning capabilities, it is crucial to consistently utilize prefixes that have already been partitioned to achieve higher transactions per second (TPS).

The key point is to gradually increase your traffic over time to let partitioning happen, and consistently utilize the partitioned Amazon S3 prefixes to keep them hot.

Therefore, I think prefixes with lower entropy per character (e.g. numbers) are favorable in S3 because there are fewer variations on each character position and request rates aggregate better to keep a partitioned prefix hot.

Delimiters

Delimiters have limited use cases:

  1. If you issue a list request with a delimiter, you can browse your hierarchy at only one level, skipping over and summarizing the (possibly millions of) keys nested at deeper levels.
  2. Indicate each level within your prefixes for Amazon S3 Storage Lens dashboards to aggregate prefix statistics. Along with the prefix depth settings, it can prevent deep levels of prefixes from showing up on the dashboard.

References

Graph View