What's changing?
To support large amounts of log data and scale without hitting S3 rate limits, we are updating how logs are stored in S3. Instead of creating a new file for every chunk we are batching chunks and writing to a file. The location and format of files will change from
/replays/YYYY/MM/DD/HH/MM/<queryUUID>/<chunkID>.json
to
/replays/YYYY/MM/DD/HH/MM/<randomUUID>.json.
This new storage format of logs includes multiple chunks in a single file, batching groups of writes together. It also can include chunks from multiple queries in the same file. A file will be generated every minute, or be generated if the file size exceeds 100MB.
What Does This Mean For Me?
This is informational only and no action is needed if you do not have any automation built that relies on file format and location. If you do have some kind of automation that parses the log files, you will need to adjust the scripts to match the new format and location of the data. You can use our SDKs which have public examples that demonstrate how to use them to reconstruct replay data.
When Are These Changes Coming? This change will occur on Thursday February 29th.