Repository navigation
[Storage] Encode text as UTF-8 when uploading an object via stream - #2235
Open
shaynhornik wants to merge 1 commit into
Open
shaynhornik wants to merge 1 commit into
shaynhornik wants to merge 1 commit into
Conversation
A stream which yields text, such as a file opened in text mode, was passed to the HTTP client as is. The client encoded it as latin-1, which fails with UnicodeEncodeError for other characters, or sent it with a content length counted in characters, while the object hash was already calculated from the UTF-8 encoded data. Text chunks are now encoded as UTF-8 in the shared upload code before they are sent, and binary streams are passed through unchanged. Generated-by: Claude Code (Claude Fable 5.2) Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[Storage] Encode text as UTF-8 when uploading an object via stream
Description
Fixes #1802.
upload_object_via_streamwith a stream that yields text (aStringIO, or a file opened in text mode) passes the stream to the HTTP client as is, while the object hash is calculated from the UTF-8 encoded data. What the client does with text depends on the urllib3 version (checked against a local HTTP server):StringIOUnicodeEncodeError(the traceback in the issue)UnicodeEncodeErrorThis affects drivers that upload a stream in one request through
StorageDriver._upload_object, such as Google Storage and CloudFiles.Change:
_upload_objectnow passes the stream through a new_get_stream_request_data():io.TextIOBase) is read inCHUNK_SIZEblocks and encoded as UTF-8 lazily, so the stream is not buffered in memory.strchunks from a plain iterator are encoded too. urllib3 already sends those as UTF-8, so this part only makes the behaviour explicit._hash_buffered_streamis unchanged, so the hash andbytes_transferrednow match what is sent.Things to know:
Transfer-Encoding: chunkedinstead of a Content-Length, since the encoded size isn't known without another pass over the stream. That is the path generators already take.read()returnsstrwithout being anio.TextIOBaseis not covered._upload_objectand is not changed here. S3 multipart, Azure Blobs, Backblaze B2 and Atmos already encode throughread_in_chunks.Status
Done, ready for review.
Checklist (tick everything that applies)
CHANGES.rst)Run locally:
libcloud/test/storage(972 passed, 12 skipped; also with urllib3 1.26.20), the full suite with tox's flags (7043 passed, 17 skipped), black, isort, flake8, mypy and rstcheck. The two new text tests fail on trunk; the binary pass-through test passes on both.Generated-by: Claude Code (Claude Fable 5.2). I reviewed and tested the change.
🤖 Generated with Claude Code