Comments (8)
samoconnor
commented on July 27, 2024
1
See: JuliaWeb/HTTP.jl#117
from awss3.jl.
jingpengw
commented on July 27, 2024
It is a problem of coroutine. Whenever I use multiple tasks, the readout of binary file changed. There is no gzip head anymore.
When I used 1 coroutine here, it works.
https://github.com/seung-lab/BigArrays.jl/blob/master/src/base.jl#L172
The writting works with multiple tasks.
from awss3.jl.
jingpengw
commented on July 27, 2024
@samoconnor you can reproduce the error with the following code. Changing the task number to 1 works.
whenever the task number is larger than 1, only one task will return the correct data. All other task will return empty array.
@testset "test coroute reading" begin
@sync begin
for i in 1:2
@async begin
@test s3_get(bucket_name, "key3") == b"data3.v3"
println("success ID: $i")
end
end
end
end Test Failed
Expression: s3_get(bucket_name, "key3") == @b_str("data3.v3")
Evaluated: UInt8[] == UInt8[0x64, 0x61, 0x74, 0x61, 0x33, 0x2e, 0x76, 0x33]
success ID: 2
test coroute reading: Error During Test
Got an exception of type CompositeException outside of a @test
There was an error during testing
Stacktrace:
[1] sync_end() at ./task.jl:287
[2] macro expansion at ./task.jl:303 [inlined]
[3] macro expansion at /usr/people/jingpeng/.julia/v0.6/AWSS3/test/runtests.jl:136 [inlin
ed]
[4] macro expansion at ./test.jl:860 [inlined]
[5] anonymous at ./<missing>:?
[6] include_from_node1(::String) at ./loading.jl:576
[7] include(::String) at ./sysimg.jl:14
[8] process_options(::Base.JLOptions) at ./client.jl:305
[9] _start() at ./client.jl:371
Test Summary: | Error Total
test coroute reading | 1 1
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
while loading /usr/people/jingpeng/.julia/v0.6/AWSS3/test/runtests.jl, in expression startin
g on line 135
======================================[ ERROR: AWSS3 ]======================================
failed process: Process(`/opt/julia-0.6/bin/julia -Cx86-64 -J/opt/julia-0.6/lib/julia/sys.so
--compile=yes --depwarn=yes --check-bounds=yes --code-coverage=none --color=yes --compileca
che=yes /usr/people/jingpeng/.julia/v0.6/AWSS3/test/runtests.jl`, ProcessExited(1)) [1]
============================================================================================
ERROR: AWSS3 had test errors
I have created a PR to add test.
#20
from awss3.jl.
jingpengw
commented on July 27, 2024
Note that before I upgraded to Julia 0.6, the mutitask reading works.
from awss3.jl.
samoconnor
commented on July 27, 2024
Hi @jingpengw, thanks for the report.
Just to confirm: are you using latest HTTP.jl, AWSCore.jl and AWSS3.jl ?
from awss3.jl.
samoconnor
commented on July 27, 2024
I've just confirmed that I get this behaviour with HTTP.jl:
@sync for i in 1:2
@async @show String(HTTP.get("http://octe.ch"))
end
String(HTTP.get("http://octe.ch")) = "<HEAD>\n<TITLE>octe.ch</TITLE> ... [snip] ... </BODY>\n"
String(HTTP.get("http://octe.ch")) = ""
String(HTTP.get("http://octe.ch")) = ""from awss3.jl.
samoconnor
commented on July 27, 2024
@jingpengw I have pushed a possible HTTP.jl fix for this to: https://github.com/samoconnor/HTTP.jl/tree/issue_117_branch
from awss3.jl.
jingpengw
commented on July 27, 2024
thanks! this was merged! we can close now.
from awss3.jl.
Related Issues (20)
- readavailable(::FileBuffer) missing in FilePathsBase < 0.9.18
- `KeyError: key "ETag" not found` in `s3_multipart_upload` HOT 6
- Deprecate global AWS config as first argument.
- `s3_get_file` usage of `BufferStream` breaks `AWS.jl` error handling HOT 4
- stat / s3_get_meta rely on case-sensitivity of headers HOT 3
- RFC: FilePaths2.jl
- Run tests with both AWS backends
- ArrowTypes support for S3Path fails with `missing` values HOT 5
- ERROR: LoadError: ArgumentError: `ssl_unsafe_read` requires `isreadable(::SSLContext)` HOT 8
- Support testing without AWS secrets HOT 6
- Add support for DeleteObjects
- `s3_get_file` allocates the whole file in memory before writing to disk HOT 1
- AWS version locked to 1.63.1 which limits us to HTTP 0.9? HOT 3
- Add a progress indicator hook
- no matching method found convert(::Type{Ptr{EzXML._Node}}, ::EzXML.Node) (1/2 union split):
- `S3Path` constructor does not respect its type parameter
- Support S3 Transfer Acceleration
- Config not passed along when concatenating with string
- Base.read method partially supported
- Documenting Cloudflare R2 Support
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from awss3.jl.