AWS IRSA Permission Denied

saydjari · September 12, 2025, 6:39pm

Trying to access “cloud data” since this platform is focused on keeping both compute and data in the cloud. In my jupyter notebook, I am trying to read in and access SPHEREx data. I am following SPHEREx Quick Release (QR): An All-Sky Spectral Survey - Registry of Open Data on AWS .

using AWSS3, AWS

aws = global_aws_config(; region="us-east-1")

which works fine, but then I just try to see what data is there by doing

p = S3Path("s3://nasa-irsa-spherex/qr/level2/", config=global_aws_config());

readdir(p)

and this returns a nasty error. I have tried several other s3 buckets, but none work. All of the documentation examples basically show access using specialized python packages that handle the queries, but that doesn’t work for me, since I am not in python. The basic AWS S3 query functionality needs to work, but maybe I am doing something silly?

AWS.AWSExceptions.AWSException: AccessDenied -- Access Denied

HTTP.Exceptions.StatusError(403, "GET", "/nasa-irsa-spherex?list-type=2&max-keys=1&delimiter=%2F&prefix=qr%2Flevel2%2F", HTTP.Messages.Response:
"""
HTTP/1.1 403 Forbidden
x-amz-bucket-region: us-east-1
x-amz-request-id: 2Q9JZ6E35424VNT3
x-amz-id-2: n6hNU/w7gF5SLNMsHQt9nTvJaNoxjwGwll1Dh8W1ALOTN7q5bZdLMRH8Wy6eVolq3zc56FvQoVk=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Fri, 12 Sep 2025 18:35:44 GMT
Server: AmazonS3

[Message Body was streamed]""")

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>2Q9JZ6E35424VNT3</RequestId><HostId>n6hNU/w7gF5SLNMsHQt9nTvJaNoxjwGwll1Dh8W1ALOTN7q5bZdLMRH8Wy6eVolq3zc56FvQoVk=</HostId></Error>


Stacktrace:
  [1] (::HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
    @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/JcAHX/src/clientlayers/ConnectionRequest.jl:144
  [2] connections
    @ ~/.julia/packages/HTTP/JcAHX/src/clientlayers/ConnectionRequest.jl:60 [inlined]
  [3] (::HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64})
    @ HTTP.RetryRequest ~/.julia/packages/HTTP/JcAHX/src/clientlayers/RetryRequest.jl:35
  [4] manageretries
    @ ~/.julia/packages/HTTP/JcAHX/src/clientlayers/RetryRequest.jl:30 [inlined]
  [5] (::HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64, retry::Bool})
    @ HTTP.CookieRequest ~/.julia/packages/HTTP/JcAHX/src/clientlayers/CookieRequest.jl:42
  [6] managecookies
    @ ~/.julia/packages/HTTP/JcAHX/src/clientlayers/CookieRequest.jl:19 [inlined]
  [7] (::HTTP.HeadersRequest.var"#defaultheaders#2"{HTTP.HeadersRequest.var"#defaultheaders#1#3"{HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Nothing, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{verbose::Int64, retry::Bool})
    @ HTTP.HeadersRequest ~/.julia/packages/HTTP/JcAHX/src/clientlayers/HeadersRequest.jl:71
  [8] defaultheaders
    @ ~/.julia/packages/HTTP/JcAHX/src/clientlayers/HeadersRequest.jl:14 [inlined]
  [9] (::HTTP.RedirectRequest.var"#redirects#3"{HTTP.RedirectRequest.var"#redirects#1#4"{HTTP.HeadersRequest.var"#defaultheaders#2"{HTTP.HeadersRequest.var"#defaultheaders#1#3"{HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Base.BufferStream, kw::@Kwargs{verbose::Int64, retry::Bool})
    @ HTTP.RedirectRequest ~/.julia/packages/HTTP/JcAHX/src/clientlayers/RedirectRequest.jl:17
 [10] redirects
    @ ~/.julia/packages/HTTP/JcAHX/src/clientlayers/RedirectRequest.jl:14 [inlined]
 [11] (::HTTP.MessageRequest.var"#makerequest#3"{HTTP.MessageRequest.var"#makerequest#1#4"{HTTP.RedirectRequest.var"#redirects#3"{HTTP.RedirectRequest.var"#redirects#1#4"{HTTP.HeadersRequest.var"#defaultheaders#2"{HTTP.HeadersRequest.var"#defaultheaders#1#3"{HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String; copyheaders::Bool, response_stream::Base.BufferStream, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{redirect::Bool, retry::Bool})
    @ HTTP.MessageRequest ~/.julia/packages/HTTP/JcAHX/src/clientlayers/MessageRequest.jl:35
 [12] makerequest
    @ ~/.julia/packages/HTTP/JcAHX/src/clientlayers/MessageRequest.jl:24 [inlined]
 [13] request(stack::HTTP.MessageRequest.var"#makerequest#3"{HTTP.MessageRequest.var"#makerequest#1#4"{HTTP.RedirectRequest.var"#redirects#3"{HTTP.RedirectRequest.var"#redirects#1#4"{HTTP.HeadersRequest.var"#defaultheaders#2"{HTTP.HeadersRequest.var"#defaultheaders#1#3"{HTTP.CookieRequest.var"#managecookies#4"{HTTP.CookieRequest.var"#managecookies#1#5"{HTTP.RetryRequest.var"#manageretries#3"{HTTP.RetryRequest.var"#manageretries#1#4"{HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}}, method::String, url::URIs.URI, h::Vector{Pair{SubString{String}, SubString{String}}}, b::String, q::Nothing; headers::Vector{Pair{SubString{String}, SubString{String}}}, body::String, query::Nothing, kw::@Kwargs{redirect::Bool, retry::Bool, response_stream::Base.BufferStream})
    @ HTTP ~/.julia/packages/HTTP/JcAHX/src/HTTP.jl:457
 [14] #request#20
    @ ~/.julia/packages/HTTP/JcAHX/src/HTTP.jl:315 [inlined]
 [15] macro expansion
    @ ~/.julia/packages/Mocking/eug7a/src/mock.jl:52 [inlined]
 [16] (::AWS.var"#48#50"{Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}})()
    @ AWS ~/.julia/packages/AWS/ZEOyZ/src/utilities/request.jl:222
 [17] (::Base.var"#106#108"{Base.var"#106#107#109"{AWS.AWSExponentialBackoff, AWS.var"#49#51", AWS.var"#48#50"{Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})(; kwargs::@Kwargs{})
    @ Base ./error.jl:300
 [18] (::Base.var"#106#108"{Base.var"#106#107#109"{AWS.AWSExponentialBackoff, AWS.var"#49#51", AWS.var"#48#50"{Request, OrderedCollections.LittleDict{Symbol, Any, Vector{Symbol}, Vector{Any}}}}})()
    @ Base ./error.jl:295
 [19] _http_request(http_backend::AWS.HTTPBackend, request::Request, response_stream::IOBuffer)
    @ AWS ~/.julia/packages/AWS/ZEOyZ/src/utilities/request.jl:247
 [20] macro expansion
    @ ~/.julia/packages/Mocking/eug7a/src/mock.jl:52 [inlined]
 [21] (::AWS.var"#41#44"{AWSConfig, Request, IOBuffer, Vector{Int64}})()
    @ AWS ~/.julia/packages/AWS/ZEOyZ/src/utilities/request.jl:131
 [22] (::AWS.var"#42#46"{AWS.var"#41#44"{AWSConfig, Request, IOBuffer, Vector{Int64}}, IOBuffer})()
    @ AWS ~/.julia/packages/AWS/ZEOyZ/src/utilities/request.jl:146
 [23] (::Base.var"#106#108"{Base.var"#106#107#109"{AWS.AWSExponentialBackoff, AWS.var"#43#47"{AWSConfig, Vector{String}, Vector{String}, Int64}, AWS.var"#42#46"{AWS.var"#41#44"{AWSConfig, Request, IOBuffer, Vector{Int64}}, IOBuffer}}})(; kwargs::@Kwargs{})
    @ Base ./error.jl:300
 [24] (::Base.var"#106#108"{Base.var"#106#107#109"{AWS.AWSExponentialBackoff, AWS.var"#43#47"{AWSConfig, Vector{String}, Vector{String}, Int64}, AWS.var"#42#46"{AWS.var"#41#44"{AWSConfig, Request, IOBuffer, Vector{Int64}}, IOBuffer}}})()
    @ Base ./error.jl:295
 [25] submit_request(aws::AWSConfig, request::Request; return_headers::Nothing)
    @ AWS ~/.julia/packages/AWS/ZEOyZ/src/utilities/request.jl:197
 [26] (::RestXMLService)(request_method::String, request_uri::String, args::Dict{String, Any}; aws_config::AWSConfig, feature_set::AWS.FeatureSet)
    @ AWS ~/.julia/packages/AWS/ZEOyZ/src/AWS.jl:352
 [27] RestXMLService
    @ ~/.julia/packages/AWS/ZEOyZ/src/AWS.jl:316 [inlined]
 [28] #list_objects_v2#132
    @ ~/.julia/packages/AWS/ZEOyZ/src/services/s3.jl:4839 [inlined]
 [29] list_objects_v2
    @ ~/.julia/packages/AWS/ZEOyZ/src/services/s3.jl:4836 [inlined]
 [30] _s3_exists_dir(aws::AWSConfig, bucket::SubString{String}, path::String)
    @ AWSS3 ~/.julia/packages/AWSS3/fnytU/src/AWSS3.jl:289
 [31] s3_exists_unversioned
    @ ~/.julia/packages/AWSS3/fnytU/src/AWSS3.jl:352 [inlined]
 [32] #s3_exists#13
    @ ~/.julia/packages/AWSS3/fnytU/src/AWSS3.jl:382 [inlined]
 [33] exists(fp::S3Path{AWSConfig})
    @ AWSS3 ~/.julia/packages/AWSS3/fnytU/src/s3path.jl:244
 [34] isdir(fp::S3Path{AWSConfig})
    @ AWSS3 ~/.julia/packages/AWSS3/fnytU/src/s3path.jl:263
 [35] readdir(fp::S3Path{AWSConfig}; join::Bool, sort::Bool)
    @ AWSS3 ~/.julia/packages/AWSS3/fnytU/src/s3path.jl:600
 [36] readdir(fp::S3Path{AWSConfig})
    @ AWSS3 ~/.julia/packages/AWSS3/fnytU/src/s3path.jl:599
 [37] top-level scope
    @ In[22]:3

djturner · September 12, 2025, 7:20pm

Hi!

I believe this is due to a credentials issue with the S3 bucket - as the IRSA bucket is in the AWS open data registry, we essentially have to send null credentials to go along with the request.

In Julia this can be achieved like this:

using AWSS3, AWS

aws = AWSConfig(; creds=nothing, region="us-east-1")

p = S3Path("s3://nasa-irsa-spherex/qr/level2/", config=aws_config)

readdir(p)

The only extra information is the `creds=nothing` argument, which is equivalent to the ‘–no-sign-request’ flag you might see in the AWS CLI sections on the bucket website.

I am in no way a Julia expert, but this solution worked for me during testing

saydjari · September 12, 2025, 7:27pm

For future documentation purposes, you might want to edit aws –> aws_config for self consistency. But yes, this worked for me. Thank you.

Fornax Community Forum

AWS IRSA Permission Denied