Apparent limit to the number of files transferred from s3 bucket

Helpdesk Data Access
1

I was trying to download PPS files for an XMM Obs ID. When I use astroquery to download PPS data files from AWS it only downloads exactly 1000 files, which is not all of the files. But when I use astroquery to download the same data from the HEASARC it downloads all 3129 files.

I’m using the Obs ID = 0123700101

3

I just tested it on my local computer and I have the same issue. So it’s not a Fornax specific issue, but it is an AWS/astroquery issue.

4

Apparently, AWS S3 API has a limit of 1000 keys per request, and that is the default in astroquery.heasarc. It will need to be updated to handle more files. Until then, you can use the aws command line (install with pip install awscli):

aws s3 --no-sign cp --recursive s3://nasa-heasarc/xmm/data/rev0/0123700101/PPS PPS