Skip to content

BUG: Fix for broken alma downloader#2490

Merged
bsipocz merged 5 commits into
astropy:mainfrom
keflavich:alma_downloader_Aug11_2022
Sep 6, 2022
Merged

BUG: Fix for broken alma downloader#2490
bsipocz merged 5 commits into
astropy:mainfrom
keflavich:alma_downloader_Aug11_2022

Conversation

@keflavich

Copy link
Copy Markdown
Contributor

This should fix #2489. The failure is already covered by tests.

@keflavich keflavich requested a review from andamian August 11, 2022 15:06
@codecov

codecov Bot commented Aug 11, 2022

Copy link
Copy Markdown

Codecov Report

Merging #2490 (0828aa2) into main (0883e92) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #2490      +/-   ##
==========================================
- Coverage   62.99%   62.98%   -0.02%     
==========================================
  Files         133      133              
  Lines       17291    17297       +6     
==========================================
+ Hits        10892    10894       +2     
- Misses       6399     6403       +4     
Impacted Files Coverage Δ
astroquery/alma/core.py 47.83% <0.00%> (-0.37%) ⬇️
astroquery/mast/observations.py 76.62% <0.00%> (+0.20%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@keflavich keflavich added the alma label Aug 11, 2022

@ceb8 ceb8 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@keflavich

Copy link
Copy Markdown
Contributor Author

Thanks. I'd like to have @andamian 's eyes on this before merging in case there is a reason to take a different approach (e.g., there are some other metadata we should be inspecting to determine what action to take in these blank-line cases)

@bsipocz bsipocz changed the title Fix for 2489: broken downloader BUG: Fix for broken alma downloader Aug 11, 2022
@bsipocz

bsipocz commented Aug 11, 2022

Copy link
Copy Markdown
Member

@keflavich - I don't see any of the tests to fail even without this PR, so they'll need to be adjusted to make sure they catch the pre-PR regression.

Also, I haven't looked into the details, but recall that #2438 might be in play here?

@bsipocz bsipocz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests should be added/refactored for this. Also it needs a changelog.

@at88mph

at88mph commented Aug 11, 2022

Copy link
Copy Markdown
Contributor

This is already addressed in #2438 .

@keflavich

Copy link
Copy Markdown
Contributor Author

OK, then I'll defer to that PR, but it's not at all obvious to me how this is addressed there. The underlying infrastructure is opaque to me.

The test that needs to fail is indeed marked xfail because it's awaiting #2438.

So I guess the new datalink service was deployed in the last ~48 hours? I have a daily cron job that started failing today, but was fine yeserday.

@bsipocz

bsipocz commented Aug 11, 2022

Copy link
Copy Markdown
Member

I'll try to get back to that PR this afternoon, as I recall my reservations were all about the modifications to the generic tooling, and nothing about the specifics in the alma module.

@bsipocz bsipocz closed this Aug 17, 2022
@bsipocz

bsipocz commented Aug 17, 2022

Copy link
Copy Markdown
Member

Closing this in favour of #2493

@keflavich keflavich reopened this Aug 22, 2022
@bsipocz bsipocz added bug and removed duplicate labels Aug 22, 2022
@bsipocz

bsipocz commented Aug 22, 2022

Copy link
Copy Markdown
Member

Can you rebase and add a changelog entry?

author Adam Ginsburg (keflavich) <keflavich@gmail.com> 1660230509 -0400
committer Adam Ginsburg (keflavich) <keflavich@gmail.com> 1661190088 -0400

attempt to fix broken downloader

account for length-1 lists

account for length-1 lists

fix logic: the number of uids doesn't matter ,the number of files does

the total size doesn't need to be filtered out cleverly, since things
that aren't files don't have size
@keflavich keflavich force-pushed the alma_downloader_Aug11_2022 branch from 59260ec to ca906ee Compare August 22, 2022 17:42
@bsipocz bsipocz added this to the v0.4.7 milestone Aug 22, 2022
@bsipocz

bsipocz commented Aug 22, 2022

Copy link
Copy Markdown
Member

Are those datalinks copies of the same products that are already been listed with non-blank access_url?

For merging, this will need the review and approval of someone knowing the details of the ALMA API, @at88mph, or @andamian.

@keflavich

keflavich commented Aug 22, 2022

Copy link
Copy Markdown
Contributor Author

Are those datalinks copies of the same products that are already been listed with non-blank access_url?

afaict, yes, the blank lines correspond to the tarballs that are now excluded from the download b/c they've been expanded.

There is likely to be a more rational way to handle this, but this is the bugfix I need now because the currently deployed version is broken. So, if other users stumble across this before we resolve the issue, this PR is an acceptable temporary workaround.

@keflavich

Copy link
Copy Markdown
Contributor Author

@at88mph @andamian will either of you be able to review this PR this week?

If not, I recommend we merge it and request post-merge review to see if there is a better solution, but we need this as a patch now.

Comment thread astroquery/alma/tests/test_alma_remote.py Outdated
Co-authored-by: Eero Vaher <eero.vaher@gmail.com>
@at88mph

at88mph commented Sep 6, 2022

Copy link
Copy Markdown
Contributor

Good catch @keflavich, thanks for this.

@bsipocz

bsipocz commented Sep 6, 2022

Copy link
Copy Markdown
Member

I take the comment of @at88mph from above as an approval, so am going ahead and merging this now. Thanks, @keflavich!

@bsipocz bsipocz merged commit 4c87d7f into astropy:main Sep 6, 2022
ceb8 pushed a commit to orionlee/astroquery that referenced this pull request Sep 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ALMA data download broken

5 participants