Fully implementing python version of USGS variometer data retrieval

[DCarSunDr]

We've determined that python can provide a more time-efficient means of retrieving the 10 Hz variometer data (see #392), and so we are pressing forward with a full adaption of "gmag_usgs_download_variometer_files.php"; in this case, I've dubbed the python version "gmag_retrieve_usgs_variometer.py", for conciseness.

We will need a script in /src/thmsoc (do we want to add a subdirectory?) which handles the actual retrieval, and a src/thmsoc/cli script which handles argument parsing.

We should determine which functions (if any) should be split into separate scripts, to make it easier to adapt other networks/use cases. For instance, an availability checker could be generalized

If we'd like to fold the "call_gmag_usgs_download_variometer_files.ksh" script into "gmag_retrieve_usgs_variometer.py", we'd need to add support for logging, possibly db_log, and enable it to send the database queries file to the mysql server.

Currently, "gmag_retrieve_usgs_variometer.py" is able to retrieve a file given the station name, a date, and a sampling rate. It as a function to construct USGS query URLs (which we might want to make part of a separate thmsoc/thmsoc_gmag url handling script, since USGS observatory magnetometers use a similar thing, but a different path, which could be handled as an argument.). It can also split the retrieval into segments, by default using 24 hour segments for 1 Hz data and 4 hour segments for 10 Hz data.

Additionally, "gmag_retrieve_usgs_variometer.py" will need to be able to:

• Accept a list under a "station names" argument
• Loop over the passed station names, and for each station name, loop over the start and end dates / date + days to retrieve data files
• Handle USGS and THEMIS station name alias. Since the toml file is used for paths, we might want a "gmag" config/definitions file which we could load under specific parameters
• Check availability from IRIS/EARTHSCOPE (maybe should also be a separate script)
• Load latest calibration history date (note: will likely need to handle xml format, but should be do-able under urllib request under their decode options)
• Create database queries as strings, append them to .sql file
• write to mirror_dir for 1 Hz data (need to update toml)
• Write files (maybe start with a temporary directory), and if script is interrupted, delete file. Once script is complete, make sure files get placed into proper workdir

[DCarSunDr]

Tentatively, I've added a GMAG_USGS_VARIOMETER_SITES list, a GMAG_USGS_VARIOMETER_SITES_TO_SOURCEALIAS dictionary, and a GMAG_SOURCE_URL dictionary to the get_categories method of product_volume.py. These will be included in the dictionary returned by get_categories, but since dictionary values should be explicitly called, they shouldn't crop up where they're not supposed to be.

The GMAG_USGS_VARIOMETER_SITES list should be the main dictionary item called for various THEMIS contexts, but the GMAG_USGS_VARIOMETER_SITES_TO_SOURCEALIAS will be important for retrieval, since we occasionally must use a different station code from that of the source (like in the case of USGS variometers, where we label "KEV" as "KEVO" to avoid duplicates of our other station codes). The URL dictionary contains not only the data network location, but also the metadata (availability and last update dates) as well.

I figured this would make the most sense since we have ASI site lists here. I could see an argument for collapsing all of this into a "usgs networks dictionary". It would be great to keep things like station codes in sync between thmsoc and spedas libraries by simply importing a definitions scheme.

[jameswilburlewis]

Sounds good so far!

With the flakiness of especially the 10hz USGS downloads, I'm wondering if it might be worth adding some logic for automated retries or timeouts, like we recently added to spd_download? (Though a 404 result that you saw at one point probably isn't something we'd want to retry....). I don't know if spd_download itself would handle your use case (working via an API rather than file-by-file downloads...maybe that would still work?), but it's possible a similar technique could be applied to however you're doing it. I think urllib3 handles most of the logic for you...it was pretty easy to support in pyspedas. We already include pyspedas as a dependency in thmsoc_python, so you should feel free to leverage any relevant pyspedas tools.

[DCarSunDr]

I think urllib3 would definitely be better than the built-in urllib I've been using up to this point. Maybe we could have an argument to tell the script to retry retrievals vs giving retrievals only one attempt. Perhaps a third setting for this option to make retries only after the rest of the files have had their first attempts made.

I think we might want to leave such functionality off, or have the retries relegated to the end of the retrieval when it comes to our regular SOC processing--since there are 350 files to write in a daily 10 Hz variometer retrieval, we don't want the script to get too hung up a single file, possibly pushing the total retrieval time past the 24 hour mark.

The database tables for the variometers record if there was data availability suggested on the IRIS/EARTHSCOPE web service, the last attempted retrieval datetime, and the last successful retrieval datetime. So we know if a retrieval attempt was made and failed, and more importantly we know that the data SHOULD be available, at least if the IRIS/EARTHSCOPE availability (which mainly tracks the raw data) is anything to go by.

Since we have availability checkers and a maintained database, I was thinking we could have a script on a separate machine (talos?) which checks the database, and can even cross-check the 1 Hz database if the file SHOULD have been written, but didn't for whatever reason. It could then create its own queue and run a retrieval when the daily retrieval on cronus should be offline, basically calling the same script. We could even give this "backlog" retrieval multiple attempts, because we know that the files should be there. Hopefully this would only be a short list, but for big time ranges the number of files to write could be comparable to regular retrieval processes.

Maybe for regular retrievals, we give skipped files 1 or 2 retries at the end of the retrieval, and have this other script revisit older files beyond the regular retrieval window.

[DCarSunDr]

After looking through the download.py pyspedas utility script, I see what you mean about the retry methods. Since 10 Hz data must be downloaded in segments (due to the USGS query returned character limit), we could essentially pass the segment queries and the base url, but we may want to modify the base script to accept an optional retry attempts number argument. I think it defaults to 0. I think we could also just call the download_file function, if we have the script do the necessary prep, which might be the nicer option since we could leave pyspedas alone.

The weird thing is that the segment retrieval isn't downloading a file per se, but writing the contents of the page to a response object (in bytes, decoded to utf-8).

And I'm not sure about those 404 errors; not sure if I need to buffer the time between request attempts.

As it currently stands, I'm using the built in urllib module, but it would be better to leverage existing methods if we could make them work here.