$Id: README 7147 2019-12-29 16:34:35Z flaterco $

    Harmbase 2:  harmonic constant management package.
    Copyright (C) 2004  David Flater.

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.


THIS PACKAGE IS AVAILABLE FROM:
http://www.flaterco.com/xtide/files.html


Introduction
------------

Harmbase is a database application for managing harmonic constants.  I built
Harmbase to solve my own problems, so it's not "plug and play" software.
It's not designed to be useful to someone who does not know how to hack Ruby
code or write interactive SQL queries.


Prerequisites
-------------

Dependency (w/ version tested)         get it from

XTide 2.15.2            http://www.flaterco.com/xtide/files.html
PostgreSQL 12.1         http://www.postgresql.org/
libdstr 1.0             http://www.flaterco.com/util/index.html
libtcd 2.2.7-r2         http://www.flaterco.com/xtide/files.html
congen 1.7              http://www.flaterco.com/xtide/files.html
Ruby 2.2                Your friendly neighborhood Linux distro
pg-1.1.4.gem            https://rubygems.org/gems/pg/

The command-line client tide (of XTide) must be in your PATH to run
fixup_datums and various test scripts.


What gets installed
-------------------

When you run make install, these two binaries get installed in $(prefix)/bin
(alias bindir):

  fixup_datums - estimate MLLW for stations that are missing it
  hbexport - dump the database to a TCD file

To avoid cluttering up bin, miscellaneous scripts are instead placed in
$(prefix)/share/harmbase2 (alias pkgdatadir) along with other dependencies.
This directory typically won't be in your path.

  harmbase2.sql - the harmbase2 schema
  scrape-MDAPI-0-stationlists.sh - download lists of tide and current stations
  scrape-MDAPI-1-tiderefs.rb - download harmonic constants and datums for tides
  scrape-MDAPI-2-geogroups.rb - download hierarchical geographic areas
  scrape-currents-regions-lists.sh - download HTML lists of current stations
  importNOS.rb - import data from CO-OPS Metadata API (MDAPI)
    importNOS_CurSubs.rb     - part of importNOS.rb
    importNOS_DodgyCode.rb   - part of importNOS.rb
    importNOS_Geogroups.rb   - part of importNOS.rb
    importNOS_JsonReaders.rb - part of importNOS.rb
    importNOS_Queries.rb     - part of importNOS.rb
    importNOS_TideRefs.rb    - part of importNOS.rb
    importNOS_TideSubs.rb    - part of importNOS.rb
    importNOS_Util.rb        - part of importNOS.rb
  parse-oldcurlists.rb - get geographic areas from HTML current station lists
  compare_LST_problems.sql - queries for testing
  misc_tests.rb - miscellaneous tests for common import problems
  getPreds.rb - given station ID, download predictions from NOAA web service
  sample.rb - pull predictions from tide and getPreds.rb for comparison
  dump_db.sh - export database to SQL without extraneous bits
  boilerplate.txz - data for dwf-export.sh
  dwf-export.sh - produce distributable SQL and TCD tarballs
  loclist.rb - dump station list to web page
  Debian.sh - build TCD file from SQL "source."
  README - this file


Setting up the schema
---------------------

Harmbase2.sql is the schema with no data sets.

$ createdb harmbase2
$ psql < harmbase2.sql     # schema file contains \connect harmbase2

Alternately, restore a database dump with data sets already in it.

$ psql harmbase2 < harmonics-dwf-*.sql


Importing
---------

As of the 2018 renovation, the import program works only with JSON files that
were downloaded from the CO-OPS Metadata API (MDAPI).  Please see "Details of
import process for NOS data" at the end of this README for the full process.

Usage: importNOS.rb N
N=1  tidal reference stations
N=2  tidal subordinate stations
N=3  current subordinate stations

Importing current reference stations is not supported at this time because
they are not available for download from the NOAA web site.  Importing
current subordinate stations may require additional steps to acquire
geographic area information that is not provided through MDAPI.

Always save a snapshot of the database BEFORE running import.  No attempt is
made to leave the database in a clean state when failures occur.

Harmonic constants that were produced by hargen are imported directly using
interactive SQL that harmgen produces.  The import program is not required
for that.


Deleting data sets
------------------

Referencing data in the table CONSTANTS will be deleted automatically via ON
DELETE CASCADE when a reference station is deleted from table DATA_SETS.
However, subordinate stations in the table DATA_SETS must be deleted before
the reference station.  (For safety, deletion does not cascade to sub
stations.)


Exporting
---------

Usage: hbexport [--flip constituent-name] [--optimize] [--free] [--nonfree] [-b YYYY] [-e YYYY] filename

--flip is a guru switch for troubleshooting the definitions of harmonic
constants.  It reverses the phase of the specified constituent.

--optimize causes export to eliminate unused constituents from the TCD file.
Otherwise, all constituents known to the database will be exported.

--free limits the export to data that are marked public domain.  By default
you get all data.

--nonfree limits the export to data that are NOT marked public domain.

-b and -e are used to select beginning and ending years for the harmonics
file.  The defaults are 1700 and 2100 (centered around 1900, the year for
which the speeds of the constituents are calibrated).


Details of import process for NOS data (December 2019)
------------------------------------------------------

These programs, scripts, and process break to some extent every year as the
NOS updates its web site.  This description includes notes on what was fixed
along the way this time.

Save snapshots of the database after every step (db_dump.sh > snapshot.sql).
Revert to the previous snapshot whenever it is necessary to fix something and
rerun (dropdb harmbase2; createdb harmbase2; psql harmbase2 < snapshot.sql).

Review reminders left in 2018.

  Check if harmonic constants for currents are available and usable yet.
    No.

  Check MDAPI changelog.  Look at
  https://tidesandcurrents.noaa.gov/mdapi/latest/.
    Current release is 1.0 (July, 2018).

Prepare database.

  Upgrade PostgreSQL; install or update pg gem for Ruby.

  Reload database from last year.
    bash-4.3$ createdb harmbase2
    bash-4.3$ psql harmbase2 < harmonics-dwf-20190620.sql

  Replace data_sets_old with the previous data.
    DROP TABLE data_sets_old;
    CREATE TABLE data_sets_old AS TABLE data_sets;

  Purge old data except for current refs.
    delete from data_sets where
      (units <> 'knots' and units <> 'knots^2') or datum is null;

  Checkpoint.
    dump_db.sh > database/00-start.sql

Revalidate the few remaining legacy current refs.

  hbexport 00-start.tcd
  export HFILE_PATH=00-start.tcd
  script -c "[path...]/sample.rb y CUR"

  New interactive map (tides and currents together):
    https://tidesandcurrents.noaa.gov/
  Old style listing of current stations:
    https://tidesandcurrents.noaa.gov/noaacurrents/Regions
  Change log for currents:
    https://tidesandcurrents.noaa.gov/current_pred_stn_history.html

  All 16 remaining legacy current stations remain within tolerances.  All are
  off by a few minutes; St. Johns River Entrance is also off slightly in
  speed.

Web scrape.

  scrape-MDAPI-0-stationlists.sh
  Run time less than 1 minute.
  (Note:  The counts can be found on line 2 of the json files.)
    tides.json
      3225 tide stations with offsets for sub stations but metadata only for
      refs.  Caution, includes great lakes.
    currents.json
      4284 current stations with offsets for sub stations but metadata only
      for refs.
    harcon.json
      1093 tide stations with links to get harmonic constants and datums.

  scrape-MDAPI-1-tiderefs.rb
  Run time over 6 hours.
    Read harcon.json and scrape harmonic constants and datums into separate
    directories and files.

  scrape-MDAPI-2-geogroups.rb
  Run time about half an hour.
    geogroups-tides.json
      Source of the multi-level headings in station listings.
    stations-by-group-tides.json
      Tide station metadata with pointers back to the relevant geogroup.

  scrape-currents-regions-lists.sh
  Run time less than 1 minute.
  Archive the most recent currents regions lists lest they should disappear.
  This is still the only way to resolve geogroups for new currents.

Look for errors in web scrape.

  Check for HTTP errors.
  bash-4.3$ fgrep HTTP tiderefs/scrape_log.txt | fgrep -v "200 OK"
  (Nothing)

  Check for truncated sets of constants.
  bash-4.3$ ls -alS `find . -name harcon.json` | less
    Matapeake, Kent Island (8572770) is the shortest with 18 constituents.
    Anchorage (9455920) has 120 constituents, descriptions after 37 are null.

Import tide refs.

  importNOS.rb 1 > import1-log.txt

  import1-log.txt:

  Skipped 1732417 Fare Ute Point
  Skipped 1778000 APIA, W. SAMOA
  Skipped 1910000 SUVA, SUVA HARBOR
  Skipped 2431000 Diego Garcia
  Skipped 2695535 Bermuda Biological Station
  Skipped 2695540 Bermuda, St. Georges Island
  Station 8531463 constituent SSA phase = 429.0
    Normalized to 69.0
  LST problem: 8728408 Dog Island, east end, St. George Sound, Florida
  LST problem: 8728486 St. George Island, NE end, St. George Sound, Florida
  LST problem: 8728669 St. George Island, Sikes Cut, St. George Sound, Florida
  LST problem: 8728694 White Beach, East Bay, Florida
  LST problem: 8728786 Eleven Mile, St. Vincent Sound, Florida
  LST problem: 8728853 White City, ICWW, Florida
  LST problem: 8728912 Port Saint Joe, St. Joseph Bay, Florida
  LST problem: 9454571 Storey Island (north side), Alaska
  LST problem: 9458519 Chowiet Island, Semidi Island, Alaska
  LST problem: 9462450 Nikolski, Umnak Island, Alaska
  LST problem: 9462723 Rootok Island, Rootok Strait, Alaska
  LST problem: 9463885 Zapadni Bay, St. George Island, Alaska
  LST problem: 9465283 Summit Island, NE side, Togiak Bay, Alaska
  LST problem: 9468039 Northeast Cape, St. Lawrence Island, Alaska
  LST problem: 9468258 Savoonga, St. Lawrence Island, Alaska
  LST problem: 9468283 Akeftapak, St. Lawrence Island, Alaska
  LST problem: 9468863 Elim, Norton Bay, Alaska
  LST problem: 9469237 Point Spencer, Port Clarence, Norton Sound, Alaska
  LST problem: 9469338 Lost River, Seward Peninsula, Alaska
  LST problem: 9497649 Prudhoe Bay #2, Alaska
  LST problem: 9497778 Cross Island (Dinkum Sands), Alaska
  Skipped 9500966 Madero, Tampico Harbor, Mexico
  Skipped 9635000 Puerto Quetzal (San Jose)
  Skipped 9640507 La Union
  Skipped 9650593 Puerto Cortes
  Skipped 9662957 Puerto Corinto
  Skipped 9710441 Settlement Point
  Skipped 9761115 Barbuda

  Diff from last year:
  18d17
  < Fixing longitude of 9460150 ATTU, MASSACRE BAY
  22,24c21
  < LST problem: 9464961 Cape Constantine, Nushagak Peninsula, Alaska
  < LST problem: 9465265 Kulukak Point, Alaska
  < LST problem: 9465283 Ne Side Summit Island, Togiak Bay A, Alaska
  ---
  > LST problem: 9465283 Summit Island, NE side, Togiak Bay, Alaska
  27a25
  > LST problem: 9468863 Elim, Norton Bay, Alaska

  Longitude of 9460150 ATTU was fixed in harcon.json.

  More churn of the Alaskan LSTs.  Ensure that timezones as assigned by
  import haven't changed.
  psql harmbase2 -f compare_LST_problems.sql -o compare_LST_problems.txt
  0 rows from third query = OK.

  Checkpoint.
    dump_db.sh > database/01-refs-imported.sql

Edit and run renamings-tiderefs.sql to fix up names of new stations.

  Note:  If this is onerous, changes to guessLongStationName might be in
  order.

  Checkpoint.
    dump_db.sh > database/02-refs-renamed.sql
    hbexport 02-refs-renamed.tcd

Install estimates for missing datums.

  The NOS import will set the datum to Mean Astronomical Tide (i.e., zero) on
  stations where the MLLW benchmark is unavailable.  When this happens,
  export a database, then run fixup_datums.

  fixup_datums will invoke tide in stats mode to estimate MLLW for each
  affected station over the 19-year epoch 1983-2001 and update the database
  accordingly.

  script -c "fixup_datums 02-refs-renamed.tcd"

  Note that changing meridians can affect the estimated MLLW.

  Checkpoint.
    dump_db.sh > database/03-fixed-datums.sql

Import tide subs.

  importNOS.rb 2 > import2-log.txt

  bash-4.3$ fgrep "Time warp" import2-log.txt
  Time warp at 8728912 Port Saint Joe, St. Joseph Bay, Florida
  Time warp at 9462144 North side, Yunaska Island, Alaska
  Time warp at 9462161 East Cove, Yunaska Island, Alaska
  Time warp at 9462239 Applegate Cove, Chuginadak Island, Aleutian Islands, Alaska
  Time warp at 9468123 Fossil River entrance, St. Lawrence Island, Alaska
  Time warp at 9468216 Niyrakpak Lagoon entrance, St. Lawrence Island, Alaska
  Time warp at TWC2223 Herbert Island, west side, Aleutian Islands, Alaska
  Time warp at TWC2229 Amukta Island, north side, Aleutian Islands, Alaska

  Same as last year.

  Skipped from blacklist:  113 (no change)
  Skipped from missing refs:  9 (no change)
  Missing refs are (no change):
  1778000 APIA, W. SAMOA (Samoa, dropped by import)
  1841275 MALAKAL HARBOR (Palau, allowable but missing)

  Diffing with last year, all but one of the "Fixing longitude"s went away.
  Remaining:  TWC2321 Etienne Bay

Edit and run renamings-tidesubs.sql to fix up names of new stations.

  Empty file.

  Checkpoint.
    dump_db.sh > database/04-tidesubs-imported.sql

If needed, load geogroups for new currents.

  For 2019-12, only need Washington.
  wget -O Washington.html \
    https://tidesandcurrents.noaa.gov/noaacurrents/Stations?g=698
  parse-oldcurlists.rb oldcurlists/Washington.html
  (Creates table currents_geogroups)

  Checkpoint.
    dump_db.sh > database/05-currents_geogroups.sql

One-time naming fix to current refs loaded without geogroups 2019-06.

  Not in any oldcurlists: PUG1525 PUG1633 PUG1737 PUG1738

  Total:           924
  Changed:         760
  Nothing to add:   85
  Too long:         79

  Checkpoint.
    dump_db.sh > database/06-renamed-summer-current-refs.sql

Import current subs.

  Acquired 32 new or previously dropped current subs.  They refer to 7 ref
  currents that were imported over the summer:
  PUG1539_12 PUG1624_47 PUG1640_18 PUG1702_16 PUG1703_16 PUG1708_16 PUG1729_21

  importNOS.rb 3 > import3-log.txt

  Ended up with three (sub) tags for Waldron Island
  PUG1716_3, PUG1716_13, PUG1716_23

  Log is identical to last year's.

  11 time warps, 7 skipped (B.C. on Wrangell), 1 null multiplier:

  Fixed null multipliers at PCT4416_1 Carson Creek Entrance, 1.4 nmi. ESE of (Depth 15ft), Icy Bay, Alaska Current

  Checkpoint.
    dump_db.sh > database/07-all-imported.sql

Edit and run renamings-cursubs.sql to fix up names of new stations.

  No changes required.

If applicable, drop currents_geogroups.

  Checkpoint.
    dump_db.sh > database/08-cleanedup.sql

One-time fix for inconsistent capitalization of "[north/south/east/west] jetty"

  8 stations renamed.

  Checkpoint.
    dump_db.sh > database/09-fixed-jetty.sql

One-time fix for inconsistent capitalization of "entrance"

  341 ~* 'entrance'
    254 ~ 'entrance
     87 ~ 'Entrance'
       18 ~ '^Entrance'
       69 ~ '.Entrance'
         11 ~ 'Dixon Entrance' (proper name, disputed water)
         58 need fixing

  update data_sets
    set name = regexp_replace(name, 'Entrance', 'entrance')
    where name ~ '.Entrance' and not name ~ 'Dixon Entrance';
  UPDATE 58

  update data_sets
    set name = 'Matagorda Bay entrance channel, Texas'
    where name = 'Matagorda Bay entrance Channel, Texas';
  UPDATE 1

  Checkpoint.
    dump_db.sh > database/10-fixed-entrance.sql
    hbexport 10-fixed-entrance.tcd

Misc tests.

  Run misc_tests.rb and either fix and rerun import or make manual fixes as
  appropriate.

  Checking for duplicate names: pass
  Checking for stations missing states: pass
  Checking for stations that switched hemispheres: pass
  Checking for Pacific islands in the wrong hemisphere: pass
  Checking time zones on St. Lawrence Island: FAIL
  9468261
  Kavalghak Bay, St. Lawrence Island, Alaska
  Checking time zones for Port Saint Joe and White City, FL: pass
  Checking for any subs referring to Port Saint Joe and White City, FL: pass

  Oops.  9468261 also has an estimated datum.

  update data_sets
    set timezone = ':America/Adak',
    notes = 'NOS uses UTC-9 for LST; HAST (since 1983) is UTC-10',
    datumkind = 'Mean Astronomical Tide',
    datum = 0,
    comments = 'Datum not available from MDAPI'
    where station_id = '9468261';

  hbexport fixKavalghakBay.tcd
  fixup_datums fixKavalghakBay.tcd
  Datum still 1.31

  Checkpoint.
    dump_db.sh > database/11-fixed-KavalghakBay.sql
    hbexport 11-fixed-KavalghakBay.tcd

  Reason for failure:  guessTimezone was looking for "St. Lawrence Island" in
  the name, but at the time the name was "Kavalghak Bay, AK, Alaska."  The
  name was fixed after the fact in renamings-tiderefs.sql.  Changed
  guessTimezone to test coordinates instead of name.

  bash-4.3$ diff import1-log.txt import1-postfix-log.txt
  23a24
  > LST problem: 9468261 Kavalghak Bay, AK, Alaska

  Count stations and make sure the numbers look reasonable.

Coverage tests.

  (Unpublished script coverage.sh)
  Test at least one station of each type:
    Tide ref
    (Current refs were already tested)
    Tide sub, * offsets
    Tide sub, + offsets
    Current sub, knots
    Current sub, knots^2
    Ref wrong LST (UTC matches, LST/LDT doesn't)
    Sub time warp (LST/LDT matches, UTC doesn't)

Random tests.

  Check random samples using sample.rb until exhausted.  Note that the
  predictions retrieved for comparison with a (sub) station will likely be
  from a corresponding harmonic station and therefore won't match very well.

Check for reminders, loose ends, TO-DOs, and FIXMEs.

Finish.

  Update ChangeLog in boilerplate.txz and run dwf-export.sh.

  Upload.

  Update station list on web using loclist.rb.

  Update files.html and news.html.


Reminders for December 2020
---------------------------

  Check if harmonic constants for currents are available and usable yet.

  Check MDAPI changelog.  Look at
  https://tidesandcurrents.noaa.gov/mdapi/latest/.

  The tidal datum epoch rolls over Jan 2021?

  Consider making changes to guessLongStationName (see FIXME comments).
