---
title: "QA/QSea User Manual"
author: "Christopher Swingley^1^, Gary Drew^2^, Heather Coletti^3^"
execute:
  echo: true
  eval: false
  warning: false
  error: false
format:
  html:
    toc: false
  pdf:
    toc: true
    colorlinks: true
    geometry:
      - top=0.75in
      - left=0.75in
      - right=0.75in
    pdf-engine: xelatex
    mainfont: "Source Sans Pro"
    sansfont: "Source Sans Pro"
    fontsize: "12pt"
    mainfontoptions:
      - Numbers=Uppercase
      - Numbers=Tabular
---

^1^ABR, Inc. - Environmental Research & Services 2842 Goldstream Rd.
Fairbanks, AK 99709

^2^U.S. Geological Survey, Alaska Science Center, 4210 University Dr.
Anchorage, AK 99508

^3^National Park Service, 4175 Geist Rd., Fairbanks, AK 99709

# Introduction

[QA/QSea](https://sealog.abrinc.com/pages/qaqsea.html) is a Shiny app written
in the R environment. This app was constructed to work in conjunction with
[SeaLog](https://sealog.abrinc.com/pages/sealog.html), a data logging
application developed by Department of Interior researchers in Alaska for the
collection of at-sea survey data. QA/QSea can read comma delimited text files
(csv) output from SeaLog, provide proofing tools, and generate exportable data
files in multiple formats including that used in the [North Pacific Pelagic
Seabird
Database](https://www.usgs.gov/data/north-pacific-pelagic-seabird-database-nppsd)
(NPPSD). Note this app cannot read dLOG or any other csv data, as it requires
JSON configuration files from SeaLog. A previous version of the app
(dLog_final) can be used for this purpose and is available.

# Requirements and Setup

The app needs a recent version of R, 4.1 or higher, and is best run from
RStudio, version 1.2 or higher. R can be downloaded from
[CRAN](https://cran.r-project.org/), and RStudio is available at
[https://posit.co/download/rstudio-desktop/](https://posit.co/download/rstudio-desktop/)

Once these are installed, open RStudio and install the required packages using
the following statement:

```{r}
install.packages(c("shiny", "shinyjs", "shinythemes", "bslib", "tidyverse",
  "stringr", "glue", "lubridate", "readxl", "hms", "knitr", "DT", "sp", "sf",
  "viridis", "leaflet", "leafem", "htmltools", "zoo", "jsonlite", "readxl"))
```

Installing these packages will take several minutes depending on the speed of
your network connection and whether your computer is building them from source
or installing pre-build binaries. On platforms where R libraries are compiled
from source (Linux, OS X), you may need to install additional system libraries
such as udunits, and libxml2. Note: if you cut and paste the above statement
and get an error, try deleting and retyping the quotation marks.

If you are running an older version of R you can download the latest version,
but if your version has changed significantly (either the major or minor number
has changed, like 3.4 → 3.5 or 3.4 → 4.0) you will need to re-install the
packages listed above.

To upgrade your existing packages to the latest version, you can do this in the
console by running `update.packages(checkBuilt = TRUE, ask = FALSE)`, or go to
the **packages** tab in RStudio and click the **Update** button.

# Reference Files, Variables

## Built-in reference files

There are two files installed in the **references** subdirectory that are used
to set defaults in QA/QSea

### Species Lists

Species lists are used to convert four letter species codes into common names
and for summary and assignment of taxonomic classes for the NPPSD. The file is
named **SeaLog-Species_CodeList.csv**.

Additional codes can be added to the taxonomic code list file using the same
format as the existing lines.

Validation within the app is based on the configuration file
**speciesValidationList.json** that was used by SeaLog during data collection.
This file is used by SeaLog for real-time validation and is also required for
QA/QSea to identify questionable observations.

### NPPSD References

The **nppsd_reference_table_data.csv** file contains the list of valid
behaviors, weather codes, platforms and survey types used by QA/QSea to
validate these fields in your observational data. If you are not submitting
data to the NPPSD and you want different codes for these fields, you can change
this reference file using the same format.

## SeaLog Data

You will also need three files from SeaLog. The observational data, and two
JSON configuration files.

### Observational Data From SeaLog

QA/QSea requires three files to operate. The SeaLog\>export folder will contain
the survey observations and settings csv files. Exported files may be
identified by their time stamp included in the name, e.g.
**observations_20210426.csv**. The settings file will identify the configuration
file names and locations used for data collection. Note: in the settings file,
forward and backward slashes are interchangeable.

### Optional Fields

The app uses the SeaLog configuration file **optionalFields.json** to determine
which of the fields used during data collection should appear in the outputs.
With a few exceptions, such as *Latitude* and *Longitude, Date* and *Time*, and
*GPSTime*, only the fields enabled in the SeaLog app will appear in your
outputs. The input csv file (and any backups created by the app) will contain
all possible Seabird fields, but those that aren't enabled will be empty (null
in the CSV, NA in R).

### Species List

You also need the species list file you used with SeaLog,
**speciesValidationList.json**. This contains the list of species codes that
are valid for your survey data. Codes in your data that aren't in this list
will be flagged in QA/QSea so you can correct them.

### NPPSD Species Crosswalk

One optional file, **nppsd_species_crosswalk.csv**, is a file that maps your
species codes to those used with NPPSD. It has three columns: "Species Code",
"Common Name", and "NPPSD Species Code". For example, if on your surveys you
code adult and juvenile Bald Eagles (BAEA) separately, you'd use the following
CSV file:

```{txt}
Species Code,Common Name,NPPSD Species Code
BAEAA,Bald Eagle (adult),BAEA
BAEAJ,Bald Eagle (juvenile),BAEA
```

You would also need to include `BAEAA` and `BAEAJ` in your
**speciesValidationList.json** so they would be listed as valid species in
SeaLog and QA/QSea.

# Starting the App

Although the app will run directly from the R command window, most
users will find running it in the RStudio environment will be easier.

To start the app in RStudio:

- Open RStudio

- Open the Project associated with the app, probably named something similar to
  seabird_data_nps. If you don't have a Project associated with the app (or you
  don't know what a project is), simply open RStudio and follow the directions
  below.

- In the **Files** pane, typically in the lower right, open the shiny_app
  folder. This folder should contain global.r, server.r, ui.r, and several
  subfolders.

- Click the **More** drop-down arrow (next to the blue gear icon) and select
  "Set As Working Directory".

- Go to the R Console, which is typically either the full left pane or the
  lower left pane in RStudio.

- Load the shiny library:

  ```{r}
  library(shiny)
  ```

- Run the app:

  ```{r}
  runApp(launch.browser = TRUE)
  ```

A new window should launch in your web browser (@fig-input-page). If you close this
window, the app will still be running within RStudio, and you can get back to
it by scrolling up to where you entered runApp() in your RStudio console,
copying the link shown below that statement, and pasting it into your web
browser's location bar. It looks something like http://127.0.0.1:3924, but the
number at the end will vary each time you run it.

If you closed the browser window and you do not want to return to the app,
press Ctrl-c after clicking in RStudio's Console window to close it.

The Shiny library only needs to be loaded once within an R session in RStudio,
so if you want to start the app again, you only need to type
`runApp(launch.browser = T)`.

![Summary of ingested csv file text and numeric values. Unique values can be
globally edited on this page.](media/image1.jpg){#fig-input-page}

# App Organization

The app is organized into nine tabs. Typically, users will work through the
processing of your file or files from left to right in the app, though some
tabs may not be of interest for all projects. Input files from SeaLog are below
the "Default" field values.

## Input Files

The first tab, **Input Files**, Users select values for *Platform Type, Survey
Type*, and method used to count flying bird (*Fly Bird Method*. For the
*Source* users should enter the name of the data manager (typically the person
running QA/QSea) and the *Principal Investigator*. QA/QSea allows users to
select the segment length using a slider. Note the results of this splitting
will be shown only on the **Split Transects** and **Split Observations** tabs.
The NPPSD tabs **tbl_LOCATION** and **tbl_DATA_OBS** will be based on 3.3 km
length transect segments unless the check-box "Use split length for NPPSD" is
selected. By default, the "Fix positive longitude" is selected to automatically
correct for missing negative signs. Note: If users are operating in areas with
a positive longitude, e.g. Russia, you will want to uncheck this box. In
addition, once you have opened files, you cannot change the defaults without
re-opening the files so be sure to populate all the fields in the Defaults
section.

If you do not change the default "split distance" on the slider it will split
transects into smaller \~1 km segments. This process works similar to how
transects are split to produce NPPSD-ready data, except you can choose your own
split distance. The logic is the same: each transect will be split into pieces
as close in length to the distance selected, with the last segment as least if
the split distance and no longer that two times the split distance. This
typically means the last segment is longer than the rest of the segments,
rather than shorter than the rest. Note: splitting occurs within transect. For
example, 2 transects each of 2.6 km in length will each yield 2 "split
transects" one \~1 km and a second \~1.6 km (using a 1 km split length). Even
if the transects were concurrent, the app will not join them prior to
splitting. Thus, transect lengths should be a consideration during the survey
design to maximize samples.

There are four file selection boxes, one for selecting one or more input CSV
file (**CSV Data Files**), one for choosing the optional fields JSON file that
was used during the data collection (**Optional Fields JSON File**), one for
choosing the species validation JSON file used by SeaLog (**Species List JSON
File**)(@fig-input-page). All three of these file entry fields need to be filled in
before the app will read and process your data. There's also an optional file,
**NPPSD Species Crosswalk** that maps your species codes and common names to
NPPSD codes.

After selecting the JSON configuration files used during data collection and
one or more data files, click the **READ DATA** button and the app will read
each file and bring the data into memory. A progress bar will appear in the
lower right corner with details on what the app is working on. When the
progress bar disappears, a variety of summary information appears below the two
buttons.

In the RStudio Console window, you may see warnings while the app reads the
data. These are normal and can happen when R attempts to convert empty values
to particular types like dates or numbers.

### Statistical Summary

Below the Input fields and buttons, is a statistical summary of the data
ingested (@fig-stat-summary). This gives you an opportunity to identify statistical
anomalies.

![Statisical summary of ingested data.](media/image3.jpg){#fig-stat-summary}

## Global Changes

The next tab over **Global Changes** shows you the **Unique text values**
(fields like observers, behaviors, species, and survey information), **Unique
integer values** (Beaufort codes, condition, and transect width), and the
species codes used on the survey that don't match NPPSD codes, **NPPSD Species
Crosswalk** (@fig-global-changes).

At the top of the page is a checkbox labelled "Automatically update data when
changing values". This checkbox is typically left unchecked so you can make
global changes to the values on this page without having to wait for the data
processing to complete. When you are finished making changes, you need to be
sure to check this box in order to reprocess your data. If you do not do this,
the data shown in the subsequent tabs will not have your global edits.

In cases where the data for a field was absent, text fields have been populated
with the word **NULL**, and integer fields with the number **9999**. Be sure
you find and replace these values with the proper code or number. Also keep in
mind that these changes are global, so if you have a situation where you left
Transect blank for multiple transects, you cannot use the **Global Changes**
tab to correct the issue because any replacement of NULL with a proper transect
will replace *all* off the missing values.

In order to make changes, double-click in the cell you want to change, edit the
value, then hit the Tab key or click outside the box. The table will refresh
itself and recount each category.

Pay particular attention to the **Transect**, **TripID** (text values) and
**TxWidth** (integer) fields. These are used to generate unique transect IDs
and calculate densities, so if they are incorrect, you will wind up with
incorrect or split transects and miscalculated densities. Also be sure to
either correct, or resolve situations with **NULL** or **9999** in the value
column.

![Graphical user interface, Global Changes](media/image2.jpg){#fig-global-changes}

::: {.callout-important}

Before moving to the next tabs, be sure to check the **Automatically update data when
changing values** checkbox at the top of the page.
:::

## Transects Tab

Location information is presented in the next 3 tabs, **Transects, Split
Transects, and tbl_LOCATION**. Following the processing of the data files and
display of summary information, users should select the **Transects** tab to
examine and, if necessary, edit the transect level data. Transects are grouped
by *TripID, Transect* and date, calculated from the timestamps of each point
(both GPS and observational data) (Figure 4.)

![Transects tab listing transects based on ID. Fields are sortable using the
button next to each field name (red
circle).](media/image4.jpg){width="6.539583333333334in"
height="5.988888888888889in"}


Additional QA/QSea columns have been added to the data from SeaLog. Because the
transects table is an aggregation of field values during a transect, it is
possible for there to be differences between values in a field within a
transect. In those cases, you will see all the values in the column just to the
right of the data column. For example, if you had a *TXWidth* of 200 m for some
observations in a transect and 300 m for others in the same transect, you would
see **200, 300** in the *tw_all* column. If the value that's actually in the
*TXWidth* column isn't correct because the app chose the wrong one, you can
change it to the other value.

The other QA/QSea columns include **invalid_columns**, which displays a list of
columns that contain values that don't match the list of accepted values or are
outside the acceptable ranges for them. The table is sorted by this column, so
[all the potentially incorrect rows appear at the top.]{.ul}

To change values, double-click on the cell you want to change and edit the
value. When you leave the cell, the data in memory is updated, any derived data
is re-calculated (like the data in the **tbl_LOCATION** or **Split Transects**
tabs), and a row will appear at the bottom in the section titled "Edited rows
(original content)". This table displays what the original data was for any row
that you've modified. Rows that have been edited will also have today's date in
the edited column, the second from the right side of the table.

If you want to delete a column from the views and downloaded results, enter any
text in the last column, delete.

Whenever an edit is made, the app saves a backup copy of all the data in the
backups directory, directly under the shiny_app directory you started from.
These files are named raw_data\_ with a timestamp in YYYYmmddHHMMSS format at
the end so they will sort correctly. If the app crashes, or you realize you
made an edit in error, you can re-load the raw data from one of these backup
files instead of the original SeaLog CSV file and any changes you made prior to
the timestamp will be preserved.

## Split Transects

The next tab "Split Transect" presents data where the original transects have
been split into lengths based on the slider on the Input Files tab. The
transect segments can be identified by a new ID based on **TripID\_Transect
Number_Start Time_segment number**. This retains linkage to the original
transect (Figure 5). Fields in this tab behave similarly to those in the
Transects tab but are summarized by transect-segment, not transect. Note:
**invalid_columns** data identified in the **Transects** tab are not shown in
the **Split Transects** tab. Editing of transect data should be done in the
**Transects** tab. This file in csv format can be downloaded using the button
at the bottom of the page.

![Split transects tab.](media/image5.jpg){width="6.53951990376203in"
height="5.988888888888889in"}

## tbl_LOCATION

The **tbl_LOCATION** tab is similar to the **Split Transets** tab, but
it splits the transects into 3.3 km lengths and includes all NPPSD
data fields. As with the Split Length tab**,** transect segments are
identified using the *master_key* a field based on *TripID, Transect
Number, Start Time, and segment number* (Figure 6). This csv file can
be downloaded using the button at the bottom of the page. Note: the
default 3.3 km length of NPPSD transects can be overridden on the
inputs page if the provided check box is selected.

![Tab tbl DATA_OBS formatted for the
NPPSD.](media/image6.jpg){width="6.539583333333334in"
height="5.988888888888889in"}

## Observations

Observations, like transect data, is presented in 3 tabs:
**Observations, Split Observations, and tbl_DATA_OBS**. The first
Observations tab is used to edit the observational data. In this tab,
potentially erroneous data is elevated to the top of the data listing
and includes an *invalid_columns* field that displays which fields in
that row of data contain potential issues. Note: users may have to
scroll over as the invalid columns field may not be visible depending
on window size. Double-clicking a cell with an issue and correcting it
will clear the *invalid_columns* field and the row will be sorted
according to trip, transect number and timestamp columns. The table
also contains a density column, which is re-calculated each time the
data is edited, and a pair of columns to identify which split transect
segment the observation falls in for both the NPS and NPPSD transect
splits (Figure 7). A backup copy of the raw data is saved in the
SeaLog\>backups directory with each change (up to the 5 most recent),
and a table showing the original data from the edited rows appears
below the current data.

![**Observations** tab where editing of individual observations should be
done.](media/image7.jpg){width="6.539583333333334in"
height="5.988888888888889in"}

## Split Observations

This tab presents views of the observational data separated by transect
segment as indicated by **Split Transects** (Figure 8). The *master_key*
is shared between **Split Transects** and **Split Observations**. As
with the transect data, edits to observations should be made in the
**Observations** tab, not in the in this tab. The observations are
grouped by **Species** and **PrimaryBehavior.** These data in csv format may
be downloaded using the button at the bottom of the page.

![The **Split Observations** tab with grouped by **Species** and
**PrimaryBehavior** for each
**master_key**.](media/image8.jpg){width="6.539583333333334in"
height="5.988888888888889in"}

## tbl_DATA_OBS

This tab presents views of the observational data separated by transect
segment based on the standard NPPSD 3.3 km distance (unless overridden
by the checkbox on the **Input Files** tab (Figure 9)). The *master_key*
is shared between **tbl_Location** and **tbl_DATA_OBS**. As with the
transect data, edits to observations should be made in the
**Observations** tab, not in the **tbl_DATA_OBS** tab. Observations are
grouped by **Common Name/nppsd_4\_letter code** and **modified_behavior.**
These data in csv format may be downloaded using the button at the
bottom of the page.

![Tab for **tbl_DATA_OBS** showing the observations organized by transect
segment and taxa.](media/image9.jpg){width="6.539583333333334in"
height="5.988888888888889in"}


## Species Summary

The Species Summary tab (Figure 10) shows a series of species summary tables,
one table for each trip and transect number. Each table is grouped by species
and presents total counts, observations, and density for each species. These
tables can be downloaded by using the button at the bottom of the page.

![**Species Summary** tab](media/image10.jpg){width="6.539583333333334in"
height="5.988888888888889in"}


## Tracks

The last tab is a map view of the data ingested by the app. It defaults to a
zoomed-out view of the transects (Figure 11). Transects are colored by
timestamp, and the number of species observations within dynamically generated
polygons is shown within the large circles on the map. The "+" and "-" buttons
allow users to adjust their map extent. If you hover over a circle, you can see
the polygon over which that number of observations is contained.

![Initial map view showing the full extent of the sample
area.](media/image11.jpg){width="6.539583333333334in"
height="5.988888888888889in"}

When you zoom in, these polygons are recalculated until you are zoomed in
sufficiently and individual observations (in blue) appear (Figure 12). Hovering
over a transect displays the track id and clicking it shows the track id and
start and end timestamp. Hovering over an observation displays the species code
and count made at that location. [Note: the transect information will be for
the transects as collected, not split transects.]{.ul}

![Map under the **Tracks** tab zoomed in to show individual
observations.](media/image12.jpg){width="6.539583333333334in"
height="5.988888888888889in"}

At the bottom of the map is a button to download a
[geopackage](https://www.geopackage.org/) containing the tracks ([both
original, and the two split versions]{.ul}), and the observations. The
geopackage is also saved in the Sealog\>geopackages directory. Geopackages are
similar to ESRI's proprietary File Geodatabase format, but can be created
without access to ArcMap/ArcGIS Pro. They can be read into ArcMap/ArcGIS Pro,
as well as other GIS programs.

# Troubleshooting

There are several conditions that can affect the operation of QA/QSea. If
difficulty is encountered during the initial ingestion, check to be sure that
the correct JSON files are being selected. Another issue with ingestion can be
related to the raw datafile having been edited outside of QA/QSea. Some changes
may make the app unable to read the file. In that case, test on the original
data file and compare with the edited file. Attempting to edit data outside of
editable tabs (**Input Files,** **Transect, Observations**) can also cause a
lockup. Closing the browser window will release the R command line and the app
can be relaunched immediately by typing `runApp(launch.browser = T)` in the
command window.

For bugs that are not resolved by the strategies above, take note of the
specific action or series of actions that triggers the behavior and contact
Chris Swingley at cswingley\@abrinc.com, 907-455-6777 x105.
