How the numbers are calculated

---
title: "SDP teacher retention — the calculation"
output: md_document
---

```{r setup, include = FALSE}
# error = FALSE so that knitting *fails* on an error rather than embedding it in
# the document. A calculation that quietly renders its own stack trace and
# carries on is worse than one that stops.
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", error = FALSE)
options(stringsAsFactors = FALSE)
```

This document is both the description of the retention calculation and the
program that performs it. Every number the site publishes about teacher
retention is produced by running this file; the prose between the chunks is
where the definitions are argued for, which is the part that usually goes
missing when a calculation is written as a script.

It is one of three implementations. The other two — the original TypeScript in
`src/`, and `calc/retention.py` — compute the same facts from the same files and
write them out in the same format, described in `calc/CONTRACT.md`. The three
output files are compared byte for byte. Where a definition below is stated
oddly, it is usually because it is stating exactly what the other two do.

```{r constants}
ROOT <- Sys.getenv("SDP_ROOT", ".")
DATA <- file.path(ROOT, "data", "extracted")

snapshots <- data.frame(
  year = 2015:2025,
  file = c(
    "employee_information_Oct2015.csv", "employee_information_Oct2016.csv",
    "employee_information_Oct2017.csv", "employee_information_Oct2018.csv",
    "employee_information_Oct2019.csv", "employee_information_Oct2020.csv",
    "employee_information_Oct2021.csv", "employee_information_Oct2022.csv",
    "Employee_Data_2023-2024/employee_information_October2023.csv",
    "employee_information_Oct2024.csv",
    "Oct 2025/employee_information_Oct2025.csv"
  )
)

years <- snapshots$year
transitions <- data.frame(from = head(years, -1), to = tail(years, -1))

# Peer levels, and the token each becomes in a contract key. Keys carry no
# spaces, so that every implementation sorts them the same way.
levels_tbl <- data.frame(
  level = c("ELEMENTARY", "MIDDLE SCHOOL", "HIGH SCHOOL"),
  token = c("ELEMENTARY", "MIDDLE_SCHOOL", "HIGH_SCHOOL")
)
scopes <- c("teacher", "teacherWithAlt", "all")
MIN_TEACHERS <- 15

k8_orgs <- c("1300", "1280", "1340", "1260", "1390", "1470",
             "2380", "2590", "2470", "2480", "2210")
schools <- rbind(
  data.frame(org = k8_orgs, level = "ELEMENTARY"),
  data.frame(org = c("2050", "2650", "2680", "6040"),
             level = c("MIDDLE SCHOOL", "HIGH SCHOOL", "HIGH SCHOOL", "HIGH SCHOOL"))
)
```

## Reading the extracts

The district publishes an "Employee Information" extract every quarter. Only the
October ones are used: October is the district's own census month, so each pair
of consecutive Octobers is one school year.

The files are **latin-1**, not UTF-8 — a handful of accented names contain bytes
that are not valid UTF-8 and would otherwise decode to replacement characters.
They are read as raw bytes and converted explicitly, rather than left to
whatever the session's locale happens to be.

```{r read}
read_extract <- function(path) {
  raw <- readBin(path, "raw", file.size(path))
  txt <- iconv(rawToChar(raw), from = "latin1", to = "UTF-8")
  read.csv(text = txt, colClasses = "character", check.names = FALSE, na.strings = NULL)
}

raw_snap <- lapply(seq_len(nrow(snapshots)), function(i) {
  path <- file.path(DATA, snapshots$file[i])
  if (!file.exists(path)) stop("missing snapshot: ", path, " - run ./fetch-data.sh")
  read_extract(path)
})
names(raw_snap) <- as.character(years)

sapply(raw_snap, nrow)
```

## Who counts as a teacher

A classroom teacher needs **both** a PFT teaching local and a TEACHER title.
Requiring the union membership as well as the title is what excludes the
`NON REP` rows — student teachers on placement, and `CONTRACTED TEACHER`
positions bought in from outside — who are not district hires and turn over by
design. It also excludes the PFT-represented counsellors, nurses and therapists,
who are not classroom staff.

The title test is "TEACHER followed by a word boundary", written out here as an
explicit character test rather than as `\b`, because the three languages
disagree about whether `\b` is Unicode-aware and this is not a place to find
that out later.

```{r definitions}
is_teacher <- function(rep, title) {
  t <- toupper(title)
  startsWith(toupper(rep), "PFT") &
    startsWith(t, "TEACHER") &
    !grepl("^[A-Za-z0-9_]", substr(t, 8, 8))
}

# Instructional staff who are not district hires.
is_alt <- function(title) {
  t <- toupper(title)
  grepl("STUDENT TEACHER", t, fixed = TRUE) | grepl("CONTRACTED TEACHER", t, fixed = TRUE)
}

# The building principal, not assistant principals or retired fill-ins.
is_principal <- function(title) trimws(toupper(title)) == "PRINCIPAL"
```

## Identity across years

The extracts carry no employee number, so a person is matched between two
Octobers by name alone. Case, punctuation and internal spacing are normalized
away, which stops a cosmetic spelling change from reading as a departure.

Two limits survive this and are stated on the site: somebody who changes their
surname reads as one departure and one arrival, and two people who share a name
read as one person.

```{r identity}
norm_name <- function(s) trimws(gsub("\\s+", " ", gsub("[^A-Z ]", "", toupper(s))))
emp_key <- function(last, first) paste0(norm_name(last), "|", norm_name(first))

staff <- lapply(names(raw_snap), function(y) {
  df <- raw_snap[[y]]
  data.frame(
    key       = emp_key(df$LAST_NAME, df$FIRST_NAME),
    org       = df$HOME_ORGANIZATION,
    level     = toupper(df$ORGANIZATION_LEVEL),
    teacher   = is_teacher(df$TYPE_OF_REPRESENTATION, df$TITLE_DESCRIPTION),
    alt       = is_alt(df$TITLE_DESCRIPTION),
    principal = is_principal(df$TITLE_DESCRIPTION),
    name      = paste(df$FIRST_NAME, df$LAST_NAME)
  )
})
names(staff) <- names(raw_snap)

data.frame(year = years,
           rows = sapply(staff, nrow),
           teachers = sapply(staff, function(e) sum(e$teacher)),
           principals = sapply(staff, function(e) sum(e$principal)))
```

## Indexing

Every school is compared against roughly 150 peers across ten transitions and
three staff definitions. Grouping the rows by school once, up front, is what
keeps that from being quadratic.

`keys_by[[year]][[scope]]` holds the identity keys of each school's staff under
one of the three definitions; the row counts are kept separately, because a
headcount counts rows and a retention rate counts distinct people.

```{r index}
scope_mask <- function(e, scope) {
  if (scope == "teacher") e$teacher
  else if (scope == "teacherWithAlt") e$teacher | e$alt
  else rep(TRUE, nrow(e))
}

keys_by <- lapply(staff, function(e) {
  out <- lapply(scopes, function(s) {
    m <- scope_mask(e, s)
    split(e$key[m], e$org[m])
  })
  names(out) <- scopes
  out
})

# Rows, not distinct people: this is the headcount used for the alternative-staff
# share and for the peer-set threshold.
rows_by <- lapply(keys_by, function(y) lapply(y, lengths))

at <- function(bucket, org) {
  v <- bucket[[org]]
  if (is.null(v)) character(0) else v
}
count_at <- function(bucket, org) {
  # `rows_by` holds named integer vectors, where indexing a name that is not
  # there is an error rather than a NULL; a school with no staff of some kind in
  # some year is an absent name, not a zero.
  v <- bucket[org]
  if (length(v) == 0 || is.na(v)) 0L else as.integer(v)
}
```

## Retention between two Octobers

Retention is the share of one October's people who are still there the next
October, counting distinct people rather than rows. It is deliberately *not*
`1 - leavers/headcount`: those differ whenever the headcount changes, and the
question is about the starting cohort.

```{r retention}
retention <- function(before, after) {
  start <- length(unique(before))
  stayed <- if (start == 0) 0L else length(intersect(before, after))
  list(stayed = as.integer(stayed), start = as.integer(start),
       rate = if (start == 0) NULL else stayed / start)
}
```

## Peer sets

A retention figure means little on its own — 2021 was a bad year everywhere. Each
school is therefore also ranked against schools of its own level with at least
`r MIN_TEACHERS` teachers that October.

A school's level is read from the **first** row that school contributes to the
file, matching the original implementation: a few organizations carry rows
labelled with more than one level, and taking the first row is a rule that all
three implementations can follow without ambiguity.

```{r peers}
peer_cache <- new.env(parent = emptyenv())

peer_orgs <- function(year, level) {
  ck <- paste(year, level)
  if (!is.null(peer_cache[[ck]])) return(peer_cache[[ck]])
  e <- staff[[as.character(year)]]
  first <- !duplicated(e$org)
  org_level <- e$level[first]
  orgs <- e$org[first]
  tcount <- lengths(split(e$key[e$teacher], factor(e$org[e$teacher], levels = orgs)))
  out <- orgs[grepl(level, org_level, fixed = TRUE) & tcount >= MIN_TEACHERS]
  peer_cache[[ck]] <- out
  out
}

rate_cache <- new.env(parent = emptyenv())

peer_rates <- function(from, to, level, scope) {
  ck <- paste(from, to, level, scope)
  if (!is.null(rate_cache[[ck]])) return(rate_cache[[ck]])
  b <- keys_by[[as.character(from)]][[scope]]
  a <- keys_by[[as.character(to)]][[scope]]
  orgs <- peer_orgs(from, level)
  out <- numeric(length(orgs))
  n <- 0L
  for (o in orgs) {
    r <- retention(at(b, o), at(a, o))
    if (!is.null(r$rate)) { n <- n + 1L; out[n] <- r$rate }
  }
  out <- out[seq_len(n)]
  rate_cache[[ck]] <- out
  out
}
```

The two summary statistics over a peer set are a median and a percentile rank.
Both are defined the plain way — the middle value, and the share of peers at or
below this school — with no interpolation and no tie correction, because the
point is to be reproducible rather than to be clever. The quartiles index
straight into the sorted rates for the same reason.

```{r stats}
med <- function(v) {
  if (!length(v)) return(NULL)
  s <- sort(v)
  mid <- length(s) %/% 2
  if (length(s) %% 2 == 1) s[mid + 1] else (s[mid] + s[mid + 1]) / 2
}

pct_rank <- function(v, target) if (!length(v)) NULL else sum(v <= target) / length(v)

quantile_at <- function(sorted, q) {
  if (!length(sorted)) return(NULL)
  sorted[floor(q * (length(sorted) - 1)) + 1]
}
```

## Where the leavers went

A school losing staff to other district schools has a different problem from one
whose teachers leave the district altogether, so each year's cohort is followed
into the next October and sorted into four outcomes.

Where the same name appears twice as a teacher in one year, the last row in the
file wins — not because that is right, but because it is what a dictionary
assignment does in the other two implementations, and the three have to agree.

```{r person-index}
teacher_index <- lapply(staff, function(e) {
  m <- e$teacher
  keep <- !duplicated(e$key[m], fromLast = TRUE)
  list(keys = e$key[m][keep], orgs = e$org[m][keep], payroll = unique(e$key))
})
```

## Principal tenures

The extracts list a principal per school per October, so a tenure is a run of
consecutive Octobers with the same name. Where a school lists two people at once
the names are joined, sorted, so that the order they appear in the file does not
create a spurious change of leadership. An October with nobody listed is its own
kind of run, not a gap to be smoothed over.

Retention is then pooled within each run. A transition belongs to whoever held
the building in its **starting** October — that is the year whose staff either
stayed or left.

```{r timeline}
compress <- function(names_by_year) {
  runs <- list()
  for (i in seq_along(names_by_year)) {
    nm <- names_by_year[i]
    yr <- years[i]
    if (length(runs) && runs[[length(runs)]]$names == nm) {
      runs[[length(runs)]]$to <- yr
    } else {
      runs[[length(runs) + 1]] <- list(names = nm, from = yr, to = yr)
    }
  }
  runs
}

principal_names <- function(year, org) {
  e <- staff[[as.character(year)]]
  nm <- e$name[e$principal & e$org == org]
  if (!length(nm)) return("(none listed)")
  # C-locale ordering, so that R, Python and JavaScript sort the same names the
  # same way regardless of the locale each happens to be running under.
  paste(nm[order(nm, method = "radix")], collapse = " + ")
}
```

## Writing the contract

The output format is a sorted TSV of `metric`, `key`, `value`. Rates are written
as their numerator and denominator, as integers, rather than as a quotient —
integers cannot drift — and the reals that remain get a fixed twelve decimal
places, which is finer than any real disagreement and coarse enough to absorb a
last-bit difference between three languages' arithmetic.

```{r contract}
NA_VALUE <- "NA"

fmt_int <- function(n) {
  n <- as.numeric(n)
  if (!is.finite(n) || n != round(n)) stop("contract: ", n, " is not an integer")
  sprintf("%.0f", n)
}

fmt_real <- function(x) {
  if (is.null(x) || (length(x) == 1 && is.na(x))) return(NA_VALUE)
  if (!is.finite(x)) stop("contract: ", x, " is not finite")
  # -0 and 0 are the same number; only their printed forms differ.
  sprintf("%.12f", if (x == 0) 0 else x)
}

fmt_text <- function(s) {
  if (grepl("[\t\r\n]", s)) stop("contract: value contains a tab or newline: ", s)
  s
}

rows <- new.env(parent = emptyenv())
rows$metric <- character(0); rows$key <- character(0); rows$value <- character(0)
add <- function(metric, key, value) {
  rows$metric <- c(rows$metric, metric)
  rows$key <- c(rows$key, key)
  rows$value <- c(rows$value, value)
}

sha256 <- function(s) digest::digest(s, algo = "sha256", serialize = FALSE)
```

## The facts

Inputs first, so that a disagreement about the numbers can be told apart from a
disagreement about what was read off the disk. The digests are the part that
does the work: two implementations that hash their parsed extracts to the same
value have demonstrably read the same file the same way, which is otherwise the
hardest thing to be sure of across three CSV readers.

```{r facts-input}
for (y in years) {
  e <- staff[[as.character(y)]]
  df <- raw_snap[[as.character(y)]]
  dump <- paste(df$LAST_NAME, df$FIRST_NAME, df$TITLE_DESCRIPTION, df$HOME_ORGANIZATION,
                df$ORGANIZATION_LEVEL, df$TYPE_OF_REPRESENTATION, sep = "\x1f")
  uk <- unique(e$key)
  add("input.rows", as.character(y), fmt_int(nrow(e)))
  add("input.digest", as.character(y), fmt_text(sha256(paste(dump, collapse = "\n"))))
  add("input.keydigest", as.character(y),
      fmt_text(sha256(paste(uk[order(uk, method = "radix")], collapse = "\n"))))
  add("input.keys", as.character(y), fmt_int(length(uk)))
  add("input.teachers", as.character(y), fmt_int(sum(e$teacher)))
  add("input.principals", as.character(y), fmt_int(sum(e$principal)))
}
```

```{r facts-peers}
for (li in seq_len(nrow(levels_tbl))) {
  level <- levels_tbl$level[li]; tok <- levels_tbl$token[li]
  for (y in years) {
    add("peer.orgs", paste0(y, "|", tok), fmt_int(length(peer_orgs(y, level))))
  }
  for (ti in seq_len(nrow(transitions))) {
    from <- transitions$from[ti]; to <- transitions$to[ti]
    for (scope in scopes) {
      r <- sort(peer_rates(from, to, level, scope))
      key <- paste0(from, "-", to, "|", tok, "|", scope)
      add("peer.n", key, fmt_int(length(r)))
      add("peer.median", key, fmt_real(med(r)))
      add("peer.q1", key, fmt_real(quantile_at(r, 0.25)))
      add("peer.q3", key, fmt_real(quantile_at(r, 0.75)))
    }
  }
}
```

```{r facts-schools}
pooled_teacher <- list(); changes <- list(); tenure <- list()

for (si in seq_len(nrow(schools))) {
  org <- schools$org[si]; level <- schools$level[si]
  tok <- levels_tbl$token[match(level, levels_tbl$level)]
  add("school.level", org, fmt_text(tok))

  series <- list()
  for (scope in scopes) {
    st <- integer(0); sc <- integer(0)
    for (ti in seq_len(nrow(transitions))) {
      from <- transitions$from[ti]; to <- transitions$to[ti]
      r <- retention(at(keys_by[[as.character(from)]][[scope]], org),
                     at(keys_by[[as.character(to)]][[scope]], org))
      p <- if (is.null(r$rate)) NULL else pct_rank(peer_rates(from, to, level, scope), r$rate)
      key <- paste0(org, "|", from, "-", to)
      add(paste0("series.", scope, ".stayed"), key, fmt_int(r$stayed))
      add(paste0("series.", scope, ".start"), key, fmt_int(r$start))
      add(paste0("series.", scope, ".percentile"), key, fmt_real(p))
      st <- c(st, r$stayed); sc <- c(sc, r$start)
    }
    series[[scope]] <- list(stayed = st, start = sc)
  }

  keep2020 <- transitions$from >= 2020
  pools <- list(
    teacher = series$teacher,
    teacherWithAlt = series$teacherWithAlt,
    all = series$all,
    teacherSince2020 = list(stayed = series$teacher$stayed[keep2020],
                            start = series$teacher$start[keep2020])
  )
  for (nm in names(pools)) {
    s <- sum(pools[[nm]]$stayed); n <- sum(pools[[nm]]$start)
    add(paste0("pooled.", nm, ".stayed"), org, fmt_int(s))
    add(paste0("pooled.", nm, ".start"), org, fmt_int(n))
    if (nm == "teacher") pooled_teacher[[org]] <- if (n == 0) NULL else s / n
  }

  for (y in years) {
    pft <- count_at(rows_by[[as.character(y)]]$teacher, org)
    alt <- count_at(rows_by[[as.character(y)]]$teacherWithAlt, org) - pft
    add("alt.pft", paste0(org, "|", y), fmt_int(pft))
    add("alt.alt", paste0(org, "|", y), fmt_int(alt))
  }

  runs <- compress(sapply(years, principal_names, org = org))
  add("principal.runs", org, fmt_int(length(runs)))
  # Runs minus one: three runs over the window means two changes.
  add("principal.changes", org, fmt_int(max(0, length(runs) - 1)))
  for (i in seq_along(runs)) {
    add("principal.run", sprintf("%s|%02d", org, i - 1),
        fmt_text(sprintf("%d-%d|%s", runs[[i]]$from, runs[[i]]$to, runs[[i]]$names)))
  }
  changes[[org]] <- max(0, length(runs) - 1)
  tenure[[org]] <- max(sapply(runs, function(r) r$to - r$from + 1))

  since <- years[years >= 2020]
  base <- unique(at(keys_by[[as.character(since[1])]]$teacher, org))
  add("cohort2020.base", org, fmt_int(length(base)))
  for (y in since) {
    now <- at(keys_by[[as.character(y)]]$teacher, org)
    add("cohort2020.remaining", paste0(org, "|", y), fmt_int(length(intersect(base, now))))
  }

  add("teacherCount", org, fmt_int(count_at(rows_by[[as.character(tail(years, 1))]]$teacher, org)))

  dest <- c(stayed = 0L, movedSchool = 0L, changedRole = 0L, leftDistrict = 0L)
  for (ti in seq_len(nrow(transitions))) {
    from <- transitions$from[ti]; to <- transitions$to[ti]
    ks <- unique(at(keys_by[[as.character(from)]]$teacher, org))
    idx <- teacher_index[[as.character(to)]]
    where <- idx$orgs[match(ks, idx$keys)]
    on_payroll <- ks %in% idx$payroll
    dest["stayed"] <- dest["stayed"] + sum(!is.na(where) & where == org)
    dest["movedSchool"] <- dest["movedSchool"] + sum(!is.na(where) & where != org)
    dest["changedRole"] <- dest["changedRole"] + sum(is.na(where) & on_payroll)
    dest["leftDistrict"] <- dest["leftDistrict"] + sum(is.na(where) & !on_payroll)
  }
  for (nm in names(dest)) add(paste0("destination.", nm), org, fmt_int(dest[[nm]]))

  eras <- list()
  for (r in runs) {
    m <- transitions$from >= r$from & transitions$from <= r$to
    s <- sum(series$teacher$stayed[m]); n <- sum(series$teacher$start[m])
    if (n > 0) eras[[length(eras) + 1]] <- list(run = r, stayed = s, start = n)
  }
  add("era.n", org, fmt_int(length(eras)))
  for (i in seq_along(eras)) {
    k <- sprintf("%s|%02d", org, i - 1)
    add("era.span", k, fmt_text(sprintf("%d-%d|%s", eras[[i]]$run$from, eras[[i]]$run$to, eras[[i]]$run$names)))
    add("era.stayed", k, fmt_int(eras[[i]]$stayed))
    add("era.start", k, fmt_int(eras[[i]]$start))
  }
}
```

## Does principal churn predict teacher retention?

This is the claim the whole project started from: that a principal needs about
five years to move a school, and that stability at the top is what drives
everything else. It is testable directly — correlate how often a school changed
principal against how well it held on to its teachers.

Pearson's r is written as an explicit loop rather than with `cor()`, so that all
three implementations add their terms up in the same order and land on the same
bits. Both input series are published alongside the coefficient, so that a
disagreement can be traced to the inputs or to the arithmetic.

```{r correlation}
pearson <- function(xs, ys) {
  n <- min(length(xs), length(ys))
  if (n < 2) return(NULL)
  sx <- 0; sy <- 0
  for (i in seq_len(n)) { sx <- sx + xs[i]; sy <- sy + ys[i] }
  mx <- sx / n; my <- sy / n
  num <- 0; dx <- 0; dy <- 0
  for (i in seq_len(n)) {
    a <- xs[i] - mx; b <- ys[i] - my
    num <- num + a * b; dx <- dx + a * a; dy <- dy + b * b
  }
  if (dx == 0 || dy == 0) return(NULL)
  num / sqrt(dx * dy)
}

rankable <- k8_orgs[!sapply(k8_orgs, function(o) is.null(pooled_teacher[[o]]))]
for (o in rankable) {
  add("correlation.churn", o, fmt_int(changes[[o]]))
  add("correlation.tenure", o, fmt_int(tenure[[o]]))
}
churn_r <- pearson(sapply(rankable, function(o) changes[[o]]),
                   sapply(rankable, function(o) pooled_teacher[[o]]))
tenure_r <- pearson(sapply(rankable, function(o) tenure[[o]]),
                    sapply(rankable, function(o) pooled_teacher[[o]]))
add("correlation.n", "-", fmt_int(length(rankable)))
add("correlation.churnVsRetention", "-", fmt_real(churn_r))
add("correlation.tenureVsRetention", "-", fmt_real(tenure_r))

data.frame(
  measure = c("principal changes vs pooled teacher retention",
              "longest tenure vs pooled teacher retention"),
  r = c(round(churn_r, 3), round(tenure_r, 3)),
  n = length(rankable)
)
```

The claim predicts a clearly negative first row.

## Pooled retention, by school

```{r summary}
summary_tbl <- data.frame(
  org = k8_orgs,
  pooled_15_25 = round(sapply(k8_orgs, function(o) pooled_teacher[[o]]), 4),
  principals = sapply(k8_orgs, function(o) changes[[o]] + 1),
  longest_run = sapply(k8_orgs, function(o) tenure[[o]])
)
summary_tbl[order(-summary_tbl$pooled_15_25), ]
```

## Output

Sorted by metric then key, in byte order. Uppercase sorts before lowercase, and
`radix` is what pins that down regardless of the session's locale.

```{r write}
stopifnot(!any(duplicated(paste0(rows$metric, "\t", rows$key))))
ord <- order(paste0(rows$metric, "\t", rows$key), method = "radix")
lines <- c("metric\tkey\tvalue",
           paste(rows$metric[ord], rows$key[ord], rows$value[ord], sep = "\t"))

out_dir <- file.path(ROOT, "calc", "out")
dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
con <- file(file.path(out_dir, "r.tsv"), open = "wb")
writeLines(lines, con, sep = "\n", useBytes = TRUE)
close(con)

cat(sprintf("facts: %d\nsha256: %s\n", length(lines) - 1,
            sha256(paste0(paste(lines, collapse = "\n"), "\n"))))
```

The TypeScript and Python implementations print the same hash.

---
title: "SDP teacher retention — the calculation"
output: md_document
---



This document is both the description of the retention calculation and the
program that performs it. Every number the site publishes about teacher
retention is produced by running this file; the prose between the chunks is
where the definitions are argued for, which is the part that usually goes
missing when a calculation is written as a script.

It is one of three implementations. The other two — the original TypeScript in
`src/`, and `calc/retention.py` — compute the same facts from the same files and
write them out in the same format, described in `calc/CONTRACT.md`. The three
output files are compared byte for byte. Where a definition below is stated
oddly, it is usually because it is stating exactly what the other two do.


```r
ROOT <- Sys.getenv("SDP_ROOT", ".")
DATA <- file.path(ROOT, "data", "extracted")

snapshots <- data.frame(
  year = 2015:2025,
  file = c(
    "employee_information_Oct2015.csv", "employee_information_Oct2016.csv",
    "employee_information_Oct2017.csv", "employee_information_Oct2018.csv",
    "employee_information_Oct2019.csv", "employee_information_Oct2020.csv",
    "employee_information_Oct2021.csv", "employee_information_Oct2022.csv",
    "Employee_Data_2023-2024/employee_information_October2023.csv",
    "employee_information_Oct2024.csv",
    "Oct 2025/employee_information_Oct2025.csv"
  )
)

years <- snapshots$year
transitions <- data.frame(from = head(years, -1), to = tail(years, -1))

# Peer levels, and the token each becomes in a contract key. Keys carry no
# spaces, so that every implementation sorts them the same way.
levels_tbl <- data.frame(
  level = c("ELEMENTARY", "MIDDLE SCHOOL", "HIGH SCHOOL"),
  token = c("ELEMENTARY", "MIDDLE_SCHOOL", "HIGH_SCHOOL")
)
scopes <- c("teacher", "teacherWithAlt", "all")
MIN_TEACHERS <- 15

k8_orgs <- c("1300", "1280", "1340", "1260", "1390", "1470",
             "2380", "2590", "2470", "2480", "2210")
schools <- rbind(
  data.frame(org = k8_orgs, level = "ELEMENTARY"),
  data.frame(org = c("2050", "2650", "2680", "6040"),
             level = c("MIDDLE SCHOOL", "HIGH SCHOOL", "HIGH SCHOOL", "HIGH SCHOOL"))
)
```

## Reading the extracts

The district publishes an "Employee Information" extract every quarter. Only the
October ones are used: October is the district's own census month, so each pair
of consecutive Octobers is one school year.

The files are **latin-1**, not UTF-8 — a handful of accented names contain bytes
that are not valid UTF-8 and would otherwise decode to replacement characters.
They are read as raw bytes and converted explicitly, rather than left to
whatever the session's locale happens to be.


```r
read_extract <- function(path) {
  raw <- readBin(path, "raw", file.size(path))
  txt <- iconv(rawToChar(raw), from = "latin1", to = "UTF-8")
  read.csv(text = txt, colClasses = "character", check.names = FALSE, na.strings = NULL)
}

raw_snap <- lapply(seq_len(nrow(snapshots)), function(i) {
  path <- file.path(DATA, snapshots$file[i])
  if (!file.exists(path)) stop("missing snapshot: ", path, " - run ./fetch-data.sh")
  read_extract(path)
})
names(raw_snap) <- as.character(years)

sapply(raw_snap, nrow)
#>  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  2025 
#> 17476 17953 18715 19540 20488 21196 21142 21636 23580 24776 25633
```

## Who counts as a teacher

A classroom teacher needs **both** a PFT teaching local and a TEACHER title.
Requiring the union membership as well as the title is what excludes the
`NON REP` rows — student teachers on placement, and `CONTRACTED TEACHER`
positions bought in from outside — who are not district hires and turn over by
design. It also excludes the PFT-represented counsellors, nurses and therapists,
who are not classroom staff.

The title test is "TEACHER followed by a word boundary", written out here as an
explicit character test rather than as `\b`, because the three languages
disagree about whether `\b` is Unicode-aware and this is not a place to find
that out later.


```r
is_teacher <- function(rep, title) {
  t <- toupper(title)
  startsWith(toupper(rep), "PFT") &
    startsWith(t, "TEACHER") &
    !grepl("^[A-Za-z0-9_]", substr(t, 8, 8))
}

# Instructional staff who are not district hires.
is_alt <- function(title) {
  t <- toupper(title)
  grepl("STUDENT TEACHER", t, fixed = TRUE) | grepl("CONTRACTED TEACHER", t, fixed = TRUE)
}

# The building principal, not assistant principals or retired fill-ins.
is_principal <- function(title) trimws(toupper(title)) == "PRINCIPAL"
```

## Identity across years

The extracts carry no employee number, so a person is matched between two
Octobers by name alone. Case, punctuation and internal spacing are normalized
away, which stops a cosmetic spelling change from reading as a departure.

Two limits survive this and are stated on the site: somebody who changes their
surname reads as one departure and one arrival, and two people who share a name
read as one person.


```r
norm_name <- function(s) trimws(gsub("\\s+", " ", gsub("[^A-Z ]", "", toupper(s))))
emp_key <- function(last, first) paste0(norm_name(last), "|", norm_name(first))

staff <- lapply(names(raw_snap), function(y) {
  df <- raw_snap[[y]]
  data.frame(
    key       = emp_key(df$LAST_NAME, df$FIRST_NAME),
    org       = df$HOME_ORGANIZATION,
    level     = toupper(df$ORGANIZATION_LEVEL),
    teacher   = is_teacher(df$TYPE_OF_REPRESENTATION, df$TITLE_DESCRIPTION),
    alt       = is_alt(df$TITLE_DESCRIPTION),
    principal = is_principal(df$TITLE_DESCRIPTION),
    name      = paste(df$FIRST_NAME, df$LAST_NAME)
  )
})
names(staff) <- names(raw_snap)

data.frame(year = years,
           rows = sapply(staff, nrow),
           teachers = sapply(staff, function(e) sum(e$teacher)),
           principals = sapply(staff, function(e) sum(e$principal)))
#>      year  rows teachers principals
#> 2015 2015 17476     8738        224
#> 2016 2016 17953     8734        227
#> 2017 2017 18715     8597        231
#> 2018 2018 19540     8728        229
#> 2019 2019 20488     8846        229
#> 2020 2020 21196     8856        232
#> 2021 2021 21142     8934        234
#> 2022 2022 21636     8839        238
#> 2023 2023 23580     9020        238
#> 2024 2024 24776     9095        233
#> 2025 2025 25633     9298        226
```

## Indexing

Every school is compared against roughly 150 peers across ten transitions and
three staff definitions. Grouping the rows by school once, up front, is what
keeps that from being quadratic.

`keys_by[[year]][[scope]]` holds the identity keys of each school's staff under
one of the three definitions; the row counts are kept separately, because a
headcount counts rows and a retention rate counts distinct people.


```r
scope_mask <- function(e, scope) {
  if (scope == "teacher") e$teacher
  else if (scope == "teacherWithAlt") e$teacher | e$alt
  else rep(TRUE, nrow(e))
}

keys_by <- lapply(staff, function(e) {
  out <- lapply(scopes, function(s) {
    m <- scope_mask(e, s)
    split(e$key[m], e$org[m])
  })
  names(out) <- scopes
  out
})

# Rows, not distinct people: this is the headcount used for the alternative-staff
# share and for the peer-set threshold.
rows_by <- lapply(keys_by, function(y) lapply(y, lengths))

at <- function(bucket, org) {
  v <- bucket[[org]]
  if (is.null(v)) character(0) else v
}
count_at <- function(bucket, org) {
  # `rows_by` holds named integer vectors, where indexing a name that is not
  # there is an error rather than a NULL; a school with no staff of some kind in
  # some year is an absent name, not a zero.
  v <- bucket[org]
  if (length(v) == 0 || is.na(v)) 0L else as.integer(v)
}
```

## Retention between two Octobers

Retention is the share of one October's people who are still there the next
October, counting distinct people rather than rows. It is deliberately *not*
`1 - leavers/headcount`: those differ whenever the headcount changes, and the
question is about the starting cohort.


```r
retention <- function(before, after) {
  start <- length(unique(before))
  stayed <- if (start == 0) 0L else length(intersect(before, after))
  list(stayed = as.integer(stayed), start = as.integer(start),
       rate = if (start == 0) NULL else stayed / start)
}
```

## Peer sets

A retention figure means little on its own — 2021 was a bad year everywhere. Each
school is therefore also ranked against schools of its own level with at least
15 teachers that October.

A school's level is read from the **first** row that school contributes to the
file, matching the original implementation: a few organizations carry rows
labelled with more than one level, and taking the first row is a rule that all
three implementations can follow without ambiguity.


```r
peer_cache <- new.env(parent = emptyenv())

peer_orgs <- function(year, level) {
  ck <- paste(year, level)
  if (!is.null(peer_cache[[ck]])) return(peer_cache[[ck]])
  e <- staff[[as.character(year)]]
  first <- !duplicated(e$org)
  org_level <- e$level[first]
  orgs <- e$org[first]
  tcount <- lengths(split(e$key[e$teacher], factor(e$org[e$teacher], levels = orgs)))
  out <- orgs[grepl(level, org_level, fixed = TRUE) & tcount >= MIN_TEACHERS]
  peer_cache[[ck]] <- out
  out
}

rate_cache <- new.env(parent = emptyenv())

peer_rates <- function(from, to, level, scope) {
  ck <- paste(from, to, level, scope)
  if (!is.null(rate_cache[[ck]])) return(rate_cache[[ck]])
  b <- keys_by[[as.character(from)]][[scope]]
  a <- keys_by[[as.character(to)]][[scope]]
  orgs <- peer_orgs(from, level)
  out <- numeric(length(orgs))
  n <- 0L
  for (o in orgs) {
    r <- retention(at(b, o), at(a, o))
    if (!is.null(r$rate)) { n <- n + 1L; out[n] <- r$rate }
  }
  out <- out[seq_len(n)]
  rate_cache[[ck]] <- out
  out
}
```

The two summary statistics over a peer set are a median and a percentile rank.
Both are defined the plain way — the middle value, and the share of peers at or
below this school — with no interpolation and no tie correction, because the
point is to be reproducible rather than to be clever. The quartiles index
straight into the sorted rates for the same reason.


```r
med <- function(v) {
  if (!length(v)) return(NULL)
  s <- sort(v)
  mid <- length(s) %/% 2
  if (length(s) %% 2 == 1) s[mid + 1] else (s[mid] + s[mid + 1]) / 2
}

pct_rank <- function(v, target) if (!length(v)) NULL else sum(v <= target) / length(v)

quantile_at <- function(sorted, q) {
  if (!length(sorted)) return(NULL)
  sorted[floor(q * (length(sorted) - 1)) + 1]
}
```

## Where the leavers went

A school losing staff to other district schools has a different problem from one
whose teachers leave the district altogether, so each year's cohort is followed
into the next October and sorted into four outcomes.

Where the same name appears twice as a teacher in one year, the last row in the
file wins — not because that is right, but because it is what a dictionary
assignment does in the other two implementations, and the three have to agree.


```r
teacher_index <- lapply(staff, function(e) {
  m <- e$teacher
  keep <- !duplicated(e$key[m], fromLast = TRUE)
  list(keys = e$key[m][keep], orgs = e$org[m][keep], payroll = unique(e$key))
})
```

## Principal tenures

The extracts list a principal per school per October, so a tenure is a run of
consecutive Octobers with the same name. Where a school lists two people at once
the names are joined, sorted, so that the order they appear in the file does not
create a spurious change of leadership. An October with nobody listed is its own
kind of run, not a gap to be smoothed over.

Retention is then pooled within each run. A transition belongs to whoever held
the building in its **starting** October — that is the year whose staff either
stayed or left.


```r
compress <- function(names_by_year) {
  runs <- list()
  for (i in seq_along(names_by_year)) {
    nm <- names_by_year[i]
    yr <- years[i]
    if (length(runs) && runs[[length(runs)]]$names == nm) {
      runs[[length(runs)]]$to <- yr
    } else {
      runs[[length(runs) + 1]] <- list(names = nm, from = yr, to = yr)
    }
  }
  runs
}

principal_names <- function(year, org) {
  e <- staff[[as.character(year)]]
  nm <- e$name[e$principal & e$org == org]
  if (!length(nm)) return("(none listed)")
  # C-locale ordering, so that R, Python and JavaScript sort the same names the
  # same way regardless of the locale each happens to be running under.
  paste(nm[order(nm, method = "radix")], collapse = " + ")
}
```

## Writing the contract

The output format is a sorted TSV of `metric`, `key`, `value`. Rates are written
as their numerator and denominator, as integers, rather than as a quotient —
integers cannot drift — and the reals that remain get a fixed twelve decimal
places, which is finer than any real disagreement and coarse enough to absorb a
last-bit difference between three languages' arithmetic.


```r
NA_VALUE <- "NA"

fmt_int <- function(n) {
  n <- as.numeric(n)
  if (!is.finite(n) || n != round(n)) stop("contract: ", n, " is not an integer")
  sprintf("%.0f", n)
}

fmt_real <- function(x) {
  if (is.null(x) || (length(x) == 1 && is.na(x))) return(NA_VALUE)
  if (!is.finite(x)) stop("contract: ", x, " is not finite")
  # -0 and 0 are the same number; only their printed forms differ.
  sprintf("%.12f", if (x == 0) 0 else x)
}

fmt_text <- function(s) {
  if (grepl("[\t\r\n]", s)) stop("contract: value contains a tab or newline: ", s)
  s
}

rows <- new.env(parent = emptyenv())
rows$metric <- character(0); rows$key <- character(0); rows$value <- character(0)
add <- function(metric, key, value) {
  rows$metric <- c(rows$metric, metric)
  rows$key <- c(rows$key, key)
  rows$value <- c(rows$value, value)
}

sha256 <- function(s) digest::digest(s, algo = "sha256", serialize = FALSE)
```

## The facts

Inputs first, so that a disagreement about the numbers can be told apart from a
disagreement about what was read off the disk. The digests are the part that
does the work: two implementations that hash their parsed extracts to the same
value have demonstrably read the same file the same way, which is otherwise the
hardest thing to be sure of across three CSV readers.


```r
for (y in years) {
  e <- staff[[as.character(y)]]
  df <- raw_snap[[as.character(y)]]
  dump <- paste(df$LAST_NAME, df$FIRST_NAME, df$TITLE_DESCRIPTION, df$HOME_ORGANIZATION,
                df$ORGANIZATION_LEVEL, df$TYPE_OF_REPRESENTATION, sep = "\x1f")
  uk <- unique(e$key)
  add("input.rows", as.character(y), fmt_int(nrow(e)))
  add("input.digest", as.character(y), fmt_text(sha256(paste(dump, collapse = "\n"))))
  add("input.keydigest", as.character(y),
      fmt_text(sha256(paste(uk[order(uk, method = "radix")], collapse = "\n"))))
  add("input.keys", as.character(y), fmt_int(length(uk)))
  add("input.teachers", as.character(y), fmt_int(sum(e$teacher)))
  add("input.principals", as.character(y), fmt_int(sum(e$principal)))
}
```


```r
for (li in seq_len(nrow(levels_tbl))) {
  level <- levels_tbl$level[li]; tok <- levels_tbl$token[li]
  for (y in years) {
    add("peer.orgs", paste0(y, "|", tok), fmt_int(length(peer_orgs(y, level))))
  }
  for (ti in seq_len(nrow(transitions))) {
    from <- transitions$from[ti]; to <- transitions$to[ti]
    for (scope in scopes) {
      r <- sort(peer_rates(from, to, level, scope))
      key <- paste0(from, "-", to, "|", tok, "|", scope)
      add("peer.n", key, fmt_int(length(r)))
      add("peer.median", key, fmt_real(med(r)))
      add("peer.q1", key, fmt_real(quantile_at(r, 0.25)))
      add("peer.q3", key, fmt_real(quantile_at(r, 0.75)))
    }
  }
}
```


```r
pooled_teacher <- list(); changes <- list(); tenure <- list()

for (si in seq_len(nrow(schools))) {
  org <- schools$org[si]; level <- schools$level[si]
  tok <- levels_tbl$token[match(level, levels_tbl$level)]
  add("school.level", org, fmt_text(tok))

  series <- list()
  for (scope in scopes) {
    st <- integer(0); sc <- integer(0)
    for (ti in seq_len(nrow(transitions))) {
      from <- transitions$from[ti]; to <- transitions$to[ti]
      r <- retention(at(keys_by[[as.character(from)]][[scope]], org),
                     at(keys_by[[as.character(to)]][[scope]], org))
      p <- if (is.null(r$rate)) NULL else pct_rank(peer_rates(from, to, level, scope), r$rate)
      key <- paste0(org, "|", from, "-", to)
      add(paste0("series.", scope, ".stayed"), key, fmt_int(r$stayed))
      add(paste0("series.", scope, ".start"), key, fmt_int(r$start))
      add(paste0("series.", scope, ".percentile"), key, fmt_real(p))
      st <- c(st, r$stayed); sc <- c(sc, r$start)
    }
    series[[scope]] <- list(stayed = st, start = sc)
  }

  keep2020 <- transitions$from >= 2020
  pools <- list(
    teacher = series$teacher,
    teacherWithAlt = series$teacherWithAlt,
    all = series$all,
    teacherSince2020 = list(stayed = series$teacher$stayed[keep2020],
                            start = series$teacher$start[keep2020])
  )
  for (nm in names(pools)) {
    s <- sum(pools[[nm]]$stayed); n <- sum(pools[[nm]]$start)
    add(paste0("pooled.", nm, ".stayed"), org, fmt_int(s))
    add(paste0("pooled.", nm, ".start"), org, fmt_int(n))
    if (nm == "teacher") pooled_teacher[[org]] <- if (n == 0) NULL else s / n
  }

  for (y in years) {
    pft <- count_at(rows_by[[as.character(y)]]$teacher, org)
    alt <- count_at(rows_by[[as.character(y)]]$teacherWithAlt, org) - pft
    add("alt.pft", paste0(org, "|", y), fmt_int(pft))
    add("alt.alt", paste0(org, "|", y), fmt_int(alt))
  }

  runs <- compress(sapply(years, principal_names, org = org))
  add("principal.runs", org, fmt_int(length(runs)))
  # Runs minus one: three runs over the window means two changes.
  add("principal.changes", org, fmt_int(max(0, length(runs) - 1)))
  for (i in seq_along(runs)) {
    add("principal.run", sprintf("%s|%02d", org, i - 1),
        fmt_text(sprintf("%d-%d|%s", runs[[i]]$from, runs[[i]]$to, runs[[i]]$names)))
  }
  changes[[org]] <- max(0, length(runs) - 1)
  tenure[[org]] <- max(sapply(runs, function(r) r$to - r$from + 1))

  since <- years[years >= 2020]
  base <- unique(at(keys_by[[as.character(since[1])]]$teacher, org))
  add("cohort2020.base", org, fmt_int(length(base)))
  for (y in since) {
    now <- at(keys_by[[as.character(y)]]$teacher, org)
    add("cohort2020.remaining", paste0(org, "|", y), fmt_int(length(intersect(base, now))))
  }

  add("teacherCount", org, fmt_int(count_at(rows_by[[as.character(tail(years, 1))]]$teacher, org)))

  dest <- c(stayed = 0L, movedSchool = 0L, changedRole = 0L, leftDistrict = 0L)
  for (ti in seq_len(nrow(transitions))) {
    from <- transitions$from[ti]; to <- transitions$to[ti]
    ks <- unique(at(keys_by[[as.character(from)]]$teacher, org))
    idx <- teacher_index[[as.character(to)]]
    where <- idx$orgs[match(ks, idx$keys)]
    on_payroll <- ks %in% idx$payroll
    dest["stayed"] <- dest["stayed"] + sum(!is.na(where) & where == org)
    dest["movedSchool"] <- dest["movedSchool"] + sum(!is.na(where) & where != org)
    dest["changedRole"] <- dest["changedRole"] + sum(is.na(where) & on_payroll)
    dest["leftDistrict"] <- dest["leftDistrict"] + sum(is.na(where) & !on_payroll)
  }
  for (nm in names(dest)) add(paste0("destination.", nm), org, fmt_int(dest[[nm]]))

  eras <- list()
  for (r in runs) {
    m <- transitions$from >= r$from & transitions$from <= r$to
    s <- sum(series$teacher$stayed[m]); n <- sum(series$teacher$start[m])
    if (n > 0) eras[[length(eras) + 1]] <- list(run = r, stayed = s, start = n)
  }
  add("era.n", org, fmt_int(length(eras)))
  for (i in seq_along(eras)) {
    k <- sprintf("%s|%02d", org, i - 1)
    add("era.span", k, fmt_text(sprintf("%d-%d|%s", eras[[i]]$run$from, eras[[i]]$run$to, eras[[i]]$run$names)))
    add("era.stayed", k, fmt_int(eras[[i]]$stayed))
    add("era.start", k, fmt_int(eras[[i]]$start))
  }
}
```

## Does principal churn predict teacher retention?

This is the claim the whole project started from: that a principal needs about
five years to move a school, and that stability at the top is what drives
everything else. It is testable directly — correlate how often a school changed
principal against how well it held on to its teachers.

Pearson's r is written as an explicit loop rather than with `cor()`, so that all
three implementations add their terms up in the same order and land on the same
bits. Both input series are published alongside the coefficient, so that a
disagreement can be traced to the inputs or to the arithmetic.


```r
pearson <- function(xs, ys) {
  n <- min(length(xs), length(ys))
  if (n < 2) return(NULL)
  sx <- 0; sy <- 0
  for (i in seq_len(n)) { sx <- sx + xs[i]; sy <- sy + ys[i] }
  mx <- sx / n; my <- sy / n
  num <- 0; dx <- 0; dy <- 0
  for (i in seq_len(n)) {
    a <- xs[i] - mx; b <- ys[i] - my
    num <- num + a * b; dx <- dx + a * a; dy <- dy + b * b
  }
  if (dx == 0 || dy == 0) return(NULL)
  num / sqrt(dx * dy)
}

rankable <- k8_orgs[!sapply(k8_orgs, function(o) is.null(pooled_teacher[[o]]))]
for (o in rankable) {
  add("correlation.churn", o, fmt_int(changes[[o]]))
  add("correlation.tenure", o, fmt_int(tenure[[o]]))
}
churn_r <- pearson(sapply(rankable, function(o) changes[[o]]),
                   sapply(rankable, function(o) pooled_teacher[[o]]))
tenure_r <- pearson(sapply(rankable, function(o) tenure[[o]]),
                    sapply(rankable, function(o) pooled_teacher[[o]]))
add("correlation.n", "-", fmt_int(length(rankable)))
add("correlation.churnVsRetention", "-", fmt_real(churn_r))
add("correlation.tenureVsRetention", "-", fmt_real(tenure_r))

data.frame(
  measure = c("principal changes vs pooled teacher retention",
              "longest tenure vs pooled teacher retention"),
  r = c(round(churn_r, 3), round(tenure_r, 3)),
  n = length(rankable)
)
#>                                         measure      r  n
#> 1 principal changes vs pooled teacher retention  0.004 11
#> 2    longest tenure vs pooled teacher retention -0.044 11
```

The claim predicts a clearly negative first row.

## Pooled retention, by school


```r
summary_tbl <- data.frame(
  org = k8_orgs,
  pooled_15_25 = round(sapply(k8_orgs, function(o) pooled_teacher[[o]]), 4),
  principals = sapply(k8_orgs, function(o) changes[[o]] + 1),
  longest_run = sapply(k8_orgs, function(o) tenure[[o]])
)
summary_tbl[order(-summary_tbl$pooled_15_25), ]
#>       org pooled_15_25 principals longest_run
#> 1280 1280       0.8924          4           6
#> 2470 2470       0.8600          1          11
#> 2380 2380       0.8492          5           4
#> 1390 1390       0.8441          1          11
#> 2590 2590       0.8314          3           5
#> 2210 2210       0.8191          2           9
#> 1470 1470       0.8110          3           6
#> 2480 2480       0.8097          4           4
#> 1340 1340       0.7945          5           4
#> 1300 1300       0.7315          4           5
#> 1260 1260       0.6788          2          10
```

## Output

Sorted by metric then key, in byte order. Uppercase sorts before lowercase, and
`radix` is what pins that down regardless of the session's locale.


```r
stopifnot(!any(duplicated(paste0(rows$metric, "\t", rows$key))))
ord <- order(paste0(rows$metric, "\t", rows$key), method = "radix")
lines <- c("metric\tkey\tvalue",
           paste(rows$metric[ord], rows$key[ord], rows$value[ord], sep = "\t"))

out_dir <- file.path(ROOT, "calc", "out")
dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
con <- file(file.path(out_dir, "r.tsv"), open = "wb")
writeLines(lines, con, sep = "\n", useBytes = TRUE)
close(con)

cat(sprintf("facts: %d\nsha256: %s\n", length(lines) - 1,
            sha256(paste0(paste(lines, collapse = "\n"), "\n"))))
#> facts: 2697
#> sha256: 5482b6dab98213dd934861682542b7823069ae4c8d3be32be97140ee5e0f5ddd
```

The TypeScript and Python implementations print the same hash.

#!/usr/bin/env python3
"""
The Python implementation of the SDP retention calculation.

This is a second, independent implementation of the same calculation the
published site runs, written against the same eleven October extracts. It shares
no code with either the TypeScript original or the R Markdown implementation —
only the contract in ``calc/CONTRACT.md``, which says what facts to emit and how
to write them down. If all three files come out byte-identical, the calculation
does not depend on who implemented it.

Standard library only, on purpose: pandas would make some of this shorter, but
the point of the exercise is that three implementations agree, and a
column-oriented library agreeing with two row-oriented ones proves less than it
appears to. What is worth checking here is the definitions, not the dataframes.

Run: npm run calc:py   (writes calc/out/retention-python.tsv)
"""

from __future__ import annotations

import csv
import math
import os
import re
import sys
from collections import OrderedDict

from contract import fmt_int, fmt_real, fmt_text, serialize, sha256

HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
DATA = os.path.join(ROOT, "data", "extracted")

# --------------------------------------------------------------------- inputs

SNAPSHOTS = [
    (2015, "employee_information_Oct2015.csv"),
    (2016, "employee_information_Oct2016.csv"),
    (2017, "employee_information_Oct2017.csv"),
    (2018, "employee_information_Oct2018.csv"),
    (2019, "employee_information_Oct2019.csv"),
    (2020, "employee_information_Oct2020.csv"),
    (2021, "employee_information_Oct2021.csv"),
    (2022, "employee_information_Oct2022.csv"),
    (2023, "Employee_Data_2023-2024/employee_information_October2023.csv"),
    (2024, "employee_information_Oct2024.csv"),
    (2025, "Oct 2025/employee_information_Oct2025.csv"),
]

YEARS = [y for y, _ in SNAPSHOTS]
TRANSITIONS = list(zip(YEARS, YEARS[1:]))

LEVELS = [("ELEMENTARY", "ELEMENTARY"), ("MIDDLE SCHOOL", "MIDDLE_SCHOOL"), ("HIGH SCHOOL", "HIGH_SCHOOL")]
SCOPES = ["teacher", "teacherWithAlt", "all"]
MIN_TEACHERS = 15

# org code -> peer level. Anything not listed is ranked against elementary peers.
K8_ORGS = ["1300", "1280", "1340", "1260", "1390", "1470", "2380", "2590", "2470", "2480", "2210"]
SECONDARY_ORGS = [("2050", "MIDDLE SCHOOL"), ("2650", "HIGH SCHOOL"), ("2680", "HIGH SCHOOL"), ("6040", "HIGH SCHOOL")]
SCHOOLS = [(org, "ELEMENTARY") for org in K8_ORGS] + SECONDARY_ORGS

# The six fields every implementation reads, in the order they are hashed.
FIELDS = ("LAST_NAME", "FIRST_NAME", "TITLE_DESCRIPTION", "HOME_ORGANIZATION",
          "ORGANIZATION_LEVEL", "TYPE_OF_REPRESENTATION")


class Employee:
    """One row of an extract, reduced to the fields the calculation uses."""

    __slots__ = ("last", "first", "title", "org", "level", "rep")

    def __init__(self, row):
        self.last = row.get("LAST_NAME") or ""
        self.first = row.get("FIRST_NAME") or ""
        self.title = row.get("TITLE_DESCRIPTION") or ""
        self.org = row.get("HOME_ORGANIZATION") or ""
        self.level = row.get("ORGANIZATION_LEVEL") or ""
        self.rep = row.get("TYPE_OF_REPRESENTATION") or ""

    def fields(self):
        return (self.last, self.first, self.title, self.org, self.level, self.rep)


def load(path):
    """Read one October extract. The extracts are latin-1, not UTF-8."""
    with open(path, encoding="latin-1", newline="") as fh:
        return [Employee(r) for r in csv.DictReader(fh)]


# ----------------------------------------------------------------- definitions

_NOT_LETTER = re.compile(r"[^A-Z ]")
_SPACES = re.compile(r"\s+")


def employee_key(e):
    """
    Identity key for matching a person across two snapshots.

    The extracts carry no employee ID, so name is all there is. Case,
    punctuation and internal spacing are normalized away so that a cosmetic
    spelling change does not read as somebody leaving.
    """
    def norm(s):
        return _SPACES.sub(" ", _NOT_LETTER.sub("", s.upper())).strip()
    return norm(e.last) + "|" + norm(e.first)


# `\b` is deliberately ASCII-only, to match the JavaScript regular expression
# this definition was first written as; Python's default is Unicode-aware.
_TEACHER_TITLE = re.compile(r"^TEACHER\b", re.ASCII)


def is_teacher(e):
    """Classroom teachers: a PFT teaching local *and* a TEACHER title."""
    return e.rep.upper().startswith("PFT") and _TEACHER_TITLE.match(e.title.upper()) is not None


def is_alt_instructional(e):
    """Student teachers and bought-in contracted placements: not district hires."""
    t = e.title.upper()
    return "STUDENT TEACHER" in t or "CONTRACTED TEACHER" in t


def is_school_leader(e):
    """The building principal — not assistant principals or retired fill-ins."""
    return e.title.strip().upper() == "PRINCIPAL"


SCOPE_FILTERS = {
    "teacher": is_teacher,
    "teacherWithAlt": lambda e: is_teacher(e) or is_alt_instructional(e),
    "all": lambda e: True,
}


# --------------------------------------------------------------------- corpus

class Corpus:
    """Indexed view over the yearly extracts, mirroring src/report.ts."""

    def __init__(self, by_year):
        self.by_year = by_year
        # year -> org -> [employee], with orgs in the order they first appear.
        self.index = {}
        for year, staff in by_year.items():
            by_org = OrderedDict()
            for e in staff:
                by_org.setdefault(e.org, []).append(e)
            self.index[year] = by_org

        # Person-level indexes, for telling a transfer from an exit. A duplicated
        # name is resolved the same way a dictionary assignment resolves it: the
        # last row in the file wins.
        self.teacher_org = {}
        self.payroll = {}
        for year, staff in by_year.items():
            t, p = {}, set()
            for e in staff:
                k = employee_key(e)
                p.add(k)
                if is_teacher(e):
                    t[k] = e.org
            self.teacher_org[year] = t
            self.payroll[year] = p

        self._peer_orgs = {}
        self._peer_rates = {}

    def staff_at(self, year, org, scope):
        keep = SCOPE_FILTERS[scope]
        return [e for e in self.index.get(year, {}).get(org, []) if keep(e)]

    def leaders_at(self, year, org):
        return [e for e in self.index.get(year, {}).get(org, []) if is_school_leader(e)]

    def peer_orgs(self, year, level, min_teachers):
        cached = self._peer_orgs.get((year, level, min_teachers))
        if cached is not None:
            return cached
        out = []
        for org, staff in self.index.get(year, {}).items():
            # The level is read off the org's first row, as the original does:
            # a handful of orgs carry rows labelled with more than one level.
            if not staff or level not in staff[0].level.upper():
                continue
            if sum(1 for e in staff if is_teacher(e)) >= min_teachers:
                out.append(org)
        self._peer_orgs[(year, level, min_teachers)] = out
        return out

    def peer_rates(self, frm, to, level, min_teachers, scope):
        cached = self._peer_rates.get((frm, to, level, min_teachers, scope))
        if cached is not None:
            return cached
        rates = []
        for org in self.peer_orgs(frm, level, min_teachers):
            r = retention_between(self.staff_at(frm, org, scope), self.staff_at(to, org, scope))
            if r[2] is not None:
                rates.append(r[2])
        self._peer_rates[(frm, to, level, min_teachers, scope)] = rates
        return rates


def retention_between(before, after):
    """(stayed, start_count, rate) for one school across consecutive Octobers."""
    b = {employee_key(e) for e in before}
    a = {employee_key(e) for e in after}
    stayed = len(b & a)
    start = len(b)
    return stayed, start, (None if start == 0 else stayed / start)


# ----------------------------------------------------------------------- stats

def median(values):
    if not values:
        return None
    s = sorted(values)
    mid = len(s) // 2
    return s[mid] if len(s) % 2 == 1 else (s[mid - 1] + s[mid]) / 2


def percentile_rank(values, target):
    """Share of `values` at or below `target`."""
    if not values:
        return None
    return sum(1 for v in values if v <= target) / len(values)


def quantile_at(sorted_rates, q):
    """The original's quartile: index into the sorted rates, no interpolation."""
    if not sorted_rates:
        return None
    return sorted_rates[math.floor(q * (len(sorted_rates) - 1))]


def pearson(xs, ys):
    """
    Pearson's r, accumulated in an explicit left-to-right loop.

    Written as a loop rather than with sum() so that all three implementations
    add their terms up in the same order, and land on the same bits.
    """
    n = min(len(xs), len(ys))
    if n < 2:
        return None
    sx = 0.0
    sy = 0.0
    for i in range(n):
        sx += xs[i]
        sy += ys[i]
    mx, my = sx / n, sy / n
    num = dx = dy = 0.0
    for i in range(n):
        a, b = xs[i] - mx, ys[i] - my
        num += a * b
        dx += a * a
        dy += b * b
    if dx == 0 or dy == 0:
        return None
    return num / math.sqrt(dx * dy)


# ------------------------------------------------------------------ the report

def compress_timeline(entries):
    """Collapse a year-by-year holder list into runs, so repeats read as one tenure."""
    runs = []
    for year, names in entries:
        if runs and runs[-1][0] == names:
            runs[-1][2] = year
        else:
            runs.append([names, year, year])
    return runs


def build():
    by_year = OrderedDict()
    for year, filename in SNAPSHOTS:
        path = os.path.join(DATA, filename)
        if not os.path.exists(path):
            raise SystemExit("missing snapshot: %s — run ./fetch-data.sh" % path)
        by_year[year] = load(path)

    corpus = Corpus(by_year)
    rows = []

    def add(metric, key, value):
        rows.append((metric, key, value))

    # ------------------------------------------------------------------ inputs
    for year in YEARS:
        staff = by_year[year]
        keys = [employee_key(e) for e in staff]
        add("input.rows", str(year), fmt_int(len(staff)))
        add("input.digest", str(year), fmt_text(sha256("\n".join("\x1f".join(e.fields()) for e in staff))))
        add("input.keydigest", str(year), fmt_text(sha256("\n".join(sorted(set(keys))))))
        add("input.keys", str(year), fmt_int(len(set(keys))))
        add("input.teachers", str(year), fmt_int(sum(1 for e in staff if is_teacher(e))))
        add("input.principals", str(year), fmt_int(sum(1 for e in staff if is_school_leader(e))))

    # -------------------------------------------------------------- peer sets
    for level, tok in LEVELS:
        for year in YEARS:
            add("peer.orgs", "%d|%s" % (year, tok), fmt_int(len(corpus.peer_orgs(year, level, MIN_TEACHERS))))
        for frm, to in TRANSITIONS:
            for scope in SCOPES:
                rates = sorted(corpus.peer_rates(frm, to, level, MIN_TEACHERS, scope))
                key = "%d-%d|%s|%s" % (frm, to, tok, scope)
                add("peer.n", key, fmt_int(len(rates)))
                add("peer.median", key, fmt_real(median(rates)))
                add("peer.q1", key, fmt_real(quantile_at(rates, 0.25)))
                add("peer.q3", key, fmt_real(quantile_at(rates, 0.75)))

    # ----------------------------------------------------------- per school
    pooled_teacher = {}
    principal_changes = {}
    longest_tenure = {}

    for org, level in SCHOOLS:
        tok = dict(LEVELS)[level]
        add("school.level", org, fmt_text(tok))

        series = {}
        for scope in SCOPES:
            out = []
            for frm, to in TRANSITIONS:
                stayed, start, rate = retention_between(
                    corpus.staff_at(frm, org, scope), corpus.staff_at(to, org, scope))
                peers = corpus.peer_rates(frm, to, level, MIN_TEACHERS, scope)
                pct = None if rate is None else percentile_rank(peers, rate)
                out.append((frm, to, stayed, start, pct))
                key = "%s|%d-%d" % (org, frm, to)
                add("series.%s.stayed" % scope, key, fmt_int(stayed))
                add("series.%s.start" % scope, key, fmt_int(start))
                add("series.%s.percentile" % scope, key, fmt_real(pct))
            series[scope] = out

        pooled = [
            ("teacher", series["teacher"]),
            ("teacherWithAlt", series["teacherWithAlt"]),
            ("all", series["all"]),
            ("teacherSince2020", [s for s in series["teacher"] if s[0] >= 2020]),
        ]
        for name, rowset in pooled:
            stayed = sum(s[2] for s in rowset)
            start = sum(s[3] for s in rowset)
            add("pooled.%s.stayed" % name, org, fmt_int(stayed))
            add("pooled.%s.start" % name, org, fmt_int(start))
            if name == "teacher":
                pooled_teacher[org] = None if start == 0 else stayed / start

        for year in YEARS:
            pft = len(corpus.staff_at(year, org, "teacher"))
            alt = len(corpus.staff_at(year, org, "teacherWithAlt")) - pft
            add("alt.pft", "%s|%d" % (org, year), fmt_int(pft))
            add("alt.alt", "%s|%d" % (org, year), fmt_int(alt))

        entries = []
        for year in YEARS:
            names = " + ".join(sorted("%s %s" % (l.first, l.last) for l in corpus.leaders_at(year, org)))
            entries.append((year, names if names else "(none listed)"))
        runs = compress_timeline(entries)
        add("principal.runs", org, fmt_int(len(runs)))
        # Runs minus one: three runs over the window means two changes.
        add("principal.changes", org, fmt_int(max(0, len(runs) - 1)))
        for n, (names, frm, to) in enumerate(runs):
            add("principal.run", "%s|%02d" % (org, n), fmt_text("%d-%d|%s" % (frm, to, names)))
        principal_changes[org] = max(0, len(runs) - 1)
        longest_tenure[org] = max(to - frm + 1 for _, frm, to in runs)

        since = [y for y in YEARS if y >= 2020]
        base_keys = {employee_key(e) for e in corpus.staff_at(since[0], org, "teacher")}
        add("cohort2020.base", org, fmt_int(len(base_keys)))
        for year in since:
            now = {employee_key(e) for e in corpus.staff_at(year, org, "teacher")}
            add("cohort2020.remaining", "%s|%d" % (org, year), fmt_int(len(base_keys & now)))

        add("teacherCount", org, fmt_int(len(corpus.staff_at(YEARS[-1], org, "teacher"))))

        dest = {"stayed": 0, "movedSchool": 0, "changedRole": 0, "leftDistrict": 0}
        for frm, to in TRANSITIONS:
            for k in {employee_key(e) for e in corpus.staff_at(frm, org, "teacher")}:
                where = corpus.teacher_org[to].get(k)
                if where == org:
                    dest["stayed"] += 1
                elif where is not None:
                    dest["movedSchool"] += 1
                elif k in corpus.payroll[to]:
                    dest["changedRole"] += 1
                else:
                    dest["leftDistrict"] += 1
        for name in ("stayed", "movedSchool", "changedRole", "leftDistrict"):
            add("destination.%s" % name, org, fmt_int(dest[name]))

        # A transition belongs to whoever held the building in its starting
        # October — that is the year whose staff either stayed or left.
        eras = []
        for names, frm, to in runs:
            mine = [s for s in series["teacher"] if frm <= s[0] <= to]
            stayed = sum(s[2] for s in mine)
            start = sum(s[3] for s in mine)
            if start > 0:
                eras.append((names, frm, to, stayed, start))
        add("era.n", org, fmt_int(len(eras)))
        for n, (names, frm, to, stayed, start) in enumerate(eras):
            add("era.span", "%s|%02d" % (org, n), fmt_text("%d-%d|%s" % (frm, to, names)))
            add("era.stayed", "%s|%02d" % (org, n), fmt_int(stayed))
            add("era.start", "%s|%02d" % (org, n), fmt_int(start))

    # ------------------------------- does principal churn predict retention?
    rankable = [org for org in K8_ORGS if pooled_teacher.get(org) is not None]
    for org in rankable:
        add("correlation.churn", org, fmt_int(principal_changes[org]))
        add("correlation.tenure", org, fmt_int(longest_tenure[org]))
    add("correlation.n", "-", fmt_int(len(rankable)))
    add("correlation.churnVsRetention", "-",
        fmt_real(pearson([principal_changes[o] for o in rankable], [pooled_teacher[o] for o in rankable])))
    add("correlation.tenureVsRetention", "-",
        fmt_real(pearson([longest_tenure[o] for o in rankable], [pooled_teacher[o] for o in rankable])))

    return rows


def main():
    out_dir = os.path.join(ROOT, "calc", "out")
    os.makedirs(out_dir, exist_ok=True)
    tsv = serialize(build())
    with open(os.path.join(out_dir, "retention-python.tsv"), "w", encoding="utf-8", newline="\n") as fh:
        fh.write(tsv)
    sys.stdout.write("wrote calc/out/retention-python.tsv (%d facts)\n" % (tsv.count("\n") - 1))


if __name__ == "__main__":
    main()

/**
 * The TypeScript implementation's side of the calculation contract.
 *
 * This is the trunk the strangler fig grew around: it computes nothing itself.
 * Every number here comes from the same `src/` modules the published site has
 * always used, and this file only writes them out in the contract's format so
 * that the R Markdown and Python implementations have something to be compared
 * against.
 *
 * Run: npm run calc:ts   (writes calc/out/typescript.tsv)
 */

import { mkdirSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'

import { int, real, serialize, text, type Row } from '../src/contract.ts'
import { createCorpus, buildSchoolReport, type Corpus, type SchoolReport } from '../src/report.ts'
import { employeeKey, isSchoolLeader, isTeacher } from '../src/retention.ts'
import { K8_SCHOOLS, SECONDARY_SCHOOLS, type SchoolRef } from '../src/schools.ts'
import { keyDigest, loadSnapshots, snapshotDigest, YEARS } from '../src/snapshots.ts'
import { median, pearson } from '../src/stats.ts'

const ROOT = join(import.meta.dirname, '..')

/** Peer level as it appears in a contract key: no spaces, so keys stay sortable. */
const LEVELS: { level: string; token: string }[] = [
  { level: 'ELEMENTARY', token: 'ELEMENTARY' },
  { level: 'MIDDLE SCHOOL', token: 'MIDDLE_SCHOOL' },
  { level: 'HIGH SCHOOL', token: 'HIGH_SCHOOL' },
]
const token = (level: string): string => LEVELS.find((l) => l.level === level)?.token ?? level

const SCOPES = ['teacher', 'teacherWithAlt', 'all'] as const

/** Minimum teaching staff for a school to count as a peer. Matches buildSchoolReport. */
const MIN_TEACHERS = 15

/** Two digits, so that run 10 sorts after run 9 rather than after run 1. */
const idx = (i: number): string => String(i).padStart(2, '0')

const transitions = YEARS.slice(0, -1).map((from, i) => ({ from, to: YEARS[i + 1] as number }))
const span = (t: { from: number; to: number }): string => `${t.from}-${t.to}`

export function emitRows(): Row[] {
  const byYear = loadSnapshots(join(ROOT, 'data', 'extracted'))
  const corpus: Corpus = createCorpus(byYear)
  const rows: Row[] = []
  const add = (metric: string, key: string, value: string): void => {
    rows.push({ metric, key, value })
  }

  // ------------------------------------------------------------------ inputs
  // Emitted first so that a disagreement about the numbers can be told apart
  // from a disagreement about what was read off disk.
  for (const year of YEARS) {
    const staff = byYear.get(year) ?? []
    add('input.rows', String(year), int(staff.length))
    add('input.digest', String(year), text(snapshotDigest(staff)))
    add('input.keydigest', String(year), text(keyDigest(staff.map(employeeKey))))
    add('input.keys', String(year), int(new Set(staff.map(employeeKey)).size))
    add('input.teachers', String(year), int(staff.filter(isTeacher).length))
    add('input.principals', String(year), int(staff.filter(isSchoolLeader).length))
  }

  // -------------------------------------------------------------- peer sets
  for (const { level, token: tok } of LEVELS) {
    for (const year of YEARS) {
      add('peer.orgs', `${year}|${tok}`, int(corpus.peerOrgs(year, level, MIN_TEACHERS).length))
    }
    for (const t of transitions) {
      for (const scope of SCOPES) {
        const rates = [...corpus.peerRates(t.from, t.to, level, MIN_TEACHERS, scope)].sort((a, b) => a - b)
        const at = (q: number): number | null => rates[Math.floor(q * (rates.length - 1))] ?? null
        const key = `${span(t)}|${tok}|${scope}`
        add('peer.n', key, int(rates.length))
        add('peer.median', key, real(median(rates)))
        add('peer.q1', key, real(at(0.25)))
        add('peer.q3', key, real(at(0.75)))
      }
    }
  }

  // ----------------------------------------------------------- per school
  const schools: SchoolRef[] = [...K8_SCHOOLS, ...SECONDARY_SCHOOLS]
  const reports = new Map<string, SchoolReport>()

  for (const school of schools) {
    const level = school.level ?? 'ELEMENTARY'
    const r = buildSchoolReport(corpus, school, YEARS, level, MIN_TEACHERS)
    reports.set(school.org, r)
    const org = school.org

    add('school.level', org, text(token(level)))

    const seriesFor = { teacher: r.teacher, teacherWithAlt: r.teacherWithAlt, all: r.allStaff }
    for (const scope of SCOPES) {
      for (const y of seriesFor[scope]) {
        const key = `${org}|${y.from}-${y.to}`
        add(`series.${scope}.stayed`, key, int(y.stayed))
        add(`series.${scope}.start`, key, int(y.startCount))
        add(`series.${scope}.percentile`, key, real(y.percentile))
      }
    }

    // Pooled rates are published as their two integers, so the comparison of the
    // headline figure is a comparison of counts rather than of floating point.
    const pooled: [string, { stayed: number; startCount: number }[]][] = [
      ['teacher', r.teacher],
      ['teacherWithAlt', r.teacherWithAlt],
      ['all', r.allStaff],
      ['teacherSince2020', r.teacher.filter((t) => t.from >= 2020)],
    ]
    for (const [name, series] of pooled) {
      add(`pooled.${name}.stayed`, org, int(series.reduce((n, y) => n + y.stayed, 0)))
      add(`pooled.${name}.start`, org, int(series.reduce((n, y) => n + y.startCount, 0)))
    }

    for (const a of r.altShare) {
      add('alt.pft', `${org}|${a.year}`, int(a.pft))
      add('alt.alt', `${org}|${a.year}`, int(a.alt))
    }

    add('principal.runs', org, int(r.principalRuns.length))
    add('principal.changes', org, int(r.principalChanges))
    r.principalRuns.forEach((run, i) => {
      add('principal.run', `${org}|${idx(i)}`, text(`${run.from}-${run.to}|${run.names}`))
    })

    // The base year's own row counts the cohort against itself, so its
    // `remaining` is the size of the cohort.
    add('cohort2020.base', org, int(r.cohort2020[0]?.remaining ?? 0))
    for (const c of r.cohort2020) add('cohort2020.remaining', `${org}|${c.year}`, int(c.remaining))

    add('teacherCount', org, int(r.teacherCount))

    add('destination.stayed', org, int(r.destinations.stayed))
    add('destination.movedSchool', org, int(r.destinations.movedSchool))
    add('destination.changedRole', org, int(r.destinations.changedRole))
    add('destination.leftDistrict', org, int(r.destinations.leftDistrict))

    add('era.n', org, int(r.eras.length))
    r.eras.forEach((e, i) => {
      add('era.span', `${org}|${idx(i)}`, text(`${e.from}-${e.to}|${e.names}`))
      add('era.stayed', `${org}|${idx(i)}`, int(e.stayed))
      add('era.start', `${org}|${idx(i)}`, int(e.startCount))
    })
  }

  // ------------------------------------------- does principal churn predict retention?
  // The two input series are emitted alongside the coefficients: if a
  // correlation disagrees, this says whether the inputs or the arithmetic did.
  const rankable = K8_SCHOOLS.map((s) => reports.get(s.org) as SchoolReport).filter((r) => r.pooledTeacher !== null)
  const longestTenure = (r: SchoolReport): number => Math.max(...r.principalRuns.map((run) => run.to - run.from + 1))

  for (const r of rankable) {
    add('correlation.churn', r.org, int(r.principalChanges))
    add('correlation.tenure', r.org, int(longestTenure(r)))
  }
  add('correlation.n', '-', int(rankable.length))
  add('correlation.churnVsRetention', '-', real(pearson(rankable.map((r) => r.principalChanges), rankable.map((r) => r.pooledTeacher as number))))
  add('correlation.tenureVsRetention', '-', real(pearson(rankable.map(longestTenure), rankable.map((r) => r.pooledTeacher as number))))

  return rows
}

if (import.meta.filename === process.argv[1]) {
  const outDir = join(ROOT, 'calc', 'out')
  mkdirSync(outDir, { recursive: true })
  const tsv = serialize(emitRows())
  writeFileSync(join(outDir, 'retention-typescript.tsv'), tsv)
  console.log(`wrote calc/out/retention-typescript.tsv (${tsv.split('\n').length - 2} facts)`)
}

metric	key	value
alt.alt	1260|2015	0
alt.alt	1260|2016	0
alt.alt	1260|2017	1
alt.alt	1260|2018	0
alt.alt	1260|2019	0
alt.alt	1260|2020	0
alt.alt	1260|2021	2
alt.alt	1260|2022	0
alt.alt	1260|2023	5
alt.alt	1260|2024	2
alt.alt	1260|2025	1
alt.alt	1280|2015	0
alt.alt	1280|2016	0
alt.alt	1280|2017	0
alt.alt	1280|2018	0
alt.alt	1280|2019	0
alt.alt	1280|2020	6
alt.alt	1280|2021	7
alt.alt	1280|2022	2
alt.alt	1280|2023	4
alt.alt	1280|2024	5
alt.alt	1280|2025	7
alt.alt	1300|2015	0

---
title: "School attendance, Philadelphia and Brooklyn — the calculation"
output: md_document
---

```{r setup, include = FALSE}
# error = FALSE so that knitting *fails* on an error rather than embedding it in
# the document.
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", error = FALSE)
options(stringsAsFactors = FALSE)
```

Philadelphia and New York both publish school attendance, and they publish
mirrored halves of the same fact: Philadelphia the share of students attending
**at least 90%** of days, New York the share **chronically absent** — missing
10% or more. This document normalizes everything to the second, which is the
federal standard metric, and then reaches for the one source that measures all
three cities on one definition in one year.

It is one of three implementations of this calculation. The others are the
original TypeScript in `src/` and `calc/attendance.py`; all three write the
contract described in `calc/CONTRACT.md` and the three files are compared byte
for byte.

```{r constants}
ROOT <- Sys.getenv("SDP_ROOT", ".")
A <- file.path(ROOT, "attendance")

nyc_years <- c("2016-17", "2017-18", "2018-19", "2019-20", "2020-21")
crdc_leas <- c("4218990", "0103360", "3600091", "3600092")
# 2017-18 is the last clean pre-pandemic collection; 2022-23 the most recent.
# The 2020-21 collection covers the remote year and is left out.
crdc_years <- c("2017", "2022")
tuscaloosa_seda_lea <- "103360"

k8 <- data.frame(
  org = c("1300", "1280", "1340", "1260", "1390", "1470", "2380", "2590", "2470", "2480", "2210"),
  label = c("Harrington", "Penn Alexander", "Lea", "Comegys", "Powel", "Alain Locke",
            "Meredith", "Nebinger", "Greenfield", "Marian Anderson", "Bache-Martin"),
  nces = c("421899006528", "421899000619", "421899006530", "421899003782", "421899003783",
           "421899004732", "421899003649", "421899003650", "421899005186", "421899003805",
           "421899003618")
)

nyc_schools <- data.frame(
  dbn = c("13K011", "13K009", "15K321", "15K107", "15K039", "15K051",
          "15K154", "15K010", "15K118", "13K133", "13K282"),
  nces = c("360009102074", "360009102069", "360009202818", "360009202403", "360009202175",
           "360009202226", "360009202557", "360009202072", "360009206406", "360009102490",
           "360009102793"),
  label = c("PS 11 Purvis J. Behan", "PS 9 Teunis G. Bergen", "PS 321 William Penn",
            "PS 107 John W. Kimball", "PS 39 Henry Bristow", "MS 51 William Alexander",
            "PS 154 Windsor Terrace", "PS 10 Magnet School", "PS 118 Maurice Sendak",
            "PS 133 William A. Butler", "PS 282 Park Slope")
)

# Each source, the short name it goes by in the contract, and the columns every
# implementation reads out of it. The digest covers those columns only: hashing
# a column nothing uses would fail the check for noise rather than for signal.
csv_sources <- list(
  ada_archive = list(file = "ex/Student_ADA_Yearly.csv",
                     cols = c("School Year", "ULCS Code", "Average Daily Attendance")),
  ada_current = list(file = "Student_ADA_School.csv",
                     cols = c("School Year", "ULCS Code", "Average Daily Attendance (YTD)")),
  attending90 = list(file = "Student_Attendance_90_School.csv",
                     cols = c("School Year", "ULCS Code", "Category", "% with 90%+ Attendance (Yearly)")),
  attending95_archive = list(file = "ex/SDP_95_Attendance_School_S.csv",
                             cols = c("School Year", "ULCS Code", "Category", "% with 95%+ Attendance (Yearly)")),
  attending95_current = list(file = "Student_Attendance_95_School.csv",
                             cols = c("School Year", "ULCS Code", "Category", "% with 95%+ Attendance (Yearly)")),
  teacher_ada = list(file = "ex/Archive_Teacher_95_ADA_2012-13_2022-23/OD_Teacher_Attendance_Schools_Suppressed_20240328.csv",
                     cols = c("school_year", "ulcscode", "pct_att_ada")),
  seda_covariates = list(file = "../seda/seda_cov_school_pool_4.1.csv",
                         cols = c("sedasch", "sedalea", "schnam"))
)
```

## Reading the files

The Philadelphia files are latin-1, and one of them carries a byte-order mark.
Left in place, that mark becomes part of the first column's name, so every
lookup against that column silently returns nothing — a failure that shows up as
an empty chart rather than as an error. It is stripped from the raw bytes,
before any decoding, which is the only place it is unambiguous.

```{r read}
read_extract <- function(rel) {
  path <- file.path(A, rel)
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-attendance.sh")
  raw <- readBin(path, "raw", file.size(path))
  # UTF-8 byte-order mark: EF BB BF.
  if (length(raw) >= 3 && identical(as.integer(raw[1:3]), c(239L, 187L, 191L))) raw <- raw[-(1:3)]
  txt <- iconv(rawToChar(raw), from = "latin1", to = "UTF-8")
  read.csv(text = txt, colClasses = "character", check.names = FALSE, na.strings = NULL)
}

sources <- lapply(csv_sources, function(s) read_extract(s$file))
sapply(sources, nrow)
```

## What a published cell means

Suppressed cells are published as `s`, not as a blank, and a school with too few
students to report is not a school with zero attendance. Anything that is not a
number is dropped rather than coerced.

```{r parse}
parse_value <- function(v) {
  t <- trimws(v)
  out <- rep(NA_real_, length(t))
  ok <- !(t == "" | tolower(t) == "s" | tolower(t) == "n/a")
  out[ok] <- suppressWarnings(as.numeric(t[ok]))
  out[!is.finite(out)] <- NA_real_
  out
}

# SDP school codes appear both zero-padded and unpadded across files.
normalize_ulcs <- function(code) {
  t <- sub("^0+", "", trimws(code))
  ifelse(t == "", "0", t)
}
```

## One series per school

Every Philadelphia file is in long format — one row per school per year, and in
some files per demographic group as well. They collapse to the same shape: a
school, a year, a number. Where a school-year appears twice, the last row in the
file wins, which is what an assignment does in the other two implementations.

```{r series}
series_from_rows <- function(df, year_col, code_col, value_col, keep = NULL) {
  if (!is.null(keep)) df <- df[keep(df), , drop = FALSE]
  v <- parse_value(df[[value_col]])
  code <- normalize_ulcs(df[[code_col]])
  year <- trimws(df[[year_col]])
  ok <- !is.na(v) & code != "" & year != ""
  v <- v[ok]; code <- code[ok]; year <- year[ok]

  out <- list()
  for (cd in split(seq_along(code), code)) {
    yrs <- year[cd]
    vals <- v[cd]
    last <- !duplicated(yrs, fromLast = TRUE)
    out[[code[cd[1]]]] <- setNames(vals[last], yrs[last])
  }
  out
}

all_students <- function(df) df$Category == "All Students"

# Two releases of the same series; the newer file wins year by year.
merge_series <- function(old, new) {
  out <- old
  for (cd in names(new)) {
    have <- out[[cd]]
    add <- new[[cd]]
    if (is.null(have)) out[[cd]] <- add
    else out[[cd]] <- c(add, have[setdiff(names(have), names(add))])
  }
  out
}
```

```{r philadelphia}
phl_ada <- merge_series(
  series_from_rows(sources$ada_archive, "School Year", "ULCS Code", "Average Daily Attendance"),
  series_from_rows(sources$ada_current, "School Year", "ULCS Code", "Average Daily Attendance (YTD)")
)

attending90 <- series_from_rows(sources$attending90, "School Year", "ULCS Code",
                                "% with 90%+ Attendance (Yearly)", all_students)
# Philadelphia publishes the attending share; the standard metric is its complement.
phl_chronic <- lapply(attending90, function(y) 100 - y)

phl_95 <- merge_series(
  series_from_rows(sources$attending95_archive, "School Year", "ULCS Code",
                   "% with 95%+ Attendance (Yearly)", all_students),
  series_from_rows(sources$attending95_current, "School Year", "ULCS Code",
                   "% with 95%+ Attendance (Yearly)", all_students)
)

teacher_ada <- series_from_rows(sources$teacher_ada, "school_year", "ulcscode", "pct_att_ada")

length(phl_ada)
```

## Reference lines

A school's absence rate means little without knowing what the district looked
like that year, so each measure also gets a districtwide median across every
school SDP reports — not across the eleven on the page.

```{r medians}
med <- function(v) {
  v <- v[!is.na(v)]
  if (!length(v)) return(NULL)
  s <- sort(v)
  mid <- length(s) %/% 2
  if (length(s) %% 2 == 1) s[mid + 1] else (s[mid] + s[mid + 1]) / 2
}

district_medians <- function(series) {
  years <- unlist(lapply(series, names), use.names = FALSE)
  vals <- unlist(series, use.names = FALSE)
  split(vals, years)
}
```

## New York

New York publishes one row per school per year, already as a chronic-absence
percentage, so nothing has to be converted — only read.

```{r newyork}
nyc_raw <- lapply(nyc_years, function(y) {
  path <- file.path(A, sprintf("nyc_attendance_%s.json", y))
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-attendance.sh")
  jsonlite::fromJSON(path, simplifyDataFrame = TRUE)
})
names(nyc_raw) <- nyc_years

nyc_by_year <- lapply(nyc_raw, function(df) {
  ada <- parse_value(df$attendance)
  chronic <- parse_value(df$chronically_absent_1)
  ok <- !is.na(df$dbn) & !is.na(ada) & !is.na(chronic)
  keep <- ok & !duplicated(df$dbn, fromLast = TRUE)
  list(dbn = df$dbn[keep], ada = ada[keep], chronic = chronic[keep])
})

sapply(nyc_by_year, function(x) length(x$dbn))
```

## One year, three cities, one definition

Philadelphia, New York and Tuscaloosa each publish attendance their own way,
over different windows, so their own figures cannot be set side by side. The
federal Civil Rights Data Collection asks every district for the same count in
the same year, which makes it the one apples-to-apples measure available.

It publishes counts rather than rates, so enrolment comes from the Common Core
of Data as the denominator — and it returns **two rows per school** even with
every other subgroup pinned to "all": one for all students and one for the
homeless subgroup. Keeping the wrong one produces single-digit rates that look
entirely plausible.

```{r crdc}
read_results <- function(rel) {
  path <- file.path(A, rel)
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-crdc.sh")
  jsonlite::fromJSON(path, simplifyDataFrame = TRUE)$results
}

read_crdc_year <- function(year) {
  absent <- list(); enrolled <- list()
  for (lea in crdc_leas) {
    a <- read_results(sprintf("crdc_absent_%s_%s.json", year, lea))
    if (!is.null(a) && nrow(a)) {
      keep <- (is.na(a$homeless) | a$homeless == 99) & !is.na(a$ncessch) &
        !is.na(a$students_chronically_absent)
      a <- a[keep, , drop = FALSE]
      last <- !duplicated(a$ncessch, fromLast = TRUE)
      absent[a$ncessch[last]] <- a$students_chronically_absent[last]
    }
    e <- read_results(sprintf("crdc_enrolled_%s_%s.json", year, lea))
    if (!is.null(e) && nrow(e)) {
      e <- e[!is.na(e$ncessch) & !is.na(e$enrollment), , drop = FALSE]
      last <- !duplicated(e$ncessch, fromLast = TRUE)
      enrolled[e$ncessch[last]] <- e$enrollment[last]
    }
  }
  list(absent = absent, enrolled = enrolled)
}

crdc <- lapply(crdc_years, read_crdc_year)
names(crdc) <- crdc_years

# CRDC encodes suppressed or not-applicable cells as negative numbers (-3, -5,
# -9). Those must not be divided; nor may a rate above 100 be reported, which
# would mean the count and the denominator came from mismatched records.
federal_rate <- function(absent, enrolled) {
  if (is.null(absent) || is.na(absent) || absent < 0) return(NULL)
  if (is.null(enrolled) || is.na(enrolled) || enrolled <= 0) return(NULL)
  rate <- (absent / enrolled) * 100
  if (rate > 100) NULL else rate
}

sapply(crdc, function(x) length(x$absent))
```

## Writing the contract

```{r contract}
NA_VALUE <- "NA"

fmt_int <- function(n) {
  n <- as.numeric(n)
  if (!is.finite(n) || n != round(n)) stop("contract: ", n, " is not an integer")
  sprintf("%.0f", n)
}

fmt_real <- function(x) {
  if (is.null(x) || length(x) == 0 || is.na(x)) return(NA_VALUE)
  if (!is.finite(x)) stop("contract: ", x, " is not finite")
  sprintf("%.12f", if (x == 0) 0 else x)
}

fmt_text <- function(s) {
  if (grepl("[\t\r\n]", s)) stop("contract: value contains a tab or newline: ", s)
  s
}

rows <- new.env(parent = emptyenv())
rows$metric <- character(0); rows$key <- character(0); rows$value <- character(0)
add <- function(metric, key, value) {
  rows$metric <- c(rows$metric, metric)
  rows$key <- c(rows$key, key)
  rows$value <- c(rows$value, value)
}

sha256 <- function(s) digest::digest(s, algo = "sha256", serialize = FALSE)
```

## The facts

```{r facts-input}
for (name in names(csv_sources)) {
  df <- sources[[name]]
  cols <- csv_sources[[name]]$cols
  dump <- do.call(paste, c(lapply(cols, function(c) df[[c]]), sep = "\x1f"))
  add("input.rows", name, fmt_int(nrow(df)))
  add("input.digest", name, fmt_text(sha256(paste(dump, collapse = "\n"))))
}

# The JSON sources are hashed as raw bytes rather than as parsed columns: there
# is no cross-language question about how to read a JSON string, and what each
# implementation makes of the numbers is checked by the facts below.
json_files <- c(
  setNames(sprintf("nyc_attendance_%s.json", nyc_years), sprintf("nyc_%s", nyc_years)),
  setNames(as.vector(outer(crdc_years, crdc_leas, function(y, l) sprintf("crdc_absent_%s_%s.json", y, l))),
           as.vector(outer(crdc_years, crdc_leas, function(y, l) sprintf("crdc_absent_%s_%s", y, l)))),
  setNames(as.vector(outer(crdc_years, crdc_leas, function(y, l) sprintf("crdc_enrolled_%s_%s.json", y, l))),
           as.vector(outer(crdc_years, crdc_leas, function(y, l) sprintf("crdc_enrolled_%s_%s", y, l))))
)
for (name in names(json_files)) {
  add("input.filedigest", name,
      fmt_text(digest::digest(file = file.path(A, json_files[[name]]), algo = "sha256")))
}
```

```{r facts-philadelphia}
series <- list(ada = phl_ada, chronic = phl_chronic, a95 = phl_95, teacherAda = teacher_ada)

for (i in seq_len(nrow(k8))) {
  org <- k8$org[i]
  code <- normalize_ulcs(org)
  for (name in names(series)) {
    v <- series[[name]][[code]]
    if (is.null(v)) next
    for (y in names(v)) add(sprintf("phl.%s", name), paste0(org, "|", y), fmt_real(v[[y]]))
  }
}

for (name in names(series)) {
  if (name == "teacherAda") next
  by_year <- district_medians(series[[name]])
  for (y in names(by_year)) {
    add(sprintf("phlMedian.%s", name), y, fmt_real(med(by_year[[y]])))
    add(sprintf("phlCount.%s", name), y, fmt_int(length(by_year[[y]])))
  }
}
```

```{r facts-newyork}
for (i in seq_len(nrow(nyc_schools))) {
  dbn <- nyc_schools$dbn[i]
  for (y in nyc_years) {
    hit <- match(dbn, nyc_by_year[[y]]$dbn)
    if (is.na(hit)) next
    add("nyc.ada", paste0(dbn, "|", y), fmt_real(nyc_by_year[[y]]$ada[hit]))
    add("nyc.chronic", paste0(dbn, "|", y), fmt_real(nyc_by_year[[y]]$chronic[hit]))
  }
}

for (y in nyc_years) {
  add("nycMedian.ada", y, fmt_real(med(nyc_by_year[[y]]$ada)))
  add("nycMedian.chronic", y, fmt_real(med(nyc_by_year[[y]]$chronic)))
  add("nycMedian.n", y, fmt_int(length(nyc_by_year[[y]]$dbn)))
}
```

```{r facts-federal}
target_id <- c(k8$nces, nyc_schools$nces)
target_city <- c(rep("Philadelphia", nrow(k8)), rep("Brooklyn", nrow(nyc_schools)))
target_label <- c(k8$label, nyc_schools$label)

# SEDA drops Alabama's leading zero from the NCES id; the federal collections
# keep all twelve digits.
cov <- sources$seda_covariates
tusc <- cov[cov$sedalea == tuscaloosa_seda_lea & cov$sedasch != "", , drop = FALSE]
tid <- ifelse(nchar(tusc$sedasch) == 11, paste0("0", tusc$sedasch), tusc$sedasch)
target_id <- c(target_id, tid)
target_city <- c(target_city, rep("Tuscaloosa City", length(tid)))
target_label <- c(target_label, trimws(tusc$schnam))

# A repeated id is one entry, resolved to the last one seen.
last <- !duplicated(target_id, fromLast = TRUE)
target_id <- target_id[last]; target_city <- target_city[last]; target_label <- target_label[last]

for (i in seq_along(target_id)) {
  add("federal.city", target_id[i], fmt_text(gsub(" ", "_", target_city[i])))
  add("federal.label", target_id[i], fmt_text(target_label[i]))
}

city_rates <- list()
for (year in crdc_years) {
  rates_by_city <- list()
  for (i in seq_along(target_id)) {
    rate <- federal_rate(crdc[[year]]$absent[[target_id[i]]], crdc[[year]]$enrolled[[target_id[i]]])
    if (is.null(rate)) next
    add("federal.rate", paste0(year, "|", target_id[i]), fmt_real(rate))
    rates_by_city[[target_city[i]]] <- c(rates_by_city[[target_city[i]]], rate)
  }
  for (city in names(rates_by_city)) {
    key <- paste0(year, "|", gsub(" ", "_", city))
    add("federalMedian.median", key, fmt_real(med(rates_by_city[[city]])))
    add("federalMedian.min", key, fmt_real(min(rates_by_city[[city]])))
    add("federalMedian.max", key, fmt_real(max(rates_by_city[[city]])))
    add("federalMedian.n", key, fmt_int(length(rates_by_city[[city]])))
  }
  city_rates[[year]] <- rates_by_city
}

do.call(rbind, lapply(crdc_years, function(y) data.frame(
  year = y,
  city = names(city_rates[[y]]),
  n = sapply(city_rates[[y]], length),
  median = round(sapply(city_rates[[y]], med), 1),
  row.names = NULL)))
```

## Does a school's absence rate track its teachers' attendance?

The two are measured independently — students by the district's own attendance
system, teachers by payroll — so a correlation between them is not an artefact
of one measure feeding the other.

Pearson's r is written as an explicit loop rather than with `cor()`, so that all
three implementations add their terms up in the same order.

```{r correlation}
pearson <- function(xs, ys) {
  n <- min(length(xs), length(ys))
  if (n < 2) return(NULL)
  sx <- 0; sy <- 0
  for (i in seq_len(n)) { sx <- sx + xs[i]; sy <- sy + ys[i] }
  mx <- sx / n; my <- sy / n
  num <- 0; dx <- 0; dy <- 0
  for (i in seq_len(n)) {
    a <- xs[i] - mx; b <- ys[i] - my
    num <- num + a * b; dx <- dx + a * a; dy <- dy + b * b
  }
  if (dx == 0 || dy == 0) return(NULL)
  num / sqrt(dx * dy)
}

chronics <- numeric(0); teachers <- numeric(0)
for (i in seq_len(nrow(k8))) {
  org <- k8$org[i]
  code <- normalize_ulcs(org)
  ch <- phl_chronic[[code]]["2024-2025"]
  ta <- teacher_ada[[code]]
  # Year keys are compared as strings, exactly as the original does; they are
  # all ASCII digits, so every language orders them the same way.
  tv <- if (is.null(ta)) NULL else med(ta[names(ta) >= "2016"])
  if (is.null(ch) || is.na(ch) || is.null(tv)) next
  chronics <- c(chronics, ch); teachers <- c(teachers, tv)
  add("correlation.chronic", org, fmt_real(unname(ch)))
  add("correlation.teacher", org, fmt_real(tv))
}
r <- pearson(chronics, teachers)
add("correlation.n", "-", fmt_int(length(chronics)))
add("correlation.chronicVsTeacherAttendance", "-", fmt_real(r))

data.frame(measure = "chronic absence vs teacher attendance",
           r = round(r, 3), n = length(chronics))
```

## Output

```{r write}
stopifnot(!any(duplicated(paste0(rows$metric, "\t", rows$key))))
ord <- order(paste0(rows$metric, "\t", rows$key), method = "radix")
lines <- c("metric\tkey\tvalue",
           paste(rows$metric[ord], rows$key[ord], rows$value[ord], sep = "\t"))

out_dir <- file.path(ROOT, "calc", "out")
dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
con <- file(file.path(out_dir, "attendance-r.tsv"), open = "wb")
writeLines(lines, con, sep = "\n", useBytes = TRUE)
close(con)

cat(sprintf("facts: %d\nsha256: %s\n", length(lines) - 1,
            sha256(paste0(paste(lines, collapse = "\n"), "\n"))))
```

The TypeScript and Python implementations print the same hash.

---
title: "School attendance, Philadelphia and Brooklyn — the calculation"
output: md_document
---



Philadelphia and New York both publish school attendance, and they publish
mirrored halves of the same fact: Philadelphia the share of students attending
**at least 90%** of days, New York the share **chronically absent** — missing
10% or more. This document normalizes everything to the second, which is the
federal standard metric, and then reaches for the one source that measures all
three cities on one definition in one year.

It is one of three implementations of this calculation. The others are the
original TypeScript in `src/` and `calc/attendance.py`; all three write the
contract described in `calc/CONTRACT.md` and the three files are compared byte
for byte.


```r
ROOT <- Sys.getenv("SDP_ROOT", ".")
A <- file.path(ROOT, "attendance")

nyc_years <- c("2016-17", "2017-18", "2018-19", "2019-20", "2020-21")
crdc_leas <- c("4218990", "0103360", "3600091", "3600092")
# 2017-18 is the last clean pre-pandemic collection; 2022-23 the most recent.
# The 2020-21 collection covers the remote year and is left out.
crdc_years <- c("2017", "2022")
tuscaloosa_seda_lea <- "103360"

k8 <- data.frame(
  org = c("1300", "1280", "1340", "1260", "1390", "1470", "2380", "2590", "2470", "2480", "2210"),
  label = c("Harrington", "Penn Alexander", "Lea", "Comegys", "Powel", "Alain Locke",
            "Meredith", "Nebinger", "Greenfield", "Marian Anderson", "Bache-Martin"),
  nces = c("421899006528", "421899000619", "421899006530", "421899003782", "421899003783",
           "421899004732", "421899003649", "421899003650", "421899005186", "421899003805",
           "421899003618")
)

nyc_schools <- data.frame(
  dbn = c("13K011", "13K009", "15K321", "15K107", "15K039", "15K051",
          "15K154", "15K010", "15K118", "13K133", "13K282"),
  nces = c("360009102074", "360009102069", "360009202818", "360009202403", "360009202175",
           "360009202226", "360009202557", "360009202072", "360009206406", "360009102490",
           "360009102793"),
  label = c("PS 11 Purvis J. Behan", "PS 9 Teunis G. Bergen", "PS 321 William Penn",
            "PS 107 John W. Kimball", "PS 39 Henry Bristow", "MS 51 William Alexander",
            "PS 154 Windsor Terrace", "PS 10 Magnet School", "PS 118 Maurice Sendak",
            "PS 133 William A. Butler", "PS 282 Park Slope")
)

# Each source, the short name it goes by in the contract, and the columns every
# implementation reads out of it. The digest covers those columns only: hashing
# a column nothing uses would fail the check for noise rather than for signal.
csv_sources <- list(
  ada_archive = list(file = "ex/Student_ADA_Yearly.csv",
                     cols = c("School Year", "ULCS Code", "Average Daily Attendance")),
  ada_current = list(file = "Student_ADA_School.csv",
                     cols = c("School Year", "ULCS Code", "Average Daily Attendance (YTD)")),
  attending90 = list(file = "Student_Attendance_90_School.csv",
                     cols = c("School Year", "ULCS Code", "Category", "% with 90%+ Attendance (Yearly)")),
  attending95_archive = list(file = "ex/SDP_95_Attendance_School_S.csv",
                             cols = c("School Year", "ULCS Code", "Category", "% with 95%+ Attendance (Yearly)")),
  attending95_current = list(file = "Student_Attendance_95_School.csv",
                             cols = c("School Year", "ULCS Code", "Category", "% with 95%+ Attendance (Yearly)")),
  teacher_ada = list(file = "ex/Archive_Teacher_95_ADA_2012-13_2022-23/OD_Teacher_Attendance_Schools_Suppressed_20240328.csv",
                     cols = c("school_year", "ulcscode", "pct_att_ada")),
  seda_covariates = list(file = "../seda/seda_cov_school_pool_4.1.csv",
                         cols = c("sedasch", "sedalea", "schnam"))
)
```

## Reading the files

The Philadelphia files are latin-1, and one of them carries a byte-order mark.
Left in place, that mark becomes part of the first column's name, so every
lookup against that column silently returns nothing — a failure that shows up as
an empty chart rather than as an error. It is stripped from the raw bytes,
before any decoding, which is the only place it is unambiguous.


```r
read_extract <- function(rel) {
  path <- file.path(A, rel)
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-attendance.sh")
  raw <- readBin(path, "raw", file.size(path))
  # UTF-8 byte-order mark: EF BB BF.
  if (length(raw) >= 3 && identical(as.integer(raw[1:3]), c(239L, 187L, 191L))) raw <- raw[-(1:3)]
  txt <- iconv(rawToChar(raw), from = "latin1", to = "UTF-8")
  read.csv(text = txt, colClasses = "character", check.names = FALSE, na.strings = NULL)
}

sources <- lapply(csv_sources, function(s) read_extract(s$file))
sapply(sources, nrow)
#>         ada_archive         ada_current         attending90 attending95_archive 
#>                1503                1074               16926               22944 
#> attending95_current         teacher_ada     seda_covariates 
#>               16926                2407               89350
```

## What a published cell means

Suppressed cells are published as `s`, not as a blank, and a school with too few
students to report is not a school with zero attendance. Anything that is not a
number is dropped rather than coerced.


```r
parse_value <- function(v) {
  t <- trimws(v)
  out <- rep(NA_real_, length(t))
  ok <- !(t == "" | tolower(t) == "s" | tolower(t) == "n/a")
  out[ok] <- suppressWarnings(as.numeric(t[ok]))
  out[!is.finite(out)] <- NA_real_
  out
}

# SDP school codes appear both zero-padded and unpadded across files.
normalize_ulcs <- function(code) {
  t <- sub("^0+", "", trimws(code))
  ifelse(t == "", "0", t)
}
```

## One series per school

Every Philadelphia file is in long format — one row per school per year, and in
some files per demographic group as well. They collapse to the same shape: a
school, a year, a number. Where a school-year appears twice, the last row in the
file wins, which is what an assignment does in the other two implementations.


```r
series_from_rows <- function(df, year_col, code_col, value_col, keep = NULL) {
  if (!is.null(keep)) df <- df[keep(df), , drop = FALSE]
  v <- parse_value(df[[value_col]])
  code <- normalize_ulcs(df[[code_col]])
  year <- trimws(df[[year_col]])
  ok <- !is.na(v) & code != "" & year != ""
  v <- v[ok]; code <- code[ok]; year <- year[ok]

  out <- list()
  for (cd in split(seq_along(code), code)) {
    yrs <- year[cd]
    vals <- v[cd]
    last <- !duplicated(yrs, fromLast = TRUE)
    out[[code[cd[1]]]] <- setNames(vals[last], yrs[last])
  }
  out
}

all_students <- function(df) df$Category == "All Students"

# Two releases of the same series; the newer file wins year by year.
merge_series <- function(old, new) {
  out <- old
  for (cd in names(new)) {
    have <- out[[cd]]
    add <- new[[cd]]
    if (is.null(have)) out[[cd]] <- add
    else out[[cd]] <- c(add, have[setdiff(names(have), names(add))])
  }
  out
}
```


```r
phl_ada <- merge_series(
  series_from_rows(sources$ada_archive, "School Year", "ULCS Code", "Average Daily Attendance"),
  series_from_rows(sources$ada_current, "School Year", "ULCS Code", "Average Daily Attendance (YTD)")
)

attending90 <- series_from_rows(sources$attending90, "School Year", "ULCS Code",
                                "% with 90%+ Attendance (Yearly)", all_students)
# Philadelphia publishes the attending share; the standard metric is its complement.
phl_chronic <- lapply(attending90, function(y) 100 - y)

phl_95 <- merge_series(
  series_from_rows(sources$attending95_archive, "School Year", "ULCS Code",
                   "% with 95%+ Attendance (Yearly)", all_students),
  series_from_rows(sources$attending95_current, "School Year", "ULCS Code",
                   "% with 95%+ Attendance (Yearly)", all_students)
)

teacher_ada <- series_from_rows(sources$teacher_ada, "school_year", "ulcscode", "pct_att_ada")

length(phl_ada)
#> [1] 227
```

## Reference lines

A school's absence rate means little without knowing what the district looked
like that year, so each measure also gets a districtwide median across every
school SDP reports — not across the eleven on the page.


```r
med <- function(v) {
  v <- v[!is.na(v)]
  if (!length(v)) return(NULL)
  s <- sort(v)
  mid <- length(s) %/% 2
  if (length(s) %% 2 == 1) s[mid + 1] else (s[mid] + s[mid + 1]) / 2
}

district_medians <- function(series) {
  years <- unlist(lapply(series, names), use.names = FALSE)
  vals <- unlist(series, use.names = FALSE)
  split(vals, years)
}
```

## New York

New York publishes one row per school per year, already as a chronic-absence
percentage, so nothing has to be converted — only read.


```r
nyc_raw <- lapply(nyc_years, function(y) {
  path <- file.path(A, sprintf("nyc_attendance_%s.json", y))
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-attendance.sh")
  jsonlite::fromJSON(path, simplifyDataFrame = TRUE)
})
names(nyc_raw) <- nyc_years

nyc_by_year <- lapply(nyc_raw, function(df) {
  ada <- parse_value(df$attendance)
  chronic <- parse_value(df$chronically_absent_1)
  ok <- !is.na(df$dbn) & !is.na(ada) & !is.na(chronic)
  keep <- ok & !duplicated(df$dbn, fromLast = TRUE)
  list(dbn = df$dbn[keep], ada = ada[keep], chronic = chronic[keep])
})

sapply(nyc_by_year, function(x) length(x$dbn))
#> 2016-17 2017-18 2018-19 2019-20 2020-21 
#>    1546    1539    1526    1527    1530
```

## One year, three cities, one definition

Philadelphia, New York and Tuscaloosa each publish attendance their own way,
over different windows, so their own figures cannot be set side by side. The
federal Civil Rights Data Collection asks every district for the same count in
the same year, which makes it the one apples-to-apples measure available.

It publishes counts rather than rates, so enrolment comes from the Common Core
of Data as the denominator — and it returns **two rows per school** even with
every other subgroup pinned to "all": one for all students and one for the
homeless subgroup. Keeping the wrong one produces single-digit rates that look
entirely plausible.


```r
read_results <- function(rel) {
  path <- file.path(A, rel)
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-crdc.sh")
  jsonlite::fromJSON(path, simplifyDataFrame = TRUE)$results
}

read_crdc_year <- function(year) {
  absent <- list(); enrolled <- list()
  for (lea in crdc_leas) {
    a <- read_results(sprintf("crdc_absent_%s_%s.json", year, lea))
    if (!is.null(a) && nrow(a)) {
      keep <- (is.na(a$homeless) | a$homeless == 99) & !is.na(a$ncessch) &
        !is.na(a$students_chronically_absent)
      a <- a[keep, , drop = FALSE]
      last <- !duplicated(a$ncessch, fromLast = TRUE)
      absent[a$ncessch[last]] <- a$students_chronically_absent[last]
    }
    e <- read_results(sprintf("crdc_enrolled_%s_%s.json", year, lea))
    if (!is.null(e) && nrow(e)) {
      e <- e[!is.na(e$ncessch) & !is.na(e$enrollment), , drop = FALSE]
      last <- !duplicated(e$ncessch, fromLast = TRUE)
      enrolled[e$ncessch[last]] <- e$enrollment[last]
    }
  }
  list(absent = absent, enrolled = enrolled)
}

crdc <- lapply(crdc_years, read_crdc_year)
names(crdc) <- crdc_years

# CRDC encodes suppressed or not-applicable cells as negative numbers (-3, -5,
# -9). Those must not be divided; nor may a rate above 100 be reported, which
# would mean the count and the denominator came from mismatched records.
federal_rate <- function(absent, enrolled) {
  if (is.null(absent) || is.na(absent) || absent < 0) return(NULL)
  if (is.null(enrolled) || is.na(enrolled) || enrolled <= 0) return(NULL)
  rate <- (absent / enrolled) * 100
  if (rate > 100) NULL else rate
}

sapply(crdc, function(x) length(x$absent))
#> 2017 2022 
#>  327  327
```

## Writing the contract


```r
NA_VALUE <- "NA"

fmt_int <- function(n) {
  n <- as.numeric(n)
  if (!is.finite(n) || n != round(n)) stop("contract: ", n, " is not an integer")
  sprintf("%.0f", n)
}

fmt_real <- function(x) {
  if (is.null(x) || length(x) == 0 || is.na(x)) return(NA_VALUE)
  if (!is.finite(x)) stop("contract: ", x, " is not finite")
  sprintf("%.12f", if (x == 0) 0 else x)
}

fmt_text <- function(s) {
  if (grepl("[\t\r\n]", s)) stop("contract: value contains a tab or newline: ", s)
  s
}

rows <- new.env(parent = emptyenv())
rows$metric <- character(0); rows$key <- character(0); rows$value <- character(0)
add <- function(metric, key, value) {
  rows$metric <- c(rows$metric, metric)
  rows$key <- c(rows$key, key)
  rows$value <- c(rows$value, value)
}

sha256 <- function(s) digest::digest(s, algo = "sha256", serialize = FALSE)
```

## The facts


```r
for (name in names(csv_sources)) {
  df <- sources[[name]]
  cols <- csv_sources[[name]]$cols
  dump <- do.call(paste, c(lapply(cols, function(c) df[[c]]), sep = "\x1f"))
  add("input.rows", name, fmt_int(nrow(df)))
  add("input.digest", name, fmt_text(sha256(paste(dump, collapse = "\n"))))
}

# The JSON sources are hashed as raw bytes rather than as parsed columns: there
# is no cross-language question about how to read a JSON string, and what each
# implementation makes of the numbers is checked by the facts below.
json_files <- c(
  setNames(sprintf("nyc_attendance_%s.json", nyc_years), sprintf("nyc_%s", nyc_years)),
  setNames(as.vector(outer(crdc_years, crdc_leas, function(y, l) sprintf("crdc_absent_%s_%s.json", y, l))),
           as.vector(outer(crdc_years, crdc_leas, function(y, l) sprintf("crdc_absent_%s_%s", y, l)))),
  setNames(as.vector(outer(crdc_years, crdc_leas, function(y, l) sprintf("crdc_enrolled_%s_%s.json", y, l))),
           as.vector(outer(crdc_years, crdc_leas, function(y, l) sprintf("crdc_enrolled_%s_%s", y, l))))
)
for (name in names(json_files)) {
  add("input.filedigest", name,
      fmt_text(digest::digest(file = file.path(A, json_files[[name]]), algo = "sha256")))
}
```


```r
series <- list(ada = phl_ada, chronic = phl_chronic, a95 = phl_95, teacherAda = teacher_ada)

for (i in seq_len(nrow(k8))) {
  org <- k8$org[i]
  code <- normalize_ulcs(org)
  for (name in names(series)) {
    v <- series[[name]][[code]]
    if (is.null(v)) next
    for (y in names(v)) add(sprintf("phl.%s", name), paste0(org, "|", y), fmt_real(v[[y]]))
  }
}

for (name in names(series)) {
  if (name == "teacherAda") next
  by_year <- district_medians(series[[name]])
  for (y in names(by_year)) {
    add(sprintf("phlMedian.%s", name), y, fmt_real(med(by_year[[y]])))
    add(sprintf("phlCount.%s", name), y, fmt_int(length(by_year[[y]])))
  }
}
```


```r
for (i in seq_len(nrow(nyc_schools))) {
  dbn <- nyc_schools$dbn[i]
  for (y in nyc_years) {
    hit <- match(dbn, nyc_by_year[[y]]$dbn)
    if (is.na(hit)) next
    add("nyc.ada", paste0(dbn, "|", y), fmt_real(nyc_by_year[[y]]$ada[hit]))
    add("nyc.chronic", paste0(dbn, "|", y), fmt_real(nyc_by_year[[y]]$chronic[hit]))
  }
}

for (y in nyc_years) {
  add("nycMedian.ada", y, fmt_real(med(nyc_by_year[[y]]$ada)))
  add("nycMedian.chronic", y, fmt_real(med(nyc_by_year[[y]]$chronic)))
  add("nycMedian.n", y, fmt_int(length(nyc_by_year[[y]]$dbn)))
}
```


```r
target_id <- c(k8$nces, nyc_schools$nces)
target_city <- c(rep("Philadelphia", nrow(k8)), rep("Brooklyn", nrow(nyc_schools)))
target_label <- c(k8$label, nyc_schools$label)

# SEDA drops Alabama's leading zero from the NCES id; the federal collections
# keep all twelve digits.
cov <- sources$seda_covariates
tusc <- cov[cov$sedalea == tuscaloosa_seda_lea & cov$sedasch != "", , drop = FALSE]
tid <- ifelse(nchar(tusc$sedasch) == 11, paste0("0", tusc$sedasch), tusc$sedasch)
target_id <- c(target_id, tid)
target_city <- c(target_city, rep("Tuscaloosa City", length(tid)))
target_label <- c(target_label, trimws(tusc$schnam))

# A repeated id is one entry, resolved to the last one seen.
last <- !duplicated(target_id, fromLast = TRUE)
target_id <- target_id[last]; target_city <- target_city[last]; target_label <- target_label[last]

for (i in seq_along(target_id)) {
  add("federal.city", target_id[i], fmt_text(gsub(" ", "_", target_city[i])))
  add("federal.label", target_id[i], fmt_text(target_label[i]))
}

city_rates <- list()
for (year in crdc_years) {
  rates_by_city <- list()
  for (i in seq_along(target_id)) {
    rate <- federal_rate(crdc[[year]]$absent[[target_id[i]]], crdc[[year]]$enrolled[[target_id[i]]])
    if (is.null(rate)) next
    add("federal.rate", paste0(year, "|", target_id[i]), fmt_real(rate))
    rates_by_city[[target_city[i]]] <- c(rates_by_city[[target_city[i]]], rate)
  }
  for (city in names(rates_by_city)) {
    key <- paste0(year, "|", gsub(" ", "_", city))
    add("federalMedian.median", key, fmt_real(med(rates_by_city[[city]])))
    add("federalMedian.min", key, fmt_real(min(rates_by_city[[city]])))
    add("federalMedian.max", key, fmt_real(max(rates_by_city[[city]])))
    add("federalMedian.n", key, fmt_int(length(rates_by_city[[city]])))
  }
  city_rates[[year]] <- rates_by_city
}

do.call(rbind, lapply(crdc_years, function(y) data.frame(
  year = y,
  city = names(city_rates[[y]]),
  n = sapply(city_rates[[y]], length),
  median = round(sapply(city_rates[[y]], med), 1),
  row.names = NULL)))
#>   year            city  n median
#> 1 2017    Philadelphia 11   17.4
#> 2 2017        Brooklyn 11    7.5
#> 3 2017 Tuscaloosa City 19   10.2
#> 4 2022    Philadelphia 11   21.5
#> 5 2022        Brooklyn 11   19.0
#> 6 2022 Tuscaloosa City 17   20.3
```

## Does a school's absence rate track its teachers' attendance?

The two are measured independently — students by the district's own attendance
system, teachers by payroll — so a correlation between them is not an artefact
of one measure feeding the other.

Pearson's r is written as an explicit loop rather than with `cor()`, so that all
three implementations add their terms up in the same order.


```r
pearson <- function(xs, ys) {
  n <- min(length(xs), length(ys))
  if (n < 2) return(NULL)
  sx <- 0; sy <- 0
  for (i in seq_len(n)) { sx <- sx + xs[i]; sy <- sy + ys[i] }
  mx <- sx / n; my <- sy / n
  num <- 0; dx <- 0; dy <- 0
  for (i in seq_len(n)) {
    a <- xs[i] - mx; b <- ys[i] - my
    num <- num + a * b; dx <- dx + a * a; dy <- dy + b * b
  }
  if (dx == 0 || dy == 0) return(NULL)
  num / sqrt(dx * dy)
}

chronics <- numeric(0); teachers <- numeric(0)
for (i in seq_len(nrow(k8))) {
  org <- k8$org[i]
  code <- normalize_ulcs(org)
  ch <- phl_chronic[[code]]["2024-2025"]
  ta <- teacher_ada[[code]]
  # Year keys are compared as strings, exactly as the original does; they are
  # all ASCII digits, so every language orders them the same way.
  tv <- if (is.null(ta)) NULL else med(ta[names(ta) >= "2016"])
  if (is.null(ch) || is.na(ch) || is.null(tv)) next
  chronics <- c(chronics, ch); teachers <- c(teachers, tv)
  add("correlation.chronic", org, fmt_real(unname(ch)))
  add("correlation.teacher", org, fmt_real(tv))
}
r <- pearson(chronics, teachers)
add("correlation.n", "-", fmt_int(length(chronics)))
add("correlation.chronicVsTeacherAttendance", "-", fmt_real(r))

data.frame(measure = "chronic absence vs teacher attendance",
           r = round(r, 3), n = length(chronics))
#>                                         measure    r  n
#> 2024-2025 chronic absence vs teacher attendance -0.7 11
```

## Output


```r
stopifnot(!any(duplicated(paste0(rows$metric, "\t", rows$key))))
ord <- order(paste0(rows$metric, "\t", rows$key), method = "radix")
lines <- c("metric\tkey\tvalue",
           paste(rows$metric[ord], rows$key[ord], rows$value[ord], sep = "\t"))

out_dir <- file.path(ROOT, "calc", "out")
dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
con <- file(file.path(out_dir, "attendance-r.tsv"), open = "wb")
writeLines(lines, con, sep = "\n", useBytes = TRUE)
close(con)

cat(sprintf("facts: %d\nsha256: %s\n", length(lines) - 1,
            sha256(paste0(paste(lines, collapse = "\n"), "\n"))))
#> facts: 868
#> sha256: 01130b46ec66dce7bc27ed085a5f3e6b94afe4ee5ca1a854b841d0883e36a174
```

The TypeScript and Python implementations print the same hash.

#!/usr/bin/env python3
"""
The Python implementation of the attendance calculation.

Philadelphia and New York both publish school attendance, and they publish
mirrored forms of the same fact: Philadelphia the share of students attending at
least 90% of days, New York the share chronically absent. Everything here is
normalized to the second, which is the federal standard metric. The federal
Civil Rights Data Collection then supplies the one year in which all three
cities — Philadelphia, Brooklyn and Tuscaloosa City — are measured the same way.

Second of three implementations; see calc/CONTRACT.md. Standard library only.

Run: npm run calc:py   (writes calc/out/attendance-python.tsv)
"""

from __future__ import annotations

import csv
import io
import json
import math
import os

from contract import NA, fmt_int, fmt_real, fmt_text, serialize, sha256

HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
A = os.path.join(ROOT, "attendance")

NYC_YEARS = ["2016-17", "2017-18", "2018-19", "2019-20", "2020-21"]
CRDC_LEAS = ["4218990", "0103360", "3600091", "3600092"]
# 2017-18 is the last clean pre-pandemic collection; 2022-23 the most recent.
CRDC_YEARS = ["2017", "2022"]
TUSCALOOSA_SEDA_LEA = "103360"

K8 = [
    ("1300", "Harrington", "421899006528"),
    ("1280", "Penn Alexander", "421899000619"),
    ("1340", "Lea", "421899006530"),
    ("1260", "Comegys", "421899003782"),
    ("1390", "Powel", "421899003783"),
    ("1470", "Alain Locke", "421899004732"),
    ("2380", "Meredith", "421899003649"),
    ("2590", "Nebinger", "421899003650"),
    ("2470", "Greenfield", "421899005186"),
    ("2480", "Marian Anderson", "421899003805"),
    ("2210", "Bache-Martin", "421899003618"),
]

NYC = [
    ("13K011", "360009102074", "PS 11 Purvis J. Behan"),
    ("13K009", "360009102069", "PS 9 Teunis G. Bergen"),
    ("15K321", "360009202818", "PS 321 William Penn"),
    ("15K107", "360009202403", "PS 107 John W. Kimball"),
    ("15K039", "360009202175", "PS 39 Henry Bristow"),
    ("15K051", "360009202226", "MS 51 William Alexander"),
    ("15K154", "360009202557", "PS 154 Windsor Terrace"),
    ("15K010", "360009202072", "PS 10 Magnet School"),
    ("15K118", "360009206406", "PS 118 Maurice Sendak"),
    ("13K133", "360009102490", "PS 133 William A. Butler"),
    ("13K282", "360009102793", "PS 282 Park Slope"),
]

# name -> (path, columns every implementation reads out of it)
CSV_SOURCES = [
    ("ada_archive", "ex/Student_ADA_Yearly.csv",
     ["School Year", "ULCS Code", "Average Daily Attendance"]),
    ("ada_current", "Student_ADA_School.csv",
     ["School Year", "ULCS Code", "Average Daily Attendance (YTD)"]),
    ("attending90", "Student_Attendance_90_School.csv",
     ["School Year", "ULCS Code", "Category", "% with 90%+ Attendance (Yearly)"]),
    ("attending95_archive", "ex/SDP_95_Attendance_School_S.csv",
     ["School Year", "ULCS Code", "Category", "% with 95%+ Attendance (Yearly)"]),
    ("attending95_current", "Student_Attendance_95_School.csv",
     ["School Year", "ULCS Code", "Category", "% with 95%+ Attendance (Yearly)"]),
    ("teacher_ada", "ex/Archive_Teacher_95_ADA_2012-13_2022-23/OD_Teacher_Attendance_Schools_Suppressed_20240328.csv",
     ["school_year", "ulcscode", "pct_att_ada"]),
    ("seda_covariates", "../seda/seda_cov_school_pool_4.1.csv",
     ["sedasch", "sedalea", "schnam"]),
]


# ----------------------------------------------------------------- reading

def read_csv(rel):
    """
    Read one attendance CSV.

    The files are latin-1, and one of them carries a byte-order mark. Left in
    place the mark becomes part of the first column's name, so every lookup on
    that column silently returns nothing — a failure that produces an empty
    chart rather than an error.
    """
    path = os.path.join(A, rel)
    if not os.path.exists(path):
        raise SystemExit("missing %s — run ./fetch-attendance.sh" % path)
    with open(path, encoding="latin-1", newline="") as fh:
        text = fh.read()
    if text.startswith(""):
        text = text[1:]
    elif text.startswith(""):
        text = text[3:]
    return list(csv.DictReader(io.StringIO(text)))


def read_json(rel):
    path = os.path.join(A, rel)
    if not os.path.exists(path):
        raise SystemExit("missing %s — run ./fetch-attendance.sh and ./fetch-crdc.sh" % path)
    with open(path, encoding="utf-8") as fh:
        return fh.read()


# ------------------------------------------------------------- definitions

def normalize_ulcs(code):
    """SDP school codes appear both zero-padded and unpadded across files."""
    t = str(code).strip().lstrip("0")
    return "0" if t == "" else t


def parse_value(v):
    """A published cell as a number. Suppressed cells are written as 's'."""
    if v is None:
        return None
    t = v.strip()
    if t == "" or t.lower() == "s" or t.lower() == "n/a":
        return None
    try:
        n = float(t)
    except ValueError:
        return None
    return n if math.isfinite(n) else None


def series_from_rows(rows, year_key, code_key, value_key, keep=lambda r: True):
    """Collapse long-format rows into school -> year -> value."""
    out = {}
    for r in rows:
        if not keep(r):
            continue
        v = parse_value(r.get(value_key))
        if v is None:
            continue
        code = normalize_ulcs(r.get(code_key) or "")
        year = (r.get(year_key) or "").strip()
        if code == "" or year == "":
            continue
        out.setdefault(code, {})[year] = v
    return out


def all_students(r):
    return r.get("Category") == "All Students"


def chronic_from_attending90(pct):
    """Philadelphia publishes the attending share; the standard metric is its complement."""
    return 100 - pct


def federal_rate(absent, enrolled):
    """
    A CRDC absentee count over its CCD enrolment, as a percentage.

    CRDC encodes suppressed or not-applicable cells as negative numbers
    (-3, -5, -9). Those must not be divided; nor may a rate above 100 be
    reported, which would mean the count and the denominator came from
    mismatched records.
    """
    if absent is None or absent < 0:
        return None
    if enrolled is None or enrolled <= 0:
        return None
    rate = (absent / enrolled) * 100
    return None if rate > 100 else rate


def median(values):
    if not values:
        return None
    s = sorted(values)
    mid = len(s) // 2
    return s[mid] if len(s) % 2 == 1 else (s[mid - 1] + s[mid]) / 2


def pearson(xs, ys):
    """Accumulated left to right, so all three implementations land on the same bits."""
    n = min(len(xs), len(ys))
    if n < 2:
        return None
    sx = sy = 0.0
    for i in range(n):
        sx += xs[i]
        sy += ys[i]
    mx, my = sx / n, sy / n
    num = dx = dy = 0.0
    for i in range(n):
        a, b = xs[i] - mx, ys[i] - my
        num += a * b
        dx += a * a
        dy += b * b
    if dx == 0 or dy == 0:
        return None
    return num / math.sqrt(dx * dy)


# --------------------------------------------------------------- the report

def merge(old, new):
    """Two releases of the same series; the newer file wins year by year."""
    out = {k: dict(v) for k, v in old.items()}
    for code, years in new.items():
        out.setdefault(code, {}).update(years)
    return out


def build():
    rows = []

    def add(metric, key, value):
        rows.append((metric, key, value))

    # ------------------------------------------------------------------ inputs
    parsed = {}
    for name, rel, columns in CSV_SOURCES:
        parsed[name] = read_csv(rel)
        add("input.rows", name, fmt_int(len(parsed[name])))
        dump = "\n".join("\x1f".join((r.get(c) or "") for c in columns) for r in parsed[name])
        add("input.digest", name, fmt_text(sha256(dump)))

    # The JSON sources are hashed as raw bytes rather than as parsed columns:
    # there is no cross-language question about how to read a JSON string, and
    # what each implementation makes of the numbers is checked by the facts below.
    json_files = [("nyc_%s" % y, "nyc_attendance_%s.json" % y) for y in NYC_YEARS]
    for year in CRDC_YEARS:
        for lea in CRDC_LEAS:
            json_files.append(("crdc_absent_%s_%s" % (year, lea), "crdc_absent_%s_%s.json" % (year, lea)))
            json_files.append(("crdc_enrolled_%s_%s" % (year, lea), "crdc_enrolled_%s_%s.json" % (year, lea)))
    raw_json = {}
    for name, rel in json_files:
        raw_json[name] = read_json(rel)
        add("input.filedigest", name, fmt_text(sha256(raw_json[name])))

    # ------------------------------------------------------------ Philadelphia
    phl_ada = merge(
        series_from_rows(parsed["ada_archive"], "School Year", "ULCS Code", "Average Daily Attendance"),
        series_from_rows(parsed["ada_current"], "School Year", "ULCS Code", "Average Daily Attendance (YTD)"),
    )
    attending90 = series_from_rows(
        parsed["attending90"], "School Year", "ULCS Code", "% with 90%+ Attendance (Yearly)", all_students)
    phl_chronic = {code: {y: chronic_from_attending90(v) for y, v in years.items()}
                   for code, years in attending90.items()}
    phl_95 = merge(
        series_from_rows(parsed["attending95_archive"], "School Year", "ULCS Code",
                         "% with 95%+ Attendance (Yearly)", all_students),
        series_from_rows(parsed["attending95_current"], "School Year", "ULCS Code",
                         "% with 95%+ Attendance (Yearly)", all_students),
    )
    teacher_ada = series_from_rows(parsed["teacher_ada"], "school_year", "ulcscode", "pct_att_ada")

    series = [("ada", phl_ada), ("chronic", phl_chronic), ("a95", phl_95), ("teacherAda", teacher_ada)]

    for org, _label, _nces in K8:
        code = normalize_ulcs(org)
        for name, src in series:
            for year, value in src.get(code, {}).items():
                add("phl.%s" % name, "%s|%s" % (org, year), fmt_real(value))

    for name, src in series:
        if name == "teacherAda":
            continue
        by_year = {}
        for years in src.values():
            for y, v in years.items():
                by_year.setdefault(y, []).append(v)
        for year, values in by_year.items():
            add("phlMedian.%s" % name, year, fmt_real(median(values)))
            add("phlCount.%s" % name, year, fmt_int(len(values)))

    # ---------------------------------------------------------------- New York
    nyc_by_year = {}
    for y in NYC_YEARS:
        by_dbn = {}
        for r in json.loads(raw_json["nyc_%s" % y]):
            ada = parse_value(r.get("attendance"))
            chronic = parse_value(r.get("chronically_absent_1"))
            if r.get("dbn") is None or ada is None or chronic is None:
                continue
            by_dbn[r["dbn"]] = (ada, chronic)
        nyc_by_year[y] = by_dbn

    for dbn, _nces, _label in NYC:
        for y in NYC_YEARS:
            hit = nyc_by_year[y].get(dbn)
            if hit is None:
                continue
            add("nyc.ada", "%s|%s" % (dbn, y), fmt_real(hit[0]))
            add("nyc.chronic", "%s|%s" % (dbn, y), fmt_real(hit[1]))

    for y in NYC_YEARS:
        all_rows = list(nyc_by_year[y].values())
        add("nycMedian.ada", y, fmt_real(median([r[0] for r in all_rows])))
        add("nycMedian.chronic", y, fmt_real(median([r[1] for r in all_rows])))
        add("nycMedian.n", y, fmt_int(len(all_rows)))

    # ----------------------------------------------- federal, all three cities
    targets = {}
    for org, label, nces in K8:
        targets[nces] = ("Philadelphia", label)
    for dbn, nces, label in NYC:
        targets[nces] = ("Brooklyn", label)
    # SEDA drops Alabama's leading zero from the NCES id; CRDC keeps all twelve.
    for r in parsed["seda_covariates"]:
        if r.get("sedalea") != TUSCALOOSA_SEDA_LEA:
            continue
        sid = r.get("sedasch")
        if not sid:
            continue
        targets["0" + sid if len(sid) == 11 else sid] = ("Tuscaloosa City", (r.get("schnam") or sid).strip())

    for nces, (city, label) in targets.items():
        add("federal.city", nces, fmt_text(city.replace(" ", "_")))
        add("federal.label", nces, fmt_text(label))

    for year in CRDC_YEARS:
        crdc_absent, crdc_enrolled = {}, {}
        for lea in CRDC_LEAS:
            for r in json.loads(raw_json["crdc_absent_%s_%s" % (year, lea)]).get("results", []):
                # CRDC returns two rows per school even with every other
                # subgroup pinned to 99: one for all students and one for the
                # homeless subgroup. Without this filter the homeless count
                # silently becomes the numerator.
                if r.get("homeless") is not None and r.get("homeless") != 99:
                    continue
                if r.get("ncessch") is not None and isinstance(r.get("students_chronically_absent"), (int, float)):
                    crdc_absent[r["ncessch"]] = r["students_chronically_absent"]
            for r in json.loads(raw_json["crdc_enrolled_%s_%s" % (year, lea)]).get("results", []):
                if r.get("ncessch") is not None and isinstance(r.get("enrollment"), (int, float)):
                    crdc_enrolled[r["ncessch"]] = r["enrollment"]

        by_city = {}
        for nces, (city, label) in targets.items():
            rate = federal_rate(crdc_absent.get(nces), crdc_enrolled.get(nces))
            if rate is None:
                continue
            add("federal.rate", "%s|%s" % (year, nces), fmt_real(rate))
            by_city.setdefault(city, []).append(rate)

        for city, rates in by_city.items():
            key = "%s|%s" % (year, city.replace(" ", "_"))
            add("federalMedian.median", key, fmt_real(median(rates)))
            add("federalMedian.min", key, fmt_real(min(rates)))
            add("federalMedian.max", key, fmt_real(max(rates)))
            add("federalMedian.n", key, fmt_int(len(rates)))

    # ------------------------------------------------------------ correlation
    chronics, teachers = [], []
    for org, _label, _nces in K8:
        code = normalize_ulcs(org)
        chronic = phl_chronic.get(code, {}).get("2024-2025")
        # Year keys are compared as strings, exactly as the original does; they
        # are all ASCII digits, so every language orders them the same way.
        teacher = median([v for y, v in teacher_ada.get(code, {}).items() if y >= "2016"])
        if chronic is None or teacher is None:
            continue
        chronics.append(chronic)
        teachers.append(teacher)
        add("correlation.chronic", org, fmt_real(chronic))
        add("correlation.teacher", org, fmt_real(teacher))
    add("correlation.n", "-", fmt_int(len(chronics)))
    add("correlation.chronicVsTeacherAttendance", "-", fmt_real(pearson(chronics, teachers)))

    return rows


def main():
    out_dir = os.path.join(ROOT, "calc", "out")
    os.makedirs(out_dir, exist_ok=True)
    tsv = serialize(build())
    with open(os.path.join(out_dir, "attendance-python.tsv"), "w", encoding="utf-8", newline="\n") as fh:
        fh.write(tsv)
    print("wrote calc/out/attendance-python.tsv (%d facts)" % (tsv.count("\n") - 1))


if __name__ == "__main__":
    main()

/**
 * The TypeScript implementation's side of the attendance contract.
 *
 * Like `retention-emit.ts`, this computes nothing of its own: every figure comes
 * from the `src/` modules the published attendance page is built from, written
 * out in the shared format so the R Markdown and Python implementations have
 * something to be compared against.
 *
 * Run: npm run calc:ts   (writes calc/out/attendance-typescript.tsv)
 */

import { createHash } from 'node:crypto'
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'

import { int, real, serialize, text, type Row } from '../src/contract.ts'
import { parseCsv } from '../src/csv.ts'
import {
  chronicFromAttending90,
  chronicFromNyc,
  federalRate,
  normalizeUlcs,
  seriesFromRows,
  type NycRow,
  type YearValue,
} from '../src/attendance.ts'
import { K8_SCHOOLS, NYC_SCHOOLS } from '../src/schools.ts'
import { median, pearson } from '../src/stats.ts'

const ROOT = join(import.meta.dirname, '..')
const A = join(ROOT, 'attendance')

const sha256 = (s: string): string => createHash('sha256').update(s, 'utf8').digest('hex')

/** Cities become contract keys, so their spaces have to go. */
const cityToken = (city: string): string => city.replace(/ /g, '_')

/**
 * Sources, with the short name each is known by in the contract and the columns
 * every implementation reads out of it.
 *
 * The digest is over those columns only. Hashing the whole row would make the
 * check fail for a column nothing uses, which is noise rather than signal.
 */
const CSV_SOURCES: { name: string; file: string; columns: string[] }[] = [
  { name: 'ada_archive', file: 'ex/Student_ADA_Yearly.csv', columns: ['School Year', 'ULCS Code', 'Average Daily Attendance'] },
  { name: 'ada_current', file: 'Student_ADA_School.csv', columns: ['School Year', 'ULCS Code', 'Average Daily Attendance (YTD)'] },
  { name: 'attending90', file: 'Student_Attendance_90_School.csv', columns: ['School Year', 'ULCS Code', 'Category', '% with 90%+ Attendance (Yearly)'] },
  { name: 'attending95_archive', file: 'ex/SDP_95_Attendance_School_S.csv', columns: ['School Year', 'ULCS Code', 'Category', '% with 95%+ Attendance (Yearly)'] },
  { name: 'attending95_current', file: 'Student_Attendance_95_School.csv', columns: ['School Year', 'ULCS Code', 'Category', '% with 95%+ Attendance (Yearly)'] },
  {
    name: 'teacher_ada',
    file: 'ex/Archive_Teacher_95_ADA_2012-13_2022-23/OD_Teacher_Attendance_Schools_Suppressed_20240328.csv',
    columns: ['school_year', 'ulcscode', 'pct_att_ada'],
  },
  { name: 'seda_covariates', file: '../seda/seda_cov_school_pool_4.1.csv', columns: ['sedasch', 'sedalea', 'schnam'] },
]

const NYC_YEARS = ['2016-17', '2017-18', '2018-19', '2019-20', '2020-21']
const CRDC_LEAS = ['4218990', '0103360', '3600091', '3600092']
/** 2017-18 is the last clean pre-pandemic collection; 2022-23 the most recent. */
const CRDC_YEARS = ['2017', '2022']
const TUSCALOOSA_SEDA_LEA = '103360'

const csv = (rel: string): Record<string, string>[] => {
  const path = join(A, rel)
  if (!existsSync(path)) throw new Error(`missing ${path} — run ./fetch-attendance.sh`)
  return parseCsv(readFileSync(path, 'latin1'))
}

const allStudents = (r: Record<string, string>): boolean => r['Category'] === 'All Students'

/** Districtwide reference line for one measure: the median across every school SDP reports. */
function districtMedians(src: Map<string, YearValue>): Map<string, number[]> {
  const byYear = new Map<string, number[]>()
  for (const years of src.values()) {
    for (const [y, v] of Object.entries(years)) {
      const bucket = byYear.get(y)
      if (bucket) bucket.push(v)
      else byYear.set(y, [v])
    }
  }
  return byYear
}

export function emitRows(): Row[] {
  const rows: Row[] = []
  const add = (metric: string, key: string, value: string): void => {
    rows.push({ metric, key, value })
  }

  // ------------------------------------------------------------------ inputs
  for (const src of CSV_SOURCES) {
    const parsed = csv(src.file)
    add('input.rows', src.name, int(parsed.length))
    add('input.digest', src.name, text(sha256(parsed.map((r) => src.columns.map((c) => r[c] ?? '').join('\x1f')).join('\n'))))
  }

  // The JSON sources are hashed as raw bytes rather than as parsed columns:
  // there is no cross-language question about how to read a JSON string, and
  // what each implementation makes of the numbers is checked by the facts below.
  const jsonFiles = [
    ...NYC_YEARS.map((y) => ({ name: `nyc_${y}`, file: `nyc_attendance_${y}.json` })),
    ...CRDC_YEARS.flatMap((y) =>
      CRDC_LEAS.flatMap((lea) => [
        { name: `crdc_absent_${y}_${lea}`, file: `crdc_absent_${y}_${lea}.json` },
        { name: `crdc_enrolled_${y}_${lea}`, file: `crdc_enrolled_${y}_${lea}.json` },
      ]),
    ),
  ]
  for (const j of jsonFiles) {
    const path = join(A, j.file)
    if (!existsSync(path)) throw new Error(`missing ${path} — run ./fetch-attendance.sh and ./fetch-crdc.sh`)
    add('input.filedigest', j.name, text(sha256(readFileSync(path, 'utf8'))))
  }

  // ------------------------------------------------------------ Philadelphia
  const adaOld = seriesFromRows(csv('ex/Student_ADA_Yearly.csv'), 'School Year', 'ULCS Code', 'Average Daily Attendance')
  const adaNew = seriesFromRows(csv('Student_ADA_School.csv'), 'School Year', 'ULCS Code', 'Average Daily Attendance (YTD)')
  const phlAda = new Map(adaOld)
  for (const [k, v] of adaNew) phlAda.set(k, { ...(phlAda.get(k) ?? {}), ...v })

  const phlAttending90 = seriesFromRows(
    csv('Student_Attendance_90_School.csv'), 'School Year', 'ULCS Code', '% with 90%+ Attendance (Yearly)', allStudents,
  )
  const phlChronic = new Map(
    [...phlAttending90].map(([code, years]): [string, YearValue] => [
      code,
      Object.fromEntries(Object.entries(years).map(([y, v]) => [y, chronicFromAttending90(v)])),
    ]),
  )

  const a95old = seriesFromRows(csv('ex/SDP_95_Attendance_School_S.csv'), 'School Year', 'ULCS Code', '% with 95%+ Attendance (Yearly)', allStudents)
  const a95new = seriesFromRows(csv('Student_Attendance_95_School.csv'), 'School Year', 'ULCS Code', '% with 95%+ Attendance (Yearly)', allStudents)
  const phl95 = new Map(a95old)
  for (const [k, v] of a95new) phl95.set(k, { ...(phl95.get(k) ?? {}), ...v })

  const teacherAda = seriesFromRows(
    csv('ex/Archive_Teacher_95_ADA_2012-13_2022-23/OD_Teacher_Attendance_Schools_Suppressed_20240328.csv'),
    'school_year', 'ulcscode', 'pct_att_ada',
  )

  const series: [string, Map<string, YearValue>][] = [
    ['ada', phlAda],
    ['chronic', phlChronic],
    ['a95', phl95],
    ['teacherAda', teacherAda],
  ]

  for (const s of K8_SCHOOLS) {
    const code = normalizeUlcs(s.org)
    for (const [name, src] of series) {
      for (const [year, value] of Object.entries(src.get(code) ?? {})) {
        add(`phl.${name}`, `${s.org}|${year}`, real(value))
      }
    }
  }

  for (const [name, src] of series) {
    if (name === 'teacherAda') continue
    for (const [year, values] of districtMedians(src)) {
      add(`phlMedian.${name}`, year, real(median(values)))
      add(`phlCount.${name}`, year, int(values.length))
    }
  }

  // ---------------------------------------------------------------- New York
  const nycByYear = new Map<string, ReturnType<typeof chronicFromNyc>>()
  for (const y of NYC_YEARS) {
    nycByYear.set(y, chronicFromNyc(JSON.parse(readFileSync(join(A, `nyc_attendance_${y}.json`), 'utf8')) as NycRow[]))
  }

  for (const s of NYC_SCHOOLS) {
    for (const y of NYC_YEARS) {
      const hit = nycByYear.get(y)?.get(s.dbn)
      if (hit === undefined) continue
      add('nyc.ada', `${s.dbn}|${y}`, real(hit.ada))
      add('nyc.chronic', `${s.dbn}|${y}`, real(hit.chronic))
    }
  }

  for (const y of NYC_YEARS) {
    const all = [...(nycByYear.get(y)?.values() ?? [])]
    add('nycMedian.ada', y, real(median(all.map((r) => r.ada))))
    add('nycMedian.chronic', y, real(median(all.map((r) => r.chronic))))
    add('nycMedian.n', y, int(all.length))
  }

  // ----------------------------------------------- federal, all three cities
  type CrdcRow = { ncessch?: string; students_chronically_absent?: number; enrollment?: number; homeless?: number }
  const readCrdc = (file: string): CrdcRow[] =>
    (JSON.parse(readFileSync(join(A, file), 'utf8')) as { results: CrdcRow[] }).results ?? []

  const targets = new Map<string, { city: string; label: string }>()
  for (const s of K8_SCHOOLS) if (s.nces) targets.set(s.nces, { city: 'Philadelphia', label: s.label })
  for (const s of NYC_SCHOOLS) targets.set(s.nces, { city: 'Brooklyn', label: s.label })
  // SEDA drops Alabama's leading zero from the NCES id; CRDC keeps all twelve.
  for (const r of csv('../seda/seda_cov_school_pool_4.1.csv')) {
    if (r['sedalea'] !== TUSCALOOSA_SEDA_LEA) continue
    const id = r['sedasch']
    if (!id) continue
    targets.set(id.length === 11 ? `0${id}` : id, { city: 'Tuscaloosa City', label: (r['schnam'] ?? id).trim() })
  }

  for (const [nces, meta] of targets) {
    add('federal.city', nces, text(cityToken(meta.city)))
    add('federal.label', nces, text(meta.label))
  }

  for (const year of CRDC_YEARS) {
    const crdcAbsent = new Map<string, number>()
    const crdcEnrolled = new Map<string, number>()
    for (const lea of CRDC_LEAS) {
      for (const r of readCrdc(`crdc_absent_${year}_${lea}.json`)) {
        // CRDC returns two rows per school even with every other subgroup
        // pinned to 99: one for all students and one for the homeless subgroup.
        if (r.homeless !== undefined && r.homeless !== 99) continue
        if (r.ncessch && typeof r.students_chronically_absent === 'number') {
          crdcAbsent.set(r.ncessch, r.students_chronically_absent)
        }
      }
      for (const r of readCrdc(`crdc_enrolled_${year}_${lea}.json`)) {
        if (r.ncessch && typeof r.enrollment === 'number') crdcEnrolled.set(r.ncessch, r.enrollment)
      }
    }

    const byCity = new Map<string, number[]>()
    for (const [nces, meta] of targets) {
      const rate = federalRate({ absent: crdcAbsent.get(nces) ?? null, enrolled: crdcEnrolled.get(nces) ?? null })
      if (rate === null) continue
      add('federal.rate', `${year}|${nces}`, real(rate))
      const bucket = byCity.get(meta.city)
      if (bucket) bucket.push(rate)
      else byCity.set(meta.city, [rate])
    }

    for (const [city, rates] of byCity) {
      const key = `${year}|${cityToken(city)}`
      add('federalMedian.median', key, real(median(rates)))
      add('federalMedian.min', key, real(Math.min(...rates)))
      add('federalMedian.max', key, real(Math.max(...rates)))
      add('federalMedian.n', key, int(rates.length))
    }
  }

  // ------------------------------------------------------------ correlation
  const withBoth: { org: string; chronic: number; teacher: number }[] = []
  for (const s of K8_SCHOOLS) {
    const code = normalizeUlcs(s.org)
    const chronic = (phlChronic.get(code) ?? {})['2024-2025']
    // Year keys are compared as strings, exactly as the original does; they are
    // all ASCII digits, so every language orders them the same way.
    const teacher = median(Object.entries(teacherAda.get(code) ?? {}).filter(([y]) => y >= '2016').map(([, v]) => v))
    if (chronic === undefined || teacher === null) continue
    withBoth.push({ org: s.org, chronic, teacher })
    add('correlation.chronic', s.org, real(chronic))
    add('correlation.teacher', s.org, real(teacher))
  }
  add('correlation.n', '-', int(withBoth.length))
  add(
    'correlation.chronicVsTeacherAttendance',
    '-',
    real(pearson(withBoth.map((s) => s.chronic), withBoth.map((s) => s.teacher))),
  )

  return rows
}

if (import.meta.filename === process.argv[1]) {
  const outDir = join(ROOT, 'calc', 'out')
  mkdirSync(outDir, { recursive: true })
  const tsv = serialize(emitRows())
  writeFileSync(join(outDir, 'attendance-typescript.tsv'), tsv)
  console.log(`wrote calc/out/attendance-typescript.tsv (${tsv.split('\n').length - 2} facts)`)
}

metric	key	value
correlation.chronic	1260	55.990000000000
correlation.chronic	1280	4.290000000000
correlation.chronic	1300	48.450000000000
correlation.chronic	1340	29.450000000000
correlation.chronic	1390	17.910000000000
correlation.chronic	1470	47.460000000000
correlation.chronic	2210	26.770000000000
correlation.chronic	2380	5.510000000000
correlation.chronic	2470	9.930000000000
correlation.chronic	2480	23.120000000000
correlation.chronic	2590	24.320000000000
correlation.chronicVsTeacherAttendance	-	-0.699502303752
correlation.n	-	11
correlation.teacher	1260	92.930000000000
correlation.teacher	1280	96.720000000000
correlation.teacher	1300	91.680000000000
correlation.teacher	1340	93.540000000000
correlation.teacher	1390	94.650000000000
correlation.teacher	1470	94.030000000000
correlation.teacher	2210	94.970000000000
correlation.teacher	2380	95.190000000000
correlation.teacher	2470	94.080000000000
correlation.teacher	2480	93.300000000000
correlation.teacher	2590	92.710000000000
federal.city	010336000584	Tuscaloosa_City
federal.city	010336001250	Tuscaloosa_City
federal.city	010336001251	Tuscaloosa_City
federal.city	010336001252	Tuscaloosa_City
federal.city	010336001255	Tuscaloosa_City
federal.city	010336001256	Tuscaloosa_City
federal.city	010336001257	Tuscaloosa_City
federal.city	010336001260	Tuscaloosa_City
federal.city	010336001266	Tuscaloosa_City
federal.city	010336001268	Tuscaloosa_City
federal.city	010336001269	Tuscaloosa_City
federal.city	010336001271	Tuscaloosa_City
federal.city	010336001632	Tuscaloosa_City
federal.city	010336001881	Tuscaloosa_City
federal.city	010336001889	Tuscaloosa_City
federal.city	010336002102	Tuscaloosa_City
federal.city	010336002134	Tuscaloosa_City
federal.city	010336002143	Tuscaloosa_City
federal.city	010336002145	Tuscaloosa_City
federal.city	360009102069	Brooklyn
federal.city	360009102074	Brooklyn
federal.city	360009102490	Brooklyn
federal.city	360009102793	Brooklyn
federal.city	360009202072	Brooklyn
federal.city	360009202175	Brooklyn
federal.city	360009202226	Brooklyn
federal.city	360009202403	Brooklyn
federal.city	360009202557	Brooklyn
federal.city	360009202818	Brooklyn
federal.city	360009206406	Brooklyn
federal.city	421899000619	Philadelphia
federal.city	421899003618	Philadelphia
federal.city	421899003649	Philadelphia
federal.city	421899003650	Philadelphia
federal.city	421899003782	Philadelphia
federal.city	421899003783	Philadelphia
federal.city	421899003805	Philadelphia
federal.city	421899004732	Philadelphia
federal.city	421899005186	Philadelphia
federal.city	421899006528	Philadelphia
federal.city	421899006530	Philadelphia
federal.label	010336000584	Rock Quarry Elementary School
federal.label	010336001250	The Alberta School of Performing Arts
federal.label	010336001251	Arcadia Elementary School
federal.label	010336001252	Central Elementary School
federal.label	010336001255	Eastwood Middle School
federal.label	010336001256	Northington Elementary School
federal.label	010336001257	Oakdale Elementary School
federal.label	010336001260	Skyland Elementary School
federal.label	010336001266	University Place Elementary School
federal.label	010336001268	Westlawn Middle School
federal.label	010336001269	Woodland Forrest Elementary School
federal.label	010336001271	Martin L King Jr Elementary School
federal.label	010336001632	Verner Elementary School
federal.label	010336001881	University Place Middle School
federal.label	010336001889	Northridge Middle School
federal.label	010336002102	Northridge High School
federal.label	010336002134	Southview Elementary School
federal.label	010336002143	Tuscaloosa Magnet School  Middle
federal.label	010336002145	Tuscaloosa Magnet School  Elementary
federal.label	360009102069	PS 9 Teunis G. Bergen
federal.label	360009102074	PS 11 Purvis J. Behan
federal.label	360009102490	PS 133 William A. Butler
federal.label	360009102793	PS 282 Park Slope
federal.label	360009202072	PS 10 Magnet School
federal.label	360009202175	PS 39 Henry Bristow
federal.label	360009202226	MS 51 William Alexander
federal.label	360009202403	PS 107 John W. Kimball
federal.label	360009202557	PS 154 Windsor Terrace
federal.label	360009202818	PS 321 William Penn
federal.label	360009206406	PS 118 Maurice Sendak
federal.label	421899000619	Penn Alexander
federal.label	421899003618	Bache-Martin
federal.label	421899003649	Meredith
federal.label	421899003650	Nebinger
federal.label	421899003782	Comegys
federal.label	421899003783	Powel
federal.label	421899003805	Marian Anderson
federal.label	421899004732	Alain Locke
federal.label	421899005186	Greenfield
federal.label	421899006528	Harrington
federal.label	421899006530	Lea
federal.rate	2017|010336000584	3.867403314917
federal.rate	2017|010336001250	9.203539823009
federal.rate	2017|010336001251	10.322580645161
federal.rate	2017|010336001252	8.235294117647
federal.rate	2017|010336001255	12.639405204461
federal.rate	2017|010336001256	14.317673378076
federal.rate	2017|010336001257	10.191082802548
federal.rate	2017|010336001260	12.250712250712
federal.rate	2017|010336001266	13.725490196078
federal.rate	2017|010336001268	13.881748071979
federal.rate	2017|010336001269	8.604206500956
federal.rate	2017|010336001271	17.201834862385
federal.rate	2017|010336001632	3.684210526316
federal.rate	2017|010336001881	8.965517241379
federal.rate	2017|010336001889	6.415094339623
federal.rate	2017|010336002102	15.069860279441
federal.rate	2017|010336002134	10.720887245841
federal.rate	2017|010336002143	1.829268292683
federal.rate	2017|010336002145	0.338983050847
federal.rate	2017|360009102069	18.666666666667
federal.rate	2017|360009102074	13.325867861142
federal.rate	2017|360009102490	16.901408450704
federal.rate	2017|360009102793	32.472826086957
federal.rate	2017|360009202072	7.502679528403
federal.rate	2017|360009202175	5.109489051095
federal.rate	2017|360009202226	4.240282685512
federal.rate	2017|360009202403	6.537102473498
federal.rate	2017|360009202557	4.642166344294
federal.rate	2017|360009202818	4.446012702893
federal.rate	2017|360009206406	9.385113268608
federal.rate	2017|421899000619	3.993055555556
federal.rate	2017|421899003618	20.204081632653
federal.rate	2017|421899003649	2.966841186736
federal.rate	2017|421899003650	7.444168734491
federal.rate	2017|421899003782	31.034482758621
federal.rate	2017|421899003783	13.257575757576
federal.rate	2017|421899003805	17.391304347826
federal.rate	2017|421899004732	33.098591549296
federal.rate	2017|421899005186	4.071661237785
federal.rate	2017|421899006528	46.739130434783
federal.rate	2017|421899006530	24.263431542461
federal.rate	2022|010336000584	8.925619834711
federal.rate	2022|010336001250	22.205663189270
federal.rate	2022|010336001251	24.090909090909
federal.rate	2022|010336001252	20.202020202020
federal.rate	2022|010336001255	29.155672823219
federal.rate	2022|010336001257	22.012578616352
federal.rate	2022|010336001260	38.961038961039
federal.rate	2022|010336001266	21.035058430718
federal.rate	2022|010336001268	28.838951310861
federal.rate	2022|010336001269	18.538324420677
federal.rate	2022|010336001271	20.278330019881
federal.rate	2022|010336001632	8.744038155803
federal.rate	2022|010336001889	12.972972972973
federal.rate	2022|010336002102	12.314410480349
federal.rate	2022|010336002134	31.697054698457
federal.rate	2022|010336002143	2.366863905325
federal.rate	2022|010336002145	3.741496598639
federal.rate	2022|360009102069	2.595628415301
federal.rate	2022|360009102074	28.245192307692
federal.rate	2022|360009102490	21.182266009852
federal.rate	2022|360009102793	12.298387096774
federal.rate	2022|360009202072	9.288299155609
federal.rate	2022|360009202175	19.186046511628
federal.rate	2022|360009202226	21.832358674464
federal.rate	2022|360009202403	18.962075848303
federal.rate	2022|360009202557	21.065375302663
federal.rate	2022|360009202818	1.891447368421
federal.rate	2022|360009206406	3.317535545024
federal.rate	2022|421899000619	0.863557858377
federal.rate	2022|421899003618	25.660377358491
federal.rate	2022|421899003649	6.201550387597
federal.rate	2022|421899003650	14.814814814815
federal.rate	2022|421899003782	66.250000000000
federal.rate	2022|421899003783	17.241379310345
federal.rate	2022|421899003805	21.498371335505
federal.rate	2022|421899004732	50.354609929078
federal.rate	2022|421899005186	2.928257686676
federal.rate	2022|421899006528	42.647058823529
federal.rate	2022|421899006530	30.736842105263
federalMedian.max	2017|Brooklyn	32.472826086957
federalMedian.max	2017|Philadelphia	46.739130434783
federalMedian.max	2017|Tuscaloosa_City	17.201834862385
federalMedian.max	2022|Brooklyn	28.245192307692
federalMedian.max	2022|Philadelphia	66.250000000000
federalMedian.max	2022|Tuscaloosa_City	38.961038961039
federalMedian.median	2017|Brooklyn	7.502679528403
federalMedian.median	2017|Philadelphia	17.391304347826
federalMedian.median	2017|Tuscaloosa_City	10.191082802548
federalMedian.median	2022|Brooklyn	18.962075848303
federalMedian.median	2022|Philadelphia	21.498371335505
federalMedian.median	2022|Tuscaloosa_City	20.278330019881
federalMedian.min	2017|Brooklyn	4.240282685512
federalMedian.min	2017|Philadelphia	2.966841186736
federalMedian.min	2017|Tuscaloosa_City	0.338983050847
federalMedian.min	2022|Brooklyn	1.891447368421
federalMedian.min	2022|Philadelphia	0.863557858377
federalMedian.min	2022|Tuscaloosa_City	2.366863905325
federalMedian.n	2017|Brooklyn	11
federalMedian.n	2017|Philadelphia	11
federalMedian.n	2017|Tuscaloosa_City	19
federalMedian.n	2022|Brooklyn	11
federalMedian.n	2022|Philadelphia	11
federalMedian.n	2022|Tuscaloosa_City	17
input.digest	ada_archive	0aa8273a29550ea4e013e94fab06668c5f03ca7de73d601a7d4391ea9a1ef8d2
input.digest	ada_current	75fc69a3ce4cf196015a6ec38712a80d8a63ebfa1954af14ad047c90cf20497a
input.digest	attending90	b52be38854930e4e102ae9d0644126f4029d7e4d923ee6748a59d2c8c2f76990
input.digest	attending95_archive	ae6a7af2595198f7438a166f661fde82855798ce82c95a22591c9870cb96231e
input.digest	attending95_current	66ab6997ac8503b1c4a2bdc9dfed7412f3c8f1da38e6d249d46ee62f507d8740
input.digest	seda_covariates	8de18e097d8bd9910c5a1d334ded0b53e39d3a7fa190740027f2214650cef885
input.digest	teacher_ada	7005f4f52d1486cb6bd8ed71ffd973e500324a6243cdf586c63e7e17b3ec359f
input.filedigest	crdc_absent_2017_0103360	245db5a18baaaf899fce31539a226e09690ed98a0f1c021e2e3d750b1c9ee3a2
input.filedigest	crdc_absent_2017_3600091	d138990de9a8f93c219e31b3088f0dee8754c106532988d84cc6eda517b0338e
input.filedigest	crdc_absent_2017_3600092	8181f9f1988a6c78562fdb6123e3444a460f0042883032f7f2f724ad8a87ff6b
input.filedigest	crdc_absent_2017_4218990	aeebd062ef76af8ed9472c9a0be47e8a1fc982c69a5b5e9eb16fb918cb940c93
input.filedigest	crdc_absent_2022_0103360	487f7c716c2d433f195b5078083c00ba886d0f51e3cea856e1b20c985ff740a6
input.filedigest	crdc_absent_2022_3600091	a8634f93eed87c954c4fa852a7f2baba19144c8ccd77b280eee751aaa2a9dff5
input.filedigest	crdc_absent_2022_3600092	c12bc5e87640e78ba52b95024194e316af5cd341382a3ad62944af3a7b3538a9
input.filedigest	crdc_absent_2022_4218990	17f9e700da52bf37f5bde57fc2d33f7cf6ea737cbfc41757ed63ebd75f62be93
input.filedigest	crdc_enrolled_2017_0103360	f28f2133f16506dff70ccdbcd2b089a208e4a04970ec9fccb47dd74890aeb0f1
input.filedigest	crdc_enrolled_2017_3600091	9958bf6704cca35c4cb9a438ebce01dd6d213dc0a9f1834a9dc02f4b3226c847
input.filedigest	crdc_enrolled_2017_3600092	44e6dfab704258349674644ae8c71f7fcac7950d22fa82b472b8beb7c1158c4f
input.filedigest	crdc_enrolled_2017_4218990	a8707579418263fa0d72ad8d545a0225ea6ae332ef4b91df710c41315e71509a
input.filedigest	crdc_enrolled_2022_0103360	08c371e38796b6b16c0eb5628a6e960a6767d0e160eea2326496dcf706023c8d
input.filedigest	crdc_enrolled_2022_3600091	0b310f0064b5ad356adc75303bdaf900b53fb281a1d8795bb55a6126bd907b21
input.filedigest	crdc_enrolled_2022_3600092	1a78eba53b537754df3f48e2050084ef79c7c753e43bb0bda13181849f860247
input.filedigest	crdc_enrolled_2022_4218990	dd170c63212f6024fdb7c5751040b0de7c0a02d108363c0d3d342e8d54b48cbf
input.filedigest	nyc_2016-17	ffd1dc428b46b85ed2fc7258a49dc66f1833861de145f8de936fc5cbff7bf05d
input.filedigest	nyc_2017-18	3d7dd135041c6527b2ac521b36fcf574b2f9720f20964dafcfde4c3e536e7245
input.filedigest	nyc_2018-19	5777c2300849906be9b508c0fa9dd655ce1fda96060ba0cf2b4df4b49d205248
input.filedigest	nyc_2019-20	26c3ef82ca43b92fca33ec84f2e3f596abeffe609eb9e45066f90724638baf9b
input.filedigest	nyc_2020-21	7abe5b2dfb4f850fea10afdb3ea3d367f43bc7999512a2443eee9efc5be27a76
input.rows	ada_archive	1503
input.rows	ada_current	1074
input.rows	attending90	16926
input.rows	attending95_archive	22944
input.rows	attending95_current	16926
input.rows	seda_covariates	89350
input.rows	teacher_ada	2407
nyc.ada	13K009|2016-17	94.100000000000
nyc.ada	13K009|2017-18	93.900000000000
nyc.ada	13K009|2018-19	94.300000000000
nyc.ada	13K009|2019-20	93.100000000000
nyc.ada	13K009|2020-21	94.200000000000
nyc.ada	13K011|2016-17	94.700000000000
nyc.ada	13K011|2017-18	94.700000000000
nyc.ada	13K011|2018-19	94.500000000000
nyc.ada	13K011|2019-20	94.800000000000
nyc.ada	13K011|2020-21	95.000000000000
nyc.ada	13K133|2016-17	93.800000000000
nyc.ada	13K133|2017-18	93.800000000000
nyc.ada	13K133|2018-19	93.800000000000
nyc.ada	13K133|2019-20	93.200000000000
nyc.ada	13K133|2020-21	95.600000000000
nyc.ada	13K282|2016-17	90.500000000000
nyc.ada	13K282|2017-18	90.800000000000
nyc.ada	13K282|2018-19	91.600000000000
nyc.ada	13K282|2019-20	92.000000000000
nyc.ada	13K282|2020-21	92.100000000000
nyc.ada	15K010|2016-17	95.400000000000
nyc.ada	15K010|2017-18	95.600000000000
nyc.ada	15K010|2018-19	95.300000000000
nyc.ada	15K010|2019-20	94.700000000000
nyc.ada	15K010|2020-21	94.300000000000
nyc.ada	15K039|2016-17	95.700000000000
nyc.ada	15K039|2017-18	96.400000000000
nyc.ada	15K039|2018-19	96.000000000000
nyc.ada	15K039|2019-20	95.500000000000
nyc.ada	15K039|2020-21	95.900000000000
nyc.ada	15K051|2016-17	96.900000000000
nyc.ada	15K051|2017-18	97.000000000000
nyc.ada	15K051|2018-19	97.100000000000
nyc.ada	15K051|2019-20	96.800000000000
nyc.ada	15K051|2020-21	89.600000000000
nyc.ada	15K107|2016-17	95.600000000000
nyc.ada	15K107|2017-18	96.200000000000
nyc.ada	15K107|2018-19	95.500000000000
nyc.ada	15K107|2019-20	94.900000000000
nyc.ada	15K107|2020-21	97.000000000000
nyc.ada	15K118|2016-17	94.200000000000
nyc.ada	15K118|2017-18	95.400000000000
nyc.ada	15K118|2018-19	94.900000000000
nyc.ada	15K118|2019-20	94.800000000000
nyc.ada	15K118|2020-21	96.400000000000
nyc.ada	15K154|2016-17	95.900000000000
nyc.ada	15K154|2017-18	96.400000000000
nyc.ada	15K154|2018-19	96.300000000000
nyc.ada	15K154|2019-20	95.600000000000
nyc.ada	15K154|2020-21	98.900000000000
nyc.ada	15K321|2016-17	96.300000000000
nyc.ada	15K321|2017-18	96.300000000000
nyc.ada	15K321|2018-19	96.100000000000
nyc.ada	15K321|2019-20	95.600000000000
nyc.ada	15K321|2020-21	97.300000000000
nyc.chronic	13K009|2016-17	16.400000000000
nyc.chronic	13K009|2017-18	20.000000000000
nyc.chronic	13K009|2018-19	17.400000000000
nyc.chronic	13K009|2019-20	22.100000000000
nyc.chronic	13K009|2020-21	16.000000000000
nyc.chronic	13K011|2016-17	12.900000000000
nyc.chronic	13K011|2017-18	13.200000000000
nyc.chronic	13K011|2018-19	15.300000000000
nyc.chronic	13K011|2019-20	13.200000000000
nyc.chronic	13K011|2020-21	15.400000000000
nyc.chronic	13K133|2016-17	19.000000000000
nyc.chronic	13K133|2017-18	18.000000000000
nyc.chronic	13K133|2018-19	18.300000000000
nyc.chronic	13K133|2019-20	22.200000000000
nyc.chronic	13K133|2020-21	13.400000000000
nyc.chronic	13K282|2016-17	34.400000000000
nyc.chronic	13K282|2017-18	33.700000000000
nyc.chronic	13K282|2018-19	29.300000000000
nyc.chronic	13K282|2019-20	24.500000000000
nyc.chronic	13K282|2020-21	24.200000000000
nyc.chronic	15K010|2016-17	9.400000000000
nyc.chronic	15K010|2017-18	7.500000000000
nyc.chronic	15K010|2018-19	9.000000000000
nyc.chronic	15K010|2019-20	12.200000000000
nyc.chronic	15K010|2020-21	17.200000000000
nyc.chronic	15K039|2016-17	6.800000000000
nyc.chronic	15K039|2017-18	5.000000000000
nyc.chronic	15K039|2018-19	6.700000000000
nyc.chronic	15K039|2019-20	8.300000000000
nyc.chronic	15K039|2020-21	9.500000000000
nyc.chronic	15K051|2016-17	4.900000000000
nyc.chronic	15K051|2017-18	5.200000000000
nyc.chronic	15K051|2018-19	5.100000000000
nyc.chronic	15K051|2019-20	6.300000000000
nyc.chronic	15K051|2020-21	34.300000000000
nyc.chronic	15K107|2016-17	8.900000000000
nyc.chronic	15K107|2017-18	6.400000000000
nyc.chronic	15K107|2018-19	7.200000000000
nyc.chronic	15K107|2019-20	9.500000000000
nyc.chronic	15K107|2020-21	5.900000000000
nyc.chronic	15K118|2016-17	13.500000000000
nyc.chronic	15K118|2017-18	9.000000000000
nyc.chronic	15K118|2018-19	12.400000000000
nyc.chronic	15K118|2019-20	12.300000000000
nyc.chronic	15K118|2020-21	8.000000000000
nyc.chronic	15K154|2016-17	6.100000000000
nyc.chronic	15K154|2017-18	5.200000000000
nyc.chronic	15K154|2018-19	6.100000000000
nyc.chronic	15K154|2019-20	7.600000000000
nyc.chronic	15K154|2020-21	0.900000000000
nyc.chronic	15K321|2016-17	4.700000000000
nyc.chronic	15K321|2017-18	4.500000000000
nyc.chronic	15K321|2018-19	5.700000000000
nyc.chronic	15K321|2019-20	7.100000000000
nyc.chronic	15K321|2020-21	3.600000000000
nycMedian.ada	2016-17	92.200000000000
nycMedian.ada	2017-18	91.800000000000
nycMedian.ada	2018-19	91.800000000000
nycMedian.ada	2019-20	92.300000000000
nycMedian.ada	2020-21	90.400000000000
nycMedian.chronic	2016-17	27.300000000000
nycMedian.chronic	2017-18	28.900000000000
nycMedian.chronic	2018-19	28.650000000000
nycMedian.chronic	2019-20	26.500000000000
nycMedian.chronic	2020-21	30.800000000000
nycMedian.n	2016-17	1546
nycMedian.n	2017-18	1539
nycMedian.n	2018-19	1526
nycMedian.n	2019-20	1527
nycMedian.n	2020-21	1530
phl.a95	1260|2013-2014	30.140000000000
phl.a95	1260|2014-2015	34.010000000000
phl.a95	1260|2015-2016	45.200000000000
phl.a95	1260|2016-2017	28.940000000000
phl.a95	1260|2017-2018	33.160000000000
phl.a95	1260|2018-2019	33.040000000000
phl.a95	1260|2019-2020	37.930000000000
phl.a95	1260|2020-2021	27.980000000000
phl.a95	1260|2021-2022	15.810000000000
phl.a95	1260|2022-2023	14.520000000000
phl.a95	1260|2023-2024	16.770000000000
phl.a95	1260|2024-2025	23.620000000000
phl.a95	1280|2013-2014	77.950000000000
phl.a95	1280|2014-2015	80.560000000000
phl.a95	1280|2015-2016	82.450000000000
phl.a95	1280|2016-2017	79.450000000000
phl.a95	1280|2017-2018	80.030000000000
phl.a95	1280|2018-2019	84.850000000000
phl.a95	1280|2019-2020	85.890000000000
phl.a95	1280|2020-2021	94.490000000000
phl.a95	1280|2021-2022	87.370000000000
phl.a95	1280|2022-2023	81.740000000000
phl.a95	1280|2023-2024	82.650000000000
phl.a95	1280|2024-2025	80.690000000000
phl.a95	1300|2013-2014	42.620000000000
phl.a95	1300|2014-2015	35.400000000000
phl.a95	1300|2015-2016	41.410000000000
phl.a95	1300|2016-2017	29.500000000000
phl.a95	1300|2017-2018	23.990000000000
phl.a95	1300|2018-2019	27.370000000000
phl.a95	1300|2019-2020	38.400000000000
phl.a95	1300|2020-2021	34.500000000000
phl.a95	1300|2021-2022	20.310000000000
phl.a95	1300|2022-2023	30.130000000000
phl.a95	1300|2023-2024	27.030000000000
phl.a95	1300|2024-2025	25.470000000000
phl.a95	1340|2013-2014	41.630000000000
phl.a95	1340|2014-2015	42.250000000000
phl.a95	1340|2015-2016	48.500000000000
phl.a95	1340|2016-2017	40.870000000000
phl.a95	1340|2017-2018	46.930000000000
phl.a95	1340|2018-2019	44.720000000000
phl.a95	1340|2019-2020	50.000000000000
phl.a95	1340|2020-2021	49.590000000000
phl.a95	1340|2021-2022	40.320000000000
phl.a95	1340|2022-2023	45.200000000000
phl.a95	1340|2023-2024	42.600000000000
phl.a95	1340|2024-2025	46.270000000000
phl.a95	1390|2013-2014	54.240000000000
phl.a95	1390|2014-2015	55.590000000000
phl.a95	1390|2015-2016	67.220000000000
phl.a95	1390|2016-2017	56.770000000000
phl.a95	1390|2017-2018	60.140000000000
phl.a95	1390|2018-2019	71.980000000000
phl.a95	1390|2019-2020	74.300000000000
phl.a95	1390|2020-2021	64.410000000000
phl.a95	1390|2021-2022	46.510000000000
phl.a95	1390|2022-2023	55.210000000000
phl.a95	1390|2023-2024	56.040000000000
phl.a95	1390|2024-2025	55.820000000000
phl.a95	1470|2013-2014	27.480000000000
phl.a95	1470|2014-2015	31.940000000000
phl.a95	1470|2015-2016	30.140000000000
phl.a95	1470|2016-2017	22.660000000000
phl.a95	1470|2017-2018	32.220000000000
phl.a95	1470|2018-2019	29.190000000000
phl.a95	1470|2019-2020	37.500000000000
phl.a95	1470|2020-2021	34.590000000000
phl.a95	1470|2021-2022	20.410000000000
phl.a95	1470|2022-2023	25.930000000000
phl.a95	1470|2023-2024	22.490000000000
phl.a95	1470|2024-2025	23.730000000000
phl.a95	2210|2013-2014	42.380000000000
phl.a95	2210|2014-2015	48.070000000000
phl.a95	2210|2015-2016	44.090000000000
phl.a95	2210|2016-2017	41.420000000000
phl.a95	2210|2017-2018	47.450000000000
phl.a95	2210|2018-2019	52.630000000000
phl.a95	2210|2019-2020	63.920000000000
phl.a95	2210|2020-2021	60.730000000000
phl.a95	2210|2021-2022	31.490000000000
phl.a95	2210|2022-2023	42.050000000000
phl.a95	2210|2023-2024	49.820000000000
phl.a95	2210|2024-2025	47.620000000000
phl.a95	2380|2013-2014	68.520000000000
phl.a95	2380|2014-2015	71.430000000000
phl.a95	2380|2015-2016	75.550000000000
phl.a95	2380|2016-2017	75.480000000000
phl.a95	2380|2017-2018	79.250000000000
phl.a95	2380|2018-2019	80.960000000000
phl.a95	2380|2019-2020	83.460000000000
phl.a95	2380|2020-2021	95.150000000000
phl.a95	2380|2021-2022	80.570000000000
phl.a95	2380|2022-2023	72.920000000000
phl.a95	2380|2023-2024	74.150000000000
phl.a95	2380|2024-2025	68.210000000000
phl.a95	2470|2013-2014	62.710000000000
phl.a95	2470|2014-2015	67.160000000000
phl.a95	2470|2015-2016	71.020000000000
phl.a95	2470|2016-2017	75.820000000000
phl.a95	2470|2017-2018	79.870000000000
phl.a95	2470|2018-2019	82.250000000000
phl.a95	2470|2019-2020	87.200000000000
phl.a95	2470|2020-2021	93.900000000000
phl.a95	2470|2021-2022	76.640000000000
phl.a95	2470|2022-2023	66.900000000000
phl.a95	2470|2023-2024	65.300000000000
phl.a95	2470|2024-2025	59.220000000000
phl.a95	2480|2013-2014	38.800000000000
phl.a95	2480|2014-2015	37.100000000000
phl.a95	2480|2015-2016	46.270000000000
phl.a95	2480|2016-2017	47.130000000000
phl.a95	2480|2017-2018	54.870000000000
phl.a95	2480|2018-2019	49.650000000000
phl.a95	2480|2019-2020	63.050000000000
phl.a95	2480|2020-2021	73.780000000000
phl.a95	2480|2021-2022	60.540000000000
phl.a95	2480|2022-2023	53.140000000000
phl.a95	2480|2023-2024	56.740000000000
phl.a95	2480|2024-2025	47.910000000000
phl.a95	2590|2013-2014	52.350000000000
phl.a95	2590|2014-2015	49.860000000000
phl.a95	2590|2015-2016	59.830000000000
phl.a95	2590|2016-2017	49.470000000000
phl.a95	2590|2017-2018	58.950000000000
phl.a95	2590|2018-2019	63.600000000000
phl.a95	2590|2019-2020	70.130000000000
phl.a95	2590|2020-2021	60.740000000000
phl.a95	2590|2021-2022	60.250000000000
phl.a95	2590|2022-2023	62.060000000000
phl.a95	2590|2023-2024	53.430000000000
phl.a95	2590|2024-2025	44.720000000000
phl.ada	1260|2013-2014	91.800000000000
phl.ada	1260|2014-2015	93.700000000000
phl.ada	1260|2015-2016	93.600000000000
phl.ada	1260|2016-2017	90.100000000000
phl.ada	1260|2017-2018	90.600000000000
phl.ada	1260|2018-2019	89.100000000000
phl.ada	1260|2019-2020	89.500000000000
phl.ada	1260|2020-2021	80.340000000000
phl.ada	1260|2021-2022	82.530000000000
phl.ada	1260|2022-2023	82.810000000000
phl.ada	1260|2023-2024	83.410000000000
phl.ada	1260|2024-2025	87.480000000000
phl.ada	1280|2013-2014	97.700000000000
phl.ada	1280|2014-2015	97.900000000000
phl.ada	1280|2015-2016	97.300000000000
phl.ada	1280|2016-2017	96.800000000000
phl.ada	1280|2017-2018	96.900000000000
phl.ada	1280|2018-2019	97.400000000000
phl.ada	1280|2019-2020	97.300000000000
phl.ada	1280|2020-2021	98.680000000000
phl.ada	1280|2021-2022	96.990000000000
phl.ada	1280|2022-2023	96.690000000000
phl.ada	1280|2023-2024	96.880000000000
phl.ada	1280|2024-2025	96.880000000000
phl.ada	1300|2013-2014	94.600000000000
phl.ada	1300|2014-2015	95.000000000000
phl.ada	1300|2015-2016	94.100000000000
phl.ada	1300|2016-2017	91.300000000000
phl.ada	1300|2017-2018	87.500000000000
phl.ada	1300|2018-2019	88.200000000000
phl.ada	1300|2019-2020	90.300000000000
phl.ada	1300|2020-2021	83.450000000000
phl.ada	1300|2021-2022	83.400000000000
phl.ada	1300|2022-2023	88.480000000000
phl.ada	1300|2023-2024	87.340000000000
phl.ada	1300|2024-2025	87.300000000000
phl.ada	1340|2013-2014	93.200000000000
phl.ada	1340|2014-2015	95.800000000000
phl.ada	1340|2015-2016	94.300000000000
phl.ada	1340|2016-2017	90.600000000000
phl.ada	1340|2017-2018	91.700000000000
phl.ada	1340|2018-2019	91.700000000000
phl.ada	1340|2019-2020	92.900000000000
phl.ada	1340|2020-2021	87.150000000000
phl.ada	1340|2021-2022	90.510000000000
phl.ada	1340|2022-2023	91.380000000000
phl.ada	1340|2023-2024	90.970000000000
phl.ada	1340|2024-2025	91.330000000000
phl.ada	1390|2013-2014	95.600000000000
phl.ada	1390|2014-2015	96.900000000000
phl.ada	1390|2015-2016	97.300000000000
phl.ada	1390|2016-2017	94.600000000000
phl.ada	1390|2017-2018	94.800000000000
phl.ada	1390|2018-2019	96.100000000000
phl.ada	1390|2019-2020	95.900000000000
phl.ada	1390|2020-2021	92.070000000000
phl.ada	1390|2021-2022	91.060000000000
phl.ada	1390|2022-2023	93.290000000000
phl.ada	1390|2023-2024	93.550000000000
phl.ada	1390|2024-2025	93.550000000000
phl.ada	1470|2013-2014	91.100000000000
phl.ada	1470|2014-2015	93.700000000000
phl.ada	1470|2015-2016	92.000000000000
phl.ada	1470|2016-2017	88.100000000000
phl.ada	1470|2017-2018	90.100000000000
phl.ada	1470|2018-2019	89.200000000000
phl.ada	1470|2019-2020	90.200000000000
phl.ada	1470|2020-2021	86.700000000000
phl.ada	1470|2021-2022	85.400000000000
phl.ada	1470|2022-2023	87.210000000000
phl.ada	1470|2023-2024	86.220000000000
phl.ada	1470|2024-2025	88.120000000000
phl.ada	2210|2013-2014	94.800000000000
phl.ada	2210|2014-2015	95.400000000000
phl.ada	2210|2015-2016	95.100000000000
phl.ada	2210|2016-2017	92.000000000000
phl.ada	2210|2017-2018	92.600000000000
phl.ada	2210|2018-2019	93.100000000000
phl.ada	2210|2019-2020	94.300000000000
phl.ada	2210|2020-2021	91.190000000000
phl.ada	2210|2021-2022	88.230000000000
phl.ada	2210|2022-2023	91.170000000000
phl.ada	2210|2023-2024	91.930000000000
phl.ada	2210|2024-2025	92.670000000000
phl.ada	2380|2013-2014	97.000000000000
phl.ada	2380|2014-2015	97.400000000000
phl.ada	2380|2015-2016	97.400000000000
phl.ada	2380|2016-2017	96.400000000000
phl.ada	2380|2017-2018	96.800000000000
phl.ada	2380|2018-2019	96.900000000000
phl.ada	2380|2019-2020	97.000000000000
phl.ada	2380|2020-2021	98.820000000000
phl.ada	2380|2021-2022	96.670000000000
phl.ada	2380|2022-2023	95.770000000000
phl.ada	2380|2023-2024	96.200000000000
phl.ada	2380|2024-2025	95.870000000000
phl.ada	2470|2013-2014	96.100000000000
phl.ada	2470|2014-2015	96.300000000000
phl.ada	2470|2015-2016	96.500000000000
phl.ada	2470|2016-2017	96.200000000000
phl.ada	2470|2017-2018	96.600000000000
phl.ada	2470|2018-2019	96.500000000000
phl.ada	2470|2019-2020	96.700000000000
phl.ada	2470|2020-2021	98.610000000000
phl.ada	2470|2021-2022	96.170000000000
phl.ada	2470|2022-2023	95.230000000000
phl.ada	2470|2023-2024	95.180000000000
phl.ada	2470|2024-2025	95.050000000000
phl.ada	2480|2013-2014	94.600000000000
phl.ada	2480|2014-2015	94.500000000000
phl.ada	2480|2015-2016	95.600000000000
phl.ada	2480|2016-2017	93.500000000000
phl.ada	2480|2017-2018	93.500000000000
phl.ada	2480|2018-2019	93.000000000000
phl.ada	2480|2019-2020	94.700000000000
phl.ada	2480|2020-2021	95.020000000000
phl.ada	2480|2021-2022	93.290000000000
phl.ada	2480|2022-2023	92.650000000000
phl.ada	2480|2023-2024	93.390000000000
phl.ada	2480|2024-2025	92.710000000000
phl.ada	2590|2013-2014	94.800000000000
phl.ada	2590|2014-2015	95.200000000000
phl.ada	2590|2015-2016	95.400000000000
phl.ada	2590|2016-2017	93.800000000000
phl.ada	2590|2017-2018	94.900000000000
phl.ada	2590|2018-2019	95.000000000000
phl.ada	2590|2019-2020	95.400000000000
phl.ada	2590|2020-2021	91.000000000000
phl.ada	2590|2021-2022	93.700000000000
phl.ada	2590|2022-2023	93.560000000000
phl.ada	2590|2023-2024	92.890000000000
phl.ada	2590|2024-2025	92.810000000000
phl.chronic	1260|2020-2021	56.650000000000
phl.chronic	1260|2021-2022	66.050000000000
phl.chronic	1260|2022-2023	66.300000000000
phl.chronic	1260|2023-2024	63.040000000000
phl.chronic	1260|2024-2025	55.990000000000
phl.chronic	1280|2020-2021	2.240000000000
phl.chronic	1280|2021-2022	2.770000000000
phl.chronic	1280|2022-2023	3.410000000000
phl.chronic	1280|2023-2024	4.080000000000
phl.chronic	1280|2024-2025	4.290000000000
phl.chronic	1300|2020-2021	48.720000000000
phl.chronic	1300|2021-2022	59.820000000000
phl.chronic	1300|2022-2023	44.810000000000
phl.chronic	1300|2023-2024	49.850000000000
phl.chronic	1300|2024-2025	48.450000000000
phl.chronic	1340|2020-2021	35.770000000000
phl.chronic	1340|2021-2022	33.330000000000
phl.chronic	1340|2022-2023	31.000000000000
phl.chronic	1340|2023-2024	32.350000000000
phl.chronic	1340|2024-2025	29.450000000000
phl.chronic	1390|2020-2021	21.610000000000
phl.chronic	1390|2021-2022	31.010000000000
phl.chronic	1390|2022-2023	20.500000000000
phl.chronic	1390|2023-2024	19.500000000000
phl.chronic	1390|2024-2025	17.910000000000
phl.chronic	1470|2020-2021	46.820000000000
phl.chronic	1470|2021-2022	59.400000000000
phl.chronic	1470|2022-2023	52.710000000000
phl.chronic	1470|2023-2024	54.410000000000
phl.chronic	1470|2024-2025	47.460000000000
phl.chronic	2210|2020-2021	27.130000000000
phl.chronic	2210|2021-2022	45.190000000000
phl.chronic	2210|2022-2023	29.920000000000
phl.chronic	2210|2023-2024	25.910000000000
phl.chronic	2210|2024-2025	26.770000000000
phl.chronic	2380|2020-2021	0.900000000000
phl.chronic	2380|2021-2022	4.530000000000
phl.chronic	2380|2022-2023	6.440000000000
phl.chronic	2380|2023-2024	5.850000000000
phl.chronic	2380|2024-2025	5.510000000000
phl.chronic	2470|2020-2021	2.830000000000
phl.chronic	2470|2021-2022	7.140000000000
phl.chronic	2470|2022-2023	9.500000000000
phl.chronic	2470|2023-2024	8.440000000000
phl.chronic	2470|2024-2025	9.930000000000
phl.chronic	2480|2020-2021	14.690000000000
phl.chronic	2480|2021-2022	23.470000000000
phl.chronic	2480|2022-2023	22.640000000000
phl.chronic	2480|2023-2024	22.260000000000
phl.chronic	2480|2024-2025	23.120000000000
phl.chronic	2590|2020-2021	25.200000000000
phl.chronic	2590|2021-2022	18.600000000000
phl.chronic	2590|2022-2023	16.330000000000
phl.chronic	2590|2023-2024	20.100000000000
phl.chronic	2590|2024-2025	24.320000000000
phl.teacherAda	1260|2012-2013	95.900000000000
phl.teacherAda	1260|2013-2014	95.090000000000
phl.teacherAda	1260|2014-2015	92.930000000000
phl.teacherAda	1260|2015-2016	91.870000000000
phl.teacherAda	1260|2016-2017	96.170000000000
phl.teacherAda	1260|2017-2018	92.930000000000
phl.teacherAda	1260|2018-2019	91.260000000000
phl.teacherAda	1260|2019-2020	96.180000000000
phl.teacherAda	1260|2020-2021	96.840000000000
phl.teacherAda	1260|2021-2022	91.830000000000
phl.teacherAda	1260|2022-2023	88.200000000000
phl.teacherAda	1280|2012-2013	96.760000000000
phl.teacherAda	1280|2013-2014	96.530000000000
phl.teacherAda	1280|2014-2015	97.540000000000
phl.teacherAda	1280|2015-2016	97.150000000000
phl.teacherAda	1280|2016-2017	97.830000000000
phl.teacherAda	1280|2017-2018	97.190000000000
phl.teacherAda	1280|2018-2019	96.250000000000
phl.teacherAda	1280|2019-2020	96.720000000000
phl.teacherAda	1280|2020-2021	97.620000000000
phl.teacherAda	1280|2021-2022	93.440000000000
phl.teacherAda	1280|2022-2023	95.300000000000
phl.teacherAda	1300|2012-2013	95.600000000000
phl.teacherAda	1300|2013-2014	91.330000000000
phl.teacherAda	1300|2014-2015	94.650000000000
phl.teacherAda	1300|2015-2016	88.690000000000
phl.teacherAda	1300|2016-2017	90.570000000000
phl.teacherAda	1300|2017-2018	90.930000000000
phl.teacherAda	1300|2018-2019	92.760000000000
phl.teacherAda	1300|2019-2020	92.610000000000
phl.teacherAda	1300|2020-2021	97.840000000000
phl.teacherAda	1300|2021-2022	91.680000000000
phl.teacherAda	1300|2022-2023	90.130000000000
phl.teacherAda	1340|2012-2013	91.660000000000
phl.teacherAda	1340|2013-2014	85.760000000000
phl.teacherAda	1340|2014-2015	89.530000000000
phl.teacherAda	1340|2015-2016	88.050000000000
phl.teacherAda	1340|2016-2017	93.040000000000
phl.teacherAda	1340|2017-2018	93.540000000000
phl.teacherAda	1340|2018-2019	93.710000000000
phl.teacherAda	1340|2019-2020	93.640000000000
phl.teacherAda	1340|2020-2021	96.630000000000
phl.teacherAda	1340|2021-2022	90.550000000000
phl.teacherAda	1340|2022-2023	92.840000000000
phl.teacherAda	1390|2012-2013	93.800000000000
phl.teacherAda	1390|2013-2014	94.350000000000
phl.teacherAda	1390|2014-2015	91.570000000000
phl.teacherAda	1390|2015-2016	96.970000000000
phl.teacherAda	1390|2016-2017	92.050000000000
phl.teacherAda	1390|2017-2018	94.650000000000
phl.teacherAda	1390|2018-2019	93.850000000000
phl.teacherAda	1390|2019-2020	95.680000000000
phl.teacherAda	1390|2020-2021	97.780000000000
phl.teacherAda	1390|2021-2022	90.740000000000
phl.teacherAda	1390|2022-2023	94.840000000000
phl.teacherAda	1470|2012-2013	93.880000000000
phl.teacherAda	1470|2013-2014	93.610000000000
phl.teacherAda	1470|2014-2015	88.330000000000
phl.teacherAda	1470|2015-2016	90.610000000000
phl.teacherAda	1470|2016-2017	90.700000000000
phl.teacherAda	1470|2017-2018	95.400000000000
phl.teacherAda	1470|2018-2019	94.030000000000
phl.teacherAda	1470|2019-2020	95.130000000000
phl.teacherAda	1470|2020-2021	96.430000000000
phl.teacherAda	1470|2021-2022	92.740000000000
phl.teacherAda	1470|2022-2023	89.260000000000
phl.teacherAda	2210|2012-2013	94.680000000000
phl.teacherAda	2210|2013-2014	92.830000000000
phl.teacherAda	2210|2014-2015	94.340000000000
phl.teacherAda	2210|2015-2016	94.570000000000
phl.teacherAda	2210|2016-2017	92.340000000000
phl.teacherAda	2210|2017-2018	95.530000000000
phl.teacherAda	2210|2018-2019	94.970000000000
phl.teacherAda	2210|2019-2020	96.180000000000
phl.teacherAda	2210|2020-2021	97.430000000000
phl.teacherAda	2210|2021-2022	93.420000000000
phl.teacherAda	2210|2022-2023	93.770000000000
phl.teacherAda	2380|2012-2013	94.200000000000
phl.teacherAda	2380|2013-2014	94.180000000000
phl.teacherAda	2380|2014-2015	91.100000000000
phl.teacherAda	2380|2015-2016	94.810000000000
phl.teacherAda	2380|2016-2017	95.190000000000
phl.teacherAda	2380|2017-2018	94.550000000000
phl.teacherAda	2380|2018-2019	97.020000000000
phl.teacherAda	2380|2019-2020	95.570000000000
phl.teacherAda	2380|2020-2021	95.360000000000
phl.teacherAda	2380|2021-2022	93.500000000000
phl.teacherAda	2380|2022-2023	93.750000000000
phl.teacherAda	2470|2012-2013	93.730000000000
phl.teacherAda	2470|2013-2014	92.400000000000
phl.teacherAda	2470|2014-2015	95.620000000000
phl.teacherAda	2470|2015-2016	95.410000000000
phl.teacherAda	2470|2016-2017	92.440000000000
phl.teacherAda	2470|2017-2018	94.930000000000
phl.teacherAda	2470|2018-2019	94.080000000000
phl.teacherAda	2470|2019-2020	95.070000000000
phl.teacherAda	2470|2020-2021	97.080000000000
phl.teacherAda	2470|2021-2022	94.060000000000
phl.teacherAda	2470|2022-2023	93.970000000000
phl.teacherAda	2480|2012-2013	91.510000000000
phl.teacherAda	2480|2013-2014	87.780000000000
phl.teacherAda	2480|2014-2015	95.860000000000
phl.teacherAda	2480|2015-2016	93.720000000000
phl.teacherAda	2480|2016-2017	93.300000000000
phl.teacherAda	2480|2017-2018	94.090000000000
phl.teacherAda	2480|2018-2019	93.170000000000
phl.teacherAda	2480|2019-2020	92.090000000000
phl.teacherAda	2480|2020-2021	97.040000000000
phl.teacherAda	2480|2021-2022	94.670000000000
phl.teacherAda	2480|2022-2023	93.000000000000
phl.teacherAda	2590|2012-2013	94.790000000000
phl.teacherAda	2590|2013-2014	95.550000000000
phl.teacherAda	2590|2014-2015	92.280000000000
phl.teacherAda	2590|2015-2016	94.250000000000
phl.teacherAda	2590|2016-2017	92.710000000000
phl.teacherAda	2590|2017-2018	91.880000000000
phl.teacherAda	2590|2018-2019	93.460000000000
phl.teacherAda	2590|2019-2020	95.890000000000
phl.teacherAda	2590|2020-2021	98.120000000000
phl.teacherAda	2590|2021-2022	91.740000000000
phl.teacherAda	2590|2022-2023	91.110000000000
phlCount.a95	2013-2014	212
phlCount.a95	2014-2015	216
phlCount.a95	2015-2016	216
phlCount.a95	2016-2017	213
phlCount.a95	2017-2018	213
phlCount.a95	2018-2019	213
phlCount.a95	2019-2020	213
phlCount.a95	2020-2021	213
phlCount.a95	2021-2022	214
phlCount.a95	2022-2023	215
phlCount.a95	2023-2024	216
phlCount.a95	2024-2025	216
phlCount.ada	2013-2014	212
phlCount.ada	2014-2015	216
phlCount.ada	2015-2016	215
phlCount.ada	2016-2017	215
phlCount.ada	2017-2018	215
phlCount.ada	2018-2019	215
phlCount.ada	2019-2020	215
phlCount.ada	2020-2021	213
phlCount.ada	2021-2022	214
phlCount.ada	2022-2023	215
phlCount.ada	2023-2024	216
phlCount.ada	2024-2025	216
phlCount.chronic	2020-2021	213
phlCount.chronic	2021-2022	214
phlCount.chronic	2022-2023	215
phlCount.chronic	2023-2024	216
phlCount.chronic	2024-2025	216
phlMedian.a95	2013-2014	38.430000000000
phlMedian.a95	2014-2015	37.570000000000
phlMedian.a95	2015-2016	41.460000000000
phlMedian.a95	2016-2017	36.410000000000
phlMedian.a95	2017-2018	41.980000000000
phlMedian.a95	2018-2019	44.720000000000
phlMedian.a95	2019-2020	53.320000000000
phlMedian.a95	2020-2021	49.440000000000
phlMedian.a95	2021-2022	31.215000000000
phlMedian.a95	2022-2023	33.620000000000
phlMedian.a95	2023-2024	32.255000000000
phlMedian.a95	2024-2025	31.665000000000
phlMedian.ada	2013-2014	93.700000000000
phlMedian.ada	2014-2015	94.700000000000
phlMedian.ada	2015-2016	94.400000000000
phlMedian.ada	2016-2017	91.400000000000
phlMedian.ada	2017-2018	92.100000000000
phlMedian.ada	2018-2019	92.300000000000
phlMedian.ada	2019-2020	93.000000000000
phlMedian.ada	2020-2021	89.060000000000
phlMedian.ada	2021-2022	88.335000000000
phlMedian.ada	2022-2023	89.170000000000
phlMedian.ada	2023-2024	89.175000000000
phlMedian.ada	2024-2025	89.915000000000
phlMedian.chronic	2020-2021	33.800000000000
phlMedian.chronic	2021-2022	44.055000000000
phlMedian.chronic	2022-2023	40.030000000000
phlMedian.chronic	2023-2024	39.580000000000
phlMedian.chronic	2024-2025	39.040000000000

---
title: "Philadelphia and Tuscaloosa on a common scale — the calculation"
output: md_document
---

```{r setup, include = FALSE}
# error = FALSE so that knitting *fails* on an error rather than embedding it in
# the document.
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", error = FALSE)
options(stringsAsFactors = FALSE)
```

Pennsylvania and Alabama use different tests with different cut scores, so the
two states' published "percent proficient" figures cannot be set side by side.
The Stanford Education Data Archive links every state's tests to one national
scale, which is what makes a West Philadelphia K-8 and a Tuscaloosa City school
comparable at all.

Both measures here are deviations from the national average, **in grade
levels**, and both can be negative:

* **level** (`cs_mn_avg_eb`) — how far above or below the national average a
  school's students score. Zero is the national average.
* **growth** (`cs_mn_grd_eb`) — how much they gain per grade compared with
  typical. Zero is a typical year's progress.

The `_eb` columns are the empirical-Bayes estimates, shrunk toward the mean —
the right choice at school level, where small samples make raw estimates noisy.

Three places are on this scale: eleven West Philadelphia K-8 schools, the
schools of Tuscaloosa City, and the Brooklyn schools around Park Slope and
Prospect Heights. New York needs an id crosswalk first — SEDA rolls its 32
geographic districts into one synthetic citywide LEA, where the federal
collections keep them apart.

Brooklyn is computed twice. The eleven schools around Park Slope were named
individually, and a median across schools somebody named is not a median for
the borough they sit in, so the whole borough is carried alongside them.

One of three implementations; the others are the original TypeScript in `src/`
and `calc/seda.py`. All three write the contract in `calc/CONTRACT.md`.

```{r constants}
ROOT <- Sys.getenv("SDP_ROOT", ".")
SEDA <- file.path(ROOT, "seda")

# Where the low-income measure runs out of room; see calc/CONTRACT.md.
poverty_ceiling <- 0.95

phila_lea <- "4218990"
tusc_city_lea <- "103360"
tusc_county_lea <- "103390"

district_names <- c("4218990" = "Philadelphia City SD",
                    "103360" = "Tuscaloosa City",
                    "103390" = "Tuscaloosa County")

# SEDA files all of New York City under one synthetic citywide LEA rather than
# the 32 geographic-district LEAs the federal collections use. The five-digit
# school number is the same in both, so an id crosswalks by swapping the prefix.
nyc_seda_lea <- "3620580"

nyc_seda_id <- function(nces) {
  if (!grepl("^[0-9]{12}$", nces)) stop("nyc_seda_id: not a 12-digit NCES id: ", nces)
  paste0(nyc_seda_lea, substr(nces, 8, 12))
}

nyc <- data.frame(
  nces = c("360009102074", "360009102069", "360009202818", "360009202403", "360009202175",
           "360009202226", "360009202557", "360009202072", "360009206406", "360009102490",
           "360009102793"),
  label = c("PS 11 Purvis J. Behan", "PS 9 Teunis G. Bergen", "PS 321 William Penn",
            "PS 107 John W. Kimball", "PS 39 Henry Bristow", "MS 51 William Alexander",
            "PS 154 Windsor Terrace", "PS 10 Magnet School", "PS 118 Maurice Sendak",
            "PS 133 William A. Butler", "PS 282 Park Slope")
)

k8 <- data.frame(
  nces = c("421899006528", "421899000619", "421899006530", "421899003782", "421899003783",
           "421899004732", "421899003649", "421899003650", "421899005186", "421899003805",
           "421899003618"),
  label = c("Harrington", "Penn Alexander", "Lea", "Comegys", "Powel", "Alain Locke",
            "Meredith", "Nebinger", "Greenfield", "Marian Anderson", "Bache-Martin")
)

# LEAs the current federal enrolment counts are read for, and CCD's codes for
# "all students" and "Black students".
demographics_dir <- file.path(ROOT, "demographics")
ccd_leas <- c("4218990", "0103360", "3600091", "3600092")
ccd_all <- 99
ccd_black <- 2

sources <- list(
  achievement = list(file = "seda_school_pool_cs_4.1.csv",
                     cols = c("sedasch", "sedaschname", "cs_mn_avg_eb", "cs_mn_grd_eb")),
  covariates = list(file = "seda_cov_school_pool_4.1.csv",
                    cols = c("sedasch", "sedalea", "schcity", "schnam", "level", "perblk",
                             "totenrl", "perecd")),
  districts = list(file = "seda2022_dist.csv",
                   cols = c("sedaadmin", "sedaadminname", "subject", "subgroup",
                            "ys_mn19_eb", "ys_mn22_eb", "ys_chg_eb"))
)
```

## Reading

```{r read}
read_seda <- function(rel) {
  path <- file.path(SEDA, rel)
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-seda.sh")
  raw <- readBin(path, "raw", file.size(path))
  txt <- iconv(rawToChar(raw), from = "latin1", to = "UTF-8")
  read.csv(text = txt, colClasses = "character", check.names = FALSE, na.strings = NULL)
}

data <- lapply(sources, function(s) read_seda(s$file))
sapply(data, nrow)
```

## Reading a SEDA number

SEDA writes some values with no leading zero — `.075` rather than `0.075` — and
uses `NA` for a suppressed cell. The shape of the string is checked before it is
converted, rather than after: R's `as.numeric` and Python's `float` each accept
strings that JavaScript's `Number` does not, and the three have to agree about
what is not a number as well as about what is.

```{r num}
NUMERIC <- "^[+-]?([0-9]+\\.?[0-9]*|\\.[0-9]+)([eE][+-]?[0-9]+)?$"

num <- function(v) {
  t <- trimws(v)
  ok <- t != "" & toupper(t) != "NA" & grepl(NUMERIC, t)
  out <- rep(NA_real_, length(t))
  out[ok] <- as.numeric(t[ok])
  out[!is.finite(out)] <- NA_real_
  out
}
```

## Percentiles, and one place three languages disagree

A school's percentile is its position in the distribution of **all** schools in
the release, not among the few dozen on the page — the point is to place these
schools nationally.

The rounding needs saying out loud. JavaScript's `Math.round` sends halves up;
R's `round` and Python's `round` both send halves to the nearest even number. A
percentile that lands exactly on 46.5 would therefore come out 47 in the
original and 46 in a naive port. This is the single most likely place for three
implementations to disagree by one, so all three do the same explicit thing.

```{r percentile}
js_round <- function(x) floor(x + 0.5)

percentile_of <- function(sorted, v) {
  if (is.na(v) || !length(sorted)) return(NA_integer_)
  js_round((sum(sorted < v) / length(sorted)) * 100)
}

med <- function(v) {
  v <- v[!is.na(v)]
  if (!length(v)) return(NULL)
  s <- sort(v)
  mid <- length(s) %/% 2
  if (length(s) %% 2 == 1) s[mid + 1] else (s[mid] + s[mid + 1]) / 2
}
```

## Joining achievement to covariates

The achievement file carries the two estimates; the covariate file carries the
district, the grade span, enrolment and the demographics. Where a school id
appears twice in the covariates the last row wins, matching what an assignment
does in the other two implementations.

```{r join}
ach <- data$achievement
cov <- data$covariates

cov_last <- !duplicated(cov$sedasch, fromLast = TRUE)
cov_id <- cov$sedasch[cov_last]
cov_lea <- cov$sedalea[cov_last]
cov_city <- cov$schcity[cov_last]
cov_name <- cov$schnam[cov_last]
cov_level <- cov$level[cov_last]
cov_blk <- num(cov$perblk)[cov_last]
cov_enr <- num(cov$totenrl)[cov_last]

a_level <- num(ach$cs_mn_avg_eb)
a_growth <- num(ach$cs_mn_grd_eb)

levels_all <- sort(a_level[!is.na(a_level)])
growths_all <- sort(a_growth[!is.na(a_growth)])

join_seda <- function(keep_idx, labels) {
  hit <- match(ach$sedasch, cov_id)
  ok <- !is.na(a_level) & !is.na(hit)
  ok[ok] <- keep_idx(hit[ok])
  idx <- which(ok)

  lab <- labels[ach$sedasch[idx]]
  # `??` in the original coalesces on absence, not on emptiness: a blank school
  # name stays blank rather than falling through to the next candidate.
  nm <- ifelse(is.na(lab), trimws(cov_name[hit[idx]]), lab)

  out <- data.frame(
    id = ach$sedasch[idx],
    name = nm,
    district = trimws(cov_lea[hit[idx]]),
    gradeLevel = trimws(cov_level[hit[idx]]),
    level = a_level[idx],
    growth = a_growth[idx],
    pctBlack = cov_blk[hit[idx]],
    enrollment = cov_enr[hit[idx]]
  )
  out$levelPct <- sapply(out$level, function(v) percentile_of(levels_all, v))
  out$growthPct <- sapply(out$growth, function(v) percentile_of(growths_all, v))
  # Descending by level. A radix order is stable, as the original's sort is, so
  # schools with the same level keep their file order.
  out[order(-out$level, method = "radix"), , drop = FALSE]
}

labels <- setNames(k8$label, k8$nces)
nyc_ids <- vapply(nyc$nces, nyc_seda_id, character(1), USE.NAMES = FALSE)
nyc_labels <- setNames(nyc$label, nyc_ids)

phila <- join_seda(function(i) cov_id[i] %in% k8$nces, labels)
tuscaloosa <- join_seda(function(i) cov_lea[i] == tusc_city_lea, character(0))
brooklyn <- join_seda(function(i) cov_id[i] %in% nyc_ids, nyc_labels)
all_brooklyn <- join_seda(
  function(i) cov_lea[i] == nyc_seda_lea & toupper(cov_city[i]) == "BROOKLYN",
  character(0))

data.frame(group = c("phila", "tuscaloosa", "brooklyn", "allBrooklyn"),
           schools = c(nrow(phila), nrow(tuscaloosa), nrow(brooklyn), nrow(all_brooklyn)))
```

## What these schools look like now

The scores are pooled over 2009-2018, and so is the racial composition SEDA
ships beside them. Intakes move: a school that was three quarters Black in that
window may be about half now. The current federal count is read alongside, so
neither figure is ever published without the other.

Only the `sex = 99` rows are kept — the per-sex rows are the same students
counted again — and CCD writes suppressed cells as negative numbers, which must
not become a negative share.

```{r demographics}
ccd_rows <- list()
ccd_digest <- character(0)
for (lea in ccd_leas) {
  path <- file.path(demographics_dir, sprintf("enrollment_%s.json", lea))
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-demographics.sh")
  ccd_digest[[sprintf("enrollment_%s", lea)]] <-
    digest::digest(file = path, algo = "sha256")
  ccd_rows[[lea]] <- jsonlite::fromJSON(path, simplifyDataFrame = TRUE)$results
}
ccd <- do.call(rbind, ccd_rows)

keep <- ccd$sex == 99 & !is.na(ccd$ncessch) & !is.na(ccd$enrollment) & ccd$enrollment >= 0
ccd <- ccd[keep, , drop = FALSE]

tot <- ccd[ccd$race == ccd_all, , drop = FALSE]
blk <- ccd[ccd$race == ccd_black, , drop = FALSE]
now_total <- setNames(tot$enrollment[!duplicated(tot$ncessch, fromLast = TRUE)],
                      tot$ncessch[!duplicated(tot$ncessch, fromLast = TRUE)])
now_black <- setNames(blk$enrollment[!duplicated(blk$ncessch, fromLast = TRUE)],
                      blk$ncessch[!duplicated(blk$ncessch, fromLast = TRUE)])

# SEDA drops Alabama's leading zero and files New York under a synthetic
# citywide LEA; both have to be undone to reach the federal counts.
nyc_federal <- setNames(nyc$nces, nyc_ids)
federal_id <- function(sedasch) {
  if (!is.na(nyc_federal[sedasch])) return(unname(nyc_federal[sedasch]))
  if (nchar(sedasch) == 11) paste0("0", sedasch) else sedasch
}

length(unique(c(names(now_total), names(now_black))))
```

## Writing the contract

```{r contract}
NA_VALUE <- "NA"

fmt_int <- function(n) {
  n <- as.numeric(n)
  if (!is.finite(n) || n != round(n)) stop("contract: ", n, " is not an integer")
  sprintf("%.0f", n)
}

fmt_real <- function(x) {
  if (is.null(x) || length(x) == 0 || is.na(x)) return(NA_VALUE)
  if (!is.finite(x)) stop("contract: ", x, " is not finite")
  sprintf("%.12f", if (x == 0) 0 else x)
}

fmt_text <- function(s) {
  if (grepl("[\t\r\n]", s)) stop("contract: value contains a tab or newline: ", s)
  s
}

rows <- new.env(parent = emptyenv())
rows$metric <- character(0); rows$key <- character(0); rows$value <- character(0)
add <- function(metric, key, value) {
  rows$metric <- c(rows$metric, metric)
  rows$key <- c(rows$key, key)
  rows$value <- c(rows$value, value)
}

sha256 <- function(s) digest::digest(s, algo = "sha256", serialize = FALSE)
```

## The facts

```{r facts-input}
for (name in names(sources)) {
  df <- data[[name]]
  cols <- sources[[name]]$cols
  dump <- do.call(paste, c(lapply(cols, function(c) df[[c]]), sep = "\x1f"))
  add("input.rows", name, fmt_int(nrow(df)))
  add("input.digest", name, fmt_text(sha256(paste(dump, collapse = "\n"))))
}

for (name in names(ccd_digest)) add("input.filedigest", name, fmt_text(ccd_digest[[name]]))
add("input.rows", "ccd_enrollment", fmt_int(sum(sapply(ccd_rows, nrow))))
add("input.schools", "ccd_enrollment", fmt_int(length(unique(c(names(now_total), names(now_black))))))
```

One line per school, carrying exactly the values the per-school facts do. The
borough is four hundred schools, and writing it out would add nearly five
thousand lines and bury everything else, so it is checked by hashing this dump
instead. One differing field in one school changes the hash.

```{r facts-schools}
school_dump <- function(df) {
  lines <- character(nrow(df))
  for (i in seq_len(nrow(df))) {
    fed <- federal_id(df$id[i])
    lines[i] <- paste(
      df$id[i], df$name[i], df$district[i], df$gradeLevel[i],
      fmt_real(df$level[i]), fmt_real(df$growth[i]),
      if (is.na(df$levelPct[i])) NA_VALUE else fmt_int(df$levelPct[i]),
      if (is.na(df$growthPct[i])) NA_VALUE else fmt_int(df$growthPct[i]),
      fmt_real(df$pctBlack[i]), fmt_real(df$enrollment[i]),
      if (is.na(now_total[fed])) NA_VALUE else fmt_int(unname(now_total[fed])),
      if (is.na(now_black[fed])) NA_VALUE else fmt_int(unname(now_black[fed])),
      fmt_int(i - 1),
      sep = "\x1f")
  }
  paste(lines, collapse = "\n")
}

emit_group <- function(group, df, per_school) {
  add("seda.n", group, fmt_int(nrow(df)))
  # The page draws a median rule on every row and plots growth for whichever
  # schools have it, so both are contract facts rather than page arithmetic.
  add("seda.median", group, fmt_real(med(df$level)))
  add("seda.withGrowth", group, fmt_int(sum(!is.na(df$growth))))
  add("seda.digest", group, fmt_text(sha256(school_dump(df))))
  if (!per_school) return(invisible(NULL))
  for (i in seq_len(nrow(df))) {
    key <- paste0(group, "|", df$id[i])
    add("seda.level", key, fmt_real(df$level[i]))
    add("seda.growth", key, fmt_real(df$growth[i]))
    add("seda.levelPct", key, if (is.na(df$levelPct[i])) NA_VALUE else fmt_int(df$levelPct[i]))
    add("seda.growthPct", key, if (is.na(df$growthPct[i])) NA_VALUE else fmt_int(df$growthPct[i]))
    add("seda.pctBlack", key, fmt_real(df$pctBlack[i]))
    add("seda.enrollment", key, fmt_real(df$enrollment[i]))
    add("seda.name", key, fmt_text(df$name[i]))
    add("seda.district", key, fmt_text(df$district[i]))
    add("seda.gradeLevel", key, fmt_text(df$gradeLevel[i]))
    # The page lists schools by level, descending. Emitting the position checks
    # the ordering too, not just the values.
    add("seda.rank", key, fmt_int(i - 1))
    fed <- federal_id(df$id[i])
    add("seda.nowEnrollment", key,
        if (is.na(now_total[fed])) NA_VALUE else fmt_int(unname(now_total[fed])))
    add("seda.nowBlack", key,
        if (is.na(now_black[fed])) NA_VALUE else fmt_int(unname(now_black[fed])))
  }
}
emit_group("phila", phila, TRUE)
emit_group("tuscaloosa", tuscaloosa, TRUE)
emit_group("brooklyn", brooklyn, TRUE)
emit_group("allBrooklyn", all_brooklyn, FALSE)

add("national.levels", "-", fmt_int(length(levels_all)))
add("national.growths", "-", fmt_int(length(growths_all)))

# How much of each district SEDA can say anything about. New York is the reason
# this is in the contract: it has a growth estimate for about a third of its
# schools, against most of Philadelphia's and all of Tuscaloosa City's.
for (pl in list(c("Philadelphia", phila_lea),
                c("Tuscaloosa_City", tusc_city_lea),
                c("New_York_City", nyc_seda_lea))) {
  ids <- cov$sedasch[cov$sedalea == pl[2]]
  m <- ach$sedasch %in% ids
  add("coverage.schools", pl[1], fmt_int(sum(m)))
  add("coverage.level", pl[1], fmt_int(sum(m & !is.na(a_level))))
  add("coverage.growth", pl[1], fmt_int(sum(m & !is.na(a_growth))))
}

# New York is absent from the 2022 district release entirely.
add("district.newYorkRows", "-", fmt_int(sum(data$districts$stateabb == "NY")))
```

## Districts, and the pandemic drop

SEDA's 2022 release is at district level and carries two snapshots — 2019 and
2022 — with the change between them. The national median district gives the drop
a reference point, so that a fall reads as bigger or smaller than everyone
else's rather than simply as a fall.

```{r facts-districts}
dist <- data$districts
subject_of <- function(s) ifelse(s == "mth", "math", "reading")

sel <- dist$subgroup == "all" & dist$sedaadmin %in% names(district_names)
d19 <- num(dist$ys_mn19_eb); d22 <- num(dist$ys_mn22_eb); dch <- num(dist$ys_chg_eb)
for (i in which(sel)) {
  key <- paste0(dist$sedaadmin[i], "|", subject_of(dist$subject[i]))
  add("district.level2019", key, fmt_real(d19[i]))
  add("district.level2022", key, fmt_real(d22[i]))
  add("district.change", key, fmt_real(dch[i]))
}
for (id in names(district_names)) add("district.name", id, fmt_text(district_names[[id]]))

for (subj in c("mth", "rla")) {
  m <- dist$subgroup == "all" & dist$subject == subj
  add("usMedian.level2019", subject_of(subj), fmt_real(med(d19[m])))
  add("usMedian.change", subject_of(subj), fmt_real(med(dch[m])))
  # Every district row for the subject, including the ones with no estimate:
  # this is the denominator the page prints.
  add("usMedian.n", subject_of(subj), fmt_int(sum(m)))
}

data.frame(
  district = district_names[dist$sedaadmin[sel]],
  subject = subject_of(dist$subject[sel]),
  level2019 = round(d19[sel], 2),
  change = round(dch[sel], 2),
  row.names = NULL
)
```

## How much of "level" is just intake

The page says that level tracks family income closely while growth barely does.
That is a claim about the whole release, not about the schools on the page, so
it is computed against every school in the file rather than cited.

`perecd` is the share of a school's students who are economically
disadvantaged. Pearson's r is an explicit loop rather than `cor()`, so all three
implementations add their terms in the same order.

```{r poverty}
pearson <- function(xs, ys) {
  n <- min(length(xs), length(ys))
  if (n < 2) return(NULL)
  sx <- 0; sy <- 0
  for (i in seq_len(n)) { sx <- sx + xs[i]; sy <- sy + ys[i] }
  mx <- sx / n; my <- sy / n
  num_ <- 0; dx <- 0; dy <- 0
  for (i in seq_len(n)) {
    a <- xs[i] - mx; b <- ys[i] - my
    num_ <- num_ + a * b; dx <- dx + a * a; dy <- dy + b * b
  }
  if (dx == 0 || dy == 0) return(NULL)
  num_ / sqrt(dx * dy)
}

perecd <- num(cov$perecd)
# Last-wins over the rows that qualify, which is what a repeated assignment does.
q <- which(!is.na(cov$sedasch) & !is.na(perecd) & perecd >= 0 & perecd <= 1)
pov_id <- cov$sedasch[q]
pov_val <- perecd[q]
keep <- !duplicated(pov_id, fromLast = TRUE)
pov_id <- pov_id[keep]; pov_val <- pov_val[keep]
add("poverty.schools", "-", fmt_int(length(pov_id)))

# Where `perecd` stops measuring anything. It is built from free-and-reduced-
# lunch eligibility, so a district on universal free meals books nearly every
# school at the maximum whatever its families earn. The share of each set at the
# ceiling is the only way to see which rows have run out of room, so it is a
# contract fact rather than something the page works out.
add("poverty.ceiling", "-", fmt_int(sum(pov_val >= poverty_ceiling)))
add("poverty.ceilingAt", "-", fmt_real(poverty_ceiling))

hit <- match(ach$sedasch, pov_id)
correlations <- list()
for (nm in c("level", "growth")) {
  v <- if (nm == "level") a_level else a_growth
  ok <- !is.na(v) & !is.na(hit)
  xs <- pov_val[hit[ok]]
  ys <- v[ok]
  correlations[[nm]] <- list(r = pearson(xs, ys), n = length(xs))
  add(sprintf("poverty.%s.r", nm), "-", fmt_real(correlations[[nm]]$r))
  add(sprintf("poverty.%s.n", nm), "-", fmt_int(correlations[[nm]]$n))
}

data.frame(
  measure = c("level vs share economically disadvantaged",
              "growth vs share economically disadvantaged"),
  r = c(round(correlations$level$r, 3), round(correlations$growth$r, 3)),
  n = c(correlations$level$n, correlations$growth$n)
)
```

## The sets the page compares

The intake table reads five sets off one national line. Its numbers were the
last part of the page not covered here, which is exactly the part a reader is
most likely to want checked.

```{r sets}
level_of <- setNames(a_level, ach$sedasch)
level_of <- level_of[!is.na(level_of)]
pov_map <- setNames(pov_val, pov_id)

set_stats <- function(name, ids) {
  levels_ <- unname(level_of[ids]); levels_ <- levels_[!is.na(levels_)]
  povs <- unname(pov_map[ids]); povs <- povs[!is.na(povs)]
  add("set.n", name, fmt_int(length(levels_)))
  add("set.level", name, fmt_real(med(levels_)))
  add("set.poverty", name, fmt_real(med(povs)))
  # The denominator is the schools with a share at all — the same set the median
  # is taken over, not `set.n`, which counts level estimates.
  add("set.ceiling", name, fmt_int(sum(povs >= poverty_ceiling)))
  add("set.ceilingOf", name, fmt_int(length(povs)))
}

cov_ids_where <- function(keep) cov$sedasch[keep]

set_stats("philaEleven", k8$nces)
set_stats("philaAll", cov_ids_where(cov$sedalea == phila_lea))
set_stats("brooklynTen", nyc_ids)
set_stats("brooklynAll",
          cov_ids_where(cov$sedalea == nyc_seda_lea & toupper(cov$schcity) == "BROOKLYN"))
set_stats("tuscaloosaCity", cov_ids_where(cov$sedalea == tusc_city_lea))
```

Least squares of level on the low-income share, across every US school — the
line the table's "predicted" column is read off. Written as an explicit loop for
the same reason Pearson's r is.

```{r fit}
ok_fit <- !is.na(a_level) & !is.na(hit)
xs <- pov_val[hit[ok_fit]]
ys <- a_level[ok_fit]
n_fit <- length(xs)
sx <- 0; sy <- 0
for (i in seq_len(n_fit)) { sx <- sx + xs[i]; sy <- sy + ys[i] }
mx <- sx / n_fit; my <- sy / n_fit
sxy <- 0; sxx <- 0
for (i in seq_len(n_fit)) {
  dx <- xs[i] - mx
  sxy <- sxy + dx * (ys[i] - my)
  sxx <- sxx + dx * dx
}
slope <- sxy / sxx
add("povertyFit.slope", "-", fmt_real(slope))
add("povertyFit.intercept", "-", fmt_real(my - slope * mx))
add("povertyFit.n", "-", fmt_int(n_fit))

data.frame(intercept = round(my - slope * mx, 3), slope = round(slope, 3), n = n_fit)
```

## Output

```{r write}
stopifnot(!any(duplicated(paste0(rows$metric, "\t", rows$key))))
ord <- order(paste0(rows$metric, "\t", rows$key), method = "radix")
lines <- c("metric\tkey\tvalue",
           paste(rows$metric[ord], rows$key[ord], rows$value[ord], sep = "\t"))

out_dir <- file.path(ROOT, "calc", "out")
dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
con <- file(file.path(out_dir, "seda-r.tsv"), open = "wb")
writeLines(lines, con, sep = "\n", useBytes = TRUE)
close(con)

cat(sprintf("facts: %d\nsha256: %s\n", length(lines) - 1,
            sha256(paste0(paste(lines, collapse = "\n"), "\n"))))
```

The TypeScript and Python implementations print the same hash.

---
title: "Philadelphia and Tuscaloosa on a common scale — the calculation"
output: md_document
---



Pennsylvania and Alabama use different tests with different cut scores, so the
two states' published "percent proficient" figures cannot be set side by side.
The Stanford Education Data Archive links every state's tests to one national
scale, which is what makes a West Philadelphia K-8 and a Tuscaloosa City school
comparable at all.

Both measures here are deviations from the national average, **in grade
levels**, and both can be negative:

* **level** (`cs_mn_avg_eb`) — how far above or below the national average a
  school's students score. Zero is the national average.
* **growth** (`cs_mn_grd_eb`) — how much they gain per grade compared with
  typical. Zero is a typical year's progress.

The `_eb` columns are the empirical-Bayes estimates, shrunk toward the mean —
the right choice at school level, where small samples make raw estimates noisy.

Three places are on this scale: eleven West Philadelphia K-8 schools, the
schools of Tuscaloosa City, and the Brooklyn schools around Park Slope and
Prospect Heights. New York needs an id crosswalk first — SEDA rolls its 32
geographic districts into one synthetic citywide LEA, where the federal
collections keep them apart.

Brooklyn is computed twice. The eleven schools around Park Slope were named
individually, and a median across schools somebody named is not a median for
the borough they sit in, so the whole borough is carried alongside them.

One of three implementations; the others are the original TypeScript in `src/`
and `calc/seda.py`. All three write the contract in `calc/CONTRACT.md`.


```r
ROOT <- Sys.getenv("SDP_ROOT", ".")
SEDA <- file.path(ROOT, "seda")

# Where the low-income measure runs out of room; see calc/CONTRACT.md.
poverty_ceiling <- 0.95

phila_lea <- "4218990"
tusc_city_lea <- "103360"
tusc_county_lea <- "103390"

district_names <- c("4218990" = "Philadelphia City SD",
                    "103360" = "Tuscaloosa City",
                    "103390" = "Tuscaloosa County")

# SEDA files all of New York City under one synthetic citywide LEA rather than
# the 32 geographic-district LEAs the federal collections use. The five-digit
# school number is the same in both, so an id crosswalks by swapping the prefix.
nyc_seda_lea <- "3620580"

nyc_seda_id <- function(nces) {
  if (!grepl("^[0-9]{12}$", nces)) stop("nyc_seda_id: not a 12-digit NCES id: ", nces)
  paste0(nyc_seda_lea, substr(nces, 8, 12))
}

nyc <- data.frame(
  nces = c("360009102074", "360009102069", "360009202818", "360009202403", "360009202175",
           "360009202226", "360009202557", "360009202072", "360009206406", "360009102490",
           "360009102793"),
  label = c("PS 11 Purvis J. Behan", "PS 9 Teunis G. Bergen", "PS 321 William Penn",
            "PS 107 John W. Kimball", "PS 39 Henry Bristow", "MS 51 William Alexander",
            "PS 154 Windsor Terrace", "PS 10 Magnet School", "PS 118 Maurice Sendak",
            "PS 133 William A. Butler", "PS 282 Park Slope")
)

k8 <- data.frame(
  nces = c("421899006528", "421899000619", "421899006530", "421899003782", "421899003783",
           "421899004732", "421899003649", "421899003650", "421899005186", "421899003805",
           "421899003618"),
  label = c("Harrington", "Penn Alexander", "Lea", "Comegys", "Powel", "Alain Locke",
            "Meredith", "Nebinger", "Greenfield", "Marian Anderson", "Bache-Martin")
)

# LEAs the current federal enrolment counts are read for, and CCD's codes for
# "all students" and "Black students".
demographics_dir <- file.path(ROOT, "demographics")
ccd_leas <- c("4218990", "0103360", "3600091", "3600092")
ccd_all <- 99
ccd_black <- 2

sources <- list(
  achievement = list(file = "seda_school_pool_cs_4.1.csv",
                     cols = c("sedasch", "sedaschname", "cs_mn_avg_eb", "cs_mn_grd_eb")),
  covariates = list(file = "seda_cov_school_pool_4.1.csv",
                    cols = c("sedasch", "sedalea", "schcity", "schnam", "level", "perblk",
                             "totenrl", "perecd")),
  districts = list(file = "seda2022_dist.csv",
                   cols = c("sedaadmin", "sedaadminname", "subject", "subgroup",
                            "ys_mn19_eb", "ys_mn22_eb", "ys_chg_eb"))
)
```

## Reading


```r
read_seda <- function(rel) {
  path <- file.path(SEDA, rel)
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-seda.sh")
  raw <- readBin(path, "raw", file.size(path))
  txt <- iconv(rawToChar(raw), from = "latin1", to = "UTF-8")
  read.csv(text = txt, colClasses = "character", check.names = FALSE, na.strings = NULL)
}

data <- lapply(sources, function(s) read_seda(s$file))
sapply(data, nrow)
#> achievement  covariates   districts 
#>       82488       89350       32804
```

## Reading a SEDA number

SEDA writes some values with no leading zero — `.075` rather than `0.075` — and
uses `NA` for a suppressed cell. The shape of the string is checked before it is
converted, rather than after: R's `as.numeric` and Python's `float` each accept
strings that JavaScript's `Number` does not, and the three have to agree about
what is not a number as well as about what is.


```r
NUMERIC <- "^[+-]?([0-9]+\\.?[0-9]*|\\.[0-9]+)([eE][+-]?[0-9]+)?$"

num <- function(v) {
  t <- trimws(v)
  ok <- t != "" & toupper(t) != "NA" & grepl(NUMERIC, t)
  out <- rep(NA_real_, length(t))
  out[ok] <- as.numeric(t[ok])
  out[!is.finite(out)] <- NA_real_
  out
}
```

## Percentiles, and one place three languages disagree

A school's percentile is its position in the distribution of **all** schools in
the release, not among the few dozen on the page — the point is to place these
schools nationally.

The rounding needs saying out loud. JavaScript's `Math.round` sends halves up;
R's `round` and Python's `round` both send halves to the nearest even number. A
percentile that lands exactly on 46.5 would therefore come out 47 in the
original and 46 in a naive port. This is the single most likely place for three
implementations to disagree by one, so all three do the same explicit thing.


```r
js_round <- function(x) floor(x + 0.5)

percentile_of <- function(sorted, v) {
  if (is.na(v) || !length(sorted)) return(NA_integer_)
  js_round((sum(sorted < v) / length(sorted)) * 100)
}

med <- function(v) {
  v <- v[!is.na(v)]
  if (!length(v)) return(NULL)
  s <- sort(v)
  mid <- length(s) %/% 2
  if (length(s) %% 2 == 1) s[mid + 1] else (s[mid] + s[mid + 1]) / 2
}
```

## Joining achievement to covariates

The achievement file carries the two estimates; the covariate file carries the
district, the grade span, enrolment and the demographics. Where a school id
appears twice in the covariates the last row wins, matching what an assignment
does in the other two implementations.


```r
ach <- data$achievement
cov <- data$covariates

cov_last <- !duplicated(cov$sedasch, fromLast = TRUE)
cov_id <- cov$sedasch[cov_last]
cov_lea <- cov$sedalea[cov_last]
cov_city <- cov$schcity[cov_last]
cov_name <- cov$schnam[cov_last]
cov_level <- cov$level[cov_last]
cov_blk <- num(cov$perblk)[cov_last]
cov_enr <- num(cov$totenrl)[cov_last]

a_level <- num(ach$cs_mn_avg_eb)
a_growth <- num(ach$cs_mn_grd_eb)

levels_all <- sort(a_level[!is.na(a_level)])
growths_all <- sort(a_growth[!is.na(a_growth)])

join_seda <- function(keep_idx, labels) {
  hit <- match(ach$sedasch, cov_id)
  ok <- !is.na(a_level) & !is.na(hit)
  ok[ok] <- keep_idx(hit[ok])
  idx <- which(ok)

  lab <- labels[ach$sedasch[idx]]
  # `??` in the original coalesces on absence, not on emptiness: a blank school
  # name stays blank rather than falling through to the next candidate.
  nm <- ifelse(is.na(lab), trimws(cov_name[hit[idx]]), lab)

  out <- data.frame(
    id = ach$sedasch[idx],
    name = nm,
    district = trimws(cov_lea[hit[idx]]),
    gradeLevel = trimws(cov_level[hit[idx]]),
    level = a_level[idx],
    growth = a_growth[idx],
    pctBlack = cov_blk[hit[idx]],
    enrollment = cov_enr[hit[idx]]
  )
  out$levelPct <- sapply(out$level, function(v) percentile_of(levels_all, v))
  out$growthPct <- sapply(out$growth, function(v) percentile_of(growths_all, v))
  # Descending by level. A radix order is stable, as the original's sort is, so
  # schools with the same level keep their file order.
  out[order(-out$level, method = "radix"), , drop = FALSE]
}

labels <- setNames(k8$label, k8$nces)
nyc_ids <- vapply(nyc$nces, nyc_seda_id, character(1), USE.NAMES = FALSE)
nyc_labels <- setNames(nyc$label, nyc_ids)

phila <- join_seda(function(i) cov_id[i] %in% k8$nces, labels)
tuscaloosa <- join_seda(function(i) cov_lea[i] == tusc_city_lea, character(0))
brooklyn <- join_seda(function(i) cov_id[i] %in% nyc_ids, nyc_labels)
all_brooklyn <- join_seda(
  function(i) cov_lea[i] == nyc_seda_lea & toupper(cov_city[i]) == "BROOKLYN",
  character(0))

data.frame(group = c("phila", "tuscaloosa", "brooklyn", "allBrooklyn"),
           schools = c(nrow(phila), nrow(tuscaloosa), nrow(brooklyn), nrow(all_brooklyn)))
#>         group schools
#> 1       phila      11
#> 2  tuscaloosa      18
#> 3    brooklyn      10
#> 4 allBrooklyn     400
```

## What these schools look like now

The scores are pooled over 2009-2018, and so is the racial composition SEDA
ships beside them. Intakes move: a school that was three quarters Black in that
window may be about half now. The current federal count is read alongside, so
neither figure is ever published without the other.

Only the `sex = 99` rows are kept — the per-sex rows are the same students
counted again — and CCD writes suppressed cells as negative numbers, which must
not become a negative share.


```r
ccd_rows <- list()
ccd_digest <- character(0)
for (lea in ccd_leas) {
  path <- file.path(demographics_dir, sprintf("enrollment_%s.json", lea))
  if (!file.exists(path)) stop("missing ", path, " - run ./fetch-demographics.sh")
  ccd_digest[[sprintf("enrollment_%s", lea)]] <-
    digest::digest(file = path, algo = "sha256")
  ccd_rows[[lea]] <- jsonlite::fromJSON(path, simplifyDataFrame = TRUE)$results
}
ccd <- do.call(rbind, ccd_rows)

keep <- ccd$sex == 99 & !is.na(ccd$ncessch) & !is.na(ccd$enrollment) & ccd$enrollment >= 0
ccd <- ccd[keep, , drop = FALSE]

tot <- ccd[ccd$race == ccd_all, , drop = FALSE]
blk <- ccd[ccd$race == ccd_black, , drop = FALSE]
now_total <- setNames(tot$enrollment[!duplicated(tot$ncessch, fromLast = TRUE)],
                      tot$ncessch[!duplicated(tot$ncessch, fromLast = TRUE)])
now_black <- setNames(blk$enrollment[!duplicated(blk$ncessch, fromLast = TRUE)],
                      blk$ncessch[!duplicated(blk$ncessch, fromLast = TRUE)])

# SEDA drops Alabama's leading zero and files New York under a synthetic
# citywide LEA; both have to be undone to reach the federal counts.
nyc_federal <- setNames(nyc$nces, nyc_ids)
federal_id <- function(sedasch) {
  if (!is.na(nyc_federal[sedasch])) return(unname(nyc_federal[sedasch]))
  if (nchar(sedasch) == 11) paste0("0", sedasch) else sedasch
}

length(unique(c(names(now_total), names(now_black))))
#> [1] 330
```

## Writing the contract


```r
NA_VALUE <- "NA"

fmt_int <- function(n) {
  n <- as.numeric(n)
  if (!is.finite(n) || n != round(n)) stop("contract: ", n, " is not an integer")
  sprintf("%.0f", n)
}

fmt_real <- function(x) {
  if (is.null(x) || length(x) == 0 || is.na(x)) return(NA_VALUE)
  if (!is.finite(x)) stop("contract: ", x, " is not finite")
  sprintf("%.12f", if (x == 0) 0 else x)
}

fmt_text <- function(s) {
  if (grepl("[\t\r\n]", s)) stop("contract: value contains a tab or newline: ", s)
  s
}

rows <- new.env(parent = emptyenv())
rows$metric <- character(0); rows$key <- character(0); rows$value <- character(0)
add <- function(metric, key, value) {
  rows$metric <- c(rows$metric, metric)
  rows$key <- c(rows$key, key)
  rows$value <- c(rows$value, value)
}

sha256 <- function(s) digest::digest(s, algo = "sha256", serialize = FALSE)
```

## The facts


```r
for (name in names(sources)) {
  df <- data[[name]]
  cols <- sources[[name]]$cols
  dump <- do.call(paste, c(lapply(cols, function(c) df[[c]]), sep = "\x1f"))
  add("input.rows", name, fmt_int(nrow(df)))
  add("input.digest", name, fmt_text(sha256(paste(dump, collapse = "\n"))))
}

for (name in names(ccd_digest)) add("input.filedigest", name, fmt_text(ccd_digest[[name]]))
add("input.rows", "ccd_enrollment", fmt_int(sum(sapply(ccd_rows, nrow))))
add("input.schools", "ccd_enrollment", fmt_int(length(unique(c(names(now_total), names(now_black))))))
```

One line per school, carrying exactly the values the per-school facts do. The
borough is four hundred schools, and writing it out would add nearly five
thousand lines and bury everything else, so it is checked by hashing this dump
instead. One differing field in one school changes the hash.


```r
school_dump <- function(df) {
  lines <- character(nrow(df))
  for (i in seq_len(nrow(df))) {
    fed <- federal_id(df$id[i])
    lines[i] <- paste(
      df$id[i], df$name[i], df$district[i], df$gradeLevel[i],
      fmt_real(df$level[i]), fmt_real(df$growth[i]),
      if (is.na(df$levelPct[i])) NA_VALUE else fmt_int(df$levelPct[i]),
      if (is.na(df$growthPct[i])) NA_VALUE else fmt_int(df$growthPct[i]),
      fmt_real(df$pctBlack[i]), fmt_real(df$enrollment[i]),
      if (is.na(now_total[fed])) NA_VALUE else fmt_int(unname(now_total[fed])),
      if (is.na(now_black[fed])) NA_VALUE else fmt_int(unname(now_black[fed])),
      fmt_int(i - 1),
      sep = "\x1f")
  }
  paste(lines, collapse = "\n")
}

emit_group <- function(group, df, per_school) {
  add("seda.n", group, fmt_int(nrow(df)))
  # The page draws a median rule on every row and plots growth for whichever
  # schools have it, so both are contract facts rather than page arithmetic.
  add("seda.median", group, fmt_real(med(df$level)))
  add("seda.withGrowth", group, fmt_int(sum(!is.na(df$growth))))
  add("seda.digest", group, fmt_text(sha256(school_dump(df))))
  if (!per_school) return(invisible(NULL))
  for (i in seq_len(nrow(df))) {
    key <- paste0(group, "|", df$id[i])
    add("seda.level", key, fmt_real(df$level[i]))
    add("seda.growth", key, fmt_real(df$growth[i]))
    add("seda.levelPct", key, if (is.na(df$levelPct[i])) NA_VALUE else fmt_int(df$levelPct[i]))
    add("seda.growthPct", key, if (is.na(df$growthPct[i])) NA_VALUE else fmt_int(df$growthPct[i]))
    add("seda.pctBlack", key, fmt_real(df$pctBlack[i]))
    add("seda.enrollment", key, fmt_real(df$enrollment[i]))
    add("seda.name", key, fmt_text(df$name[i]))
    add("seda.district", key, fmt_text(df$district[i]))
    add("seda.gradeLevel", key, fmt_text(df$gradeLevel[i]))
    # The page lists schools by level, descending. Emitting the position checks
    # the ordering too, not just the values.
    add("seda.rank", key, fmt_int(i - 1))
    fed <- federal_id(df$id[i])
    add("seda.nowEnrollment", key,
        if (is.na(now_total[fed])) NA_VALUE else fmt_int(unname(now_total[fed])))
    add("seda.nowBlack", key,
        if (is.na(now_black[fed])) NA_VALUE else fmt_int(unname(now_black[fed])))
  }
}
emit_group("phila", phila, TRUE)
emit_group("tuscaloosa", tuscaloosa, TRUE)
emit_group("brooklyn", brooklyn, TRUE)
emit_group("allBrooklyn", all_brooklyn, FALSE)

add("national.levels", "-", fmt_int(length(levels_all)))
add("national.growths", "-", fmt_int(length(growths_all)))

# How much of each district SEDA can say anything about. New York is the reason
# this is in the contract: it has a growth estimate for about a third of its
# schools, against most of Philadelphia's and all of Tuscaloosa City's.
for (pl in list(c("Philadelphia", phila_lea),
                c("Tuscaloosa_City", tusc_city_lea),
                c("New_York_City", nyc_seda_lea))) {
  ids <- cov$sedasch[cov$sedalea == pl[2]]
  m <- ach$sedasch %in% ids
  add("coverage.schools", pl[1], fmt_int(sum(m)))
  add("coverage.level", pl[1], fmt_int(sum(m & !is.na(a_level))))
  add("coverage.growth", pl[1], fmt_int(sum(m & !is.na(a_growth))))
}

# New York is absent from the 2022 district release entirely.
add("district.newYorkRows", "-", fmt_int(sum(data$districts$stateabb == "NY")))
```

## Districts, and the pandemic drop

SEDA's 2022 release is at district level and carries two snapshots — 2019 and
2022 — with the change between them. The national median district gives the drop
a reference point, so that a fall reads as bigger or smaller than everyone
else's rather than simply as a fall.


```r
dist <- data$districts
subject_of <- function(s) ifelse(s == "mth", "math", "reading")

sel <- dist$subgroup == "all" & dist$sedaadmin %in% names(district_names)
d19 <- num(dist$ys_mn19_eb); d22 <- num(dist$ys_mn22_eb); dch <- num(dist$ys_chg_eb)
for (i in which(sel)) {
  key <- paste0(dist$sedaadmin[i], "|", subject_of(dist$subject[i]))
  add("district.level2019", key, fmt_real(d19[i]))
  add("district.level2022", key, fmt_real(d22[i]))
  add("district.change", key, fmt_real(dch[i]))
}
for (id in names(district_names)) add("district.name", id, fmt_text(district_names[[id]]))

for (subj in c("mth", "rla")) {
  m <- dist$subgroup == "all" & dist$subject == subj
  add("usMedian.level2019", subject_of(subj), fmt_real(med(d19[m])))
  add("usMedian.change", subject_of(subj), fmt_real(med(dch[m])))
  # Every district row for the subject, including the ones with no estimate:
  # this is the denominator the page prints.
  add("usMedian.n", subject_of(subj), fmt_int(sum(m)))
}

data.frame(
  district = district_names[dist$sedaadmin[sel]],
  subject = subject_of(dist$subject[sel]),
  level2019 = round(d19[sel], 2),
  change = round(dch[sel], 2),
  row.names = NULL
)
#>               district subject level2019 change
#> 1      Tuscaloosa City    math     -0.43  -0.10
#> 2      Tuscaloosa City reading     -0.30  -0.04
#> 3    Tuscaloosa County    math     -0.38  -0.03
#> 4    Tuscaloosa County reading     -0.27   0.03
#> 5 Philadelphia City SD    math     -0.60  -0.26
#> 6 Philadelphia City SD reading     -0.52  -0.08
```

## How much of "level" is just intake

The page says that level tracks family income closely while growth barely does.
That is a claim about the whole release, not about the schools on the page, so
it is computed against every school in the file rather than cited.

`perecd` is the share of a school's students who are economically
disadvantaged. Pearson's r is an explicit loop rather than `cor()`, so all three
implementations add their terms in the same order.


```r
pearson <- function(xs, ys) {
  n <- min(length(xs), length(ys))
  if (n < 2) return(NULL)
  sx <- 0; sy <- 0
  for (i in seq_len(n)) { sx <- sx + xs[i]; sy <- sy + ys[i] }
  mx <- sx / n; my <- sy / n
  num_ <- 0; dx <- 0; dy <- 0
  for (i in seq_len(n)) {
    a <- xs[i] - mx; b <- ys[i] - my
    num_ <- num_ + a * b; dx <- dx + a * a; dy <- dy + b * b
  }
  if (dx == 0 || dy == 0) return(NULL)
  num_ / sqrt(dx * dy)
}

perecd <- num(cov$perecd)
# Last-wins over the rows that qualify, which is what a repeated assignment does.
q <- which(!is.na(cov$sedasch) & !is.na(perecd) & perecd >= 0 & perecd <= 1)
pov_id <- cov$sedasch[q]
pov_val <- perecd[q]
keep <- !duplicated(pov_id, fromLast = TRUE)
pov_id <- pov_id[keep]; pov_val <- pov_val[keep]
add("poverty.schools", "-", fmt_int(length(pov_id)))

# Where `perecd` stops measuring anything. It is built from free-and-reduced-
# lunch eligibility, so a district on universal free meals books nearly every
# school at the maximum whatever its families earn. The share of each set at the
# ceiling is the only way to see which rows have run out of room, so it is a
# contract fact rather than something the page works out.
add("poverty.ceiling", "-", fmt_int(sum(pov_val >= poverty_ceiling)))
add("poverty.ceilingAt", "-", fmt_real(poverty_ceiling))

hit <- match(ach$sedasch, pov_id)
correlations <- list()
for (nm in c("level", "growth")) {
  v <- if (nm == "level") a_level else a_growth
  ok <- !is.na(v) & !is.na(hit)
  xs <- pov_val[hit[ok]]
  ys <- v[ok]
  correlations[[nm]] <- list(r = pearson(xs, ys), n = length(xs))
  add(sprintf("poverty.%s.r", nm), "-", fmt_real(correlations[[nm]]$r))
  add(sprintf("poverty.%s.n", nm), "-", fmt_int(correlations[[nm]]$n))
}

data.frame(
  measure = c("level vs share economically disadvantaged",
              "growth vs share economically disadvantaged"),
  r = c(round(correlations$level$r, 3), round(correlations$growth$r, 3)),
  n = c(correlations$level$n, correlations$growth$n)
)
#>                                      measure      r     n
#> 1  level vs share economically disadvantaged -0.814 72714
#> 2 growth vs share economically disadvantaged -0.118 51016
```

## The sets the page compares

The intake table reads five sets off one national line. Its numbers were the
last part of the page not covered here, which is exactly the part a reader is
most likely to want checked.


```r
level_of <- setNames(a_level, ach$sedasch)
level_of <- level_of[!is.na(level_of)]
pov_map <- setNames(pov_val, pov_id)

set_stats <- function(name, ids) {
  levels_ <- unname(level_of[ids]); levels_ <- levels_[!is.na(levels_)]
  povs <- unname(pov_map[ids]); povs <- povs[!is.na(povs)]
  add("set.n", name, fmt_int(length(levels_)))
  add("set.level", name, fmt_real(med(levels_)))
  add("set.poverty", name, fmt_real(med(povs)))
  # The denominator is the schools with a share at all — the same set the median
  # is taken over, not `set.n`, which counts level estimates.
  add("set.ceiling", name, fmt_int(sum(povs >= poverty_ceiling)))
  add("set.ceilingOf", name, fmt_int(length(povs)))
}

cov_ids_where <- function(keep) cov$sedasch[keep]

set_stats("philaEleven", k8$nces)
set_stats("philaAll", cov_ids_where(cov$sedalea == phila_lea))
set_stats("brooklynTen", nyc_ids)
set_stats("brooklynAll",
          cov_ids_where(cov$sedalea == nyc_seda_lea & toupper(cov$schcity) == "BROOKLYN"))
set_stats("tuscaloosaCity", cov_ids_where(cov$sedalea == tusc_city_lea))
```

Least squares of level on the low-income share, across every US school — the
line the table's "predicted" column is read off. Written as an explicit loop for
the same reason Pearson's r is.


```r
ok_fit <- !is.na(a_level) & !is.na(hit)
xs <- pov_val[hit[ok_fit]]
ys <- a_level[ok_fit]
n_fit <- length(xs)
sx <- 0; sy <- 0
for (i in seq_len(n_fit)) { sx <- sx + xs[i]; sy <- sy + ys[i] }
mx <- sx / n_fit; my <- sy / n_fit
sxy <- 0; sxx <- 0
for (i in seq_len(n_fit)) {
  dx <- xs[i] - mx
  sxy <- sxy + dx * (ys[i] - my)
  sxx <- sxx + dx * dx
}
slope <- sxy / sxx
add("povertyFit.slope", "-", fmt_real(slope))
add("povertyFit.intercept", "-", fmt_real(my - slope * mx))
add("povertyFit.n", "-", fmt_int(n_fit))

data.frame(intercept = round(my - slope * mx, 3), slope = round(slope, 3), n = n_fit)
#>   intercept  slope     n
#> 1     0.699 -1.334 72714
```

## Output


```r
stopifnot(!any(duplicated(paste0(rows$metric, "\t", rows$key))))
ord <- order(paste0(rows$metric, "\t", rows$key), method = "radix")
lines <- c("metric\tkey\tvalue",
           paste(rows$metric[ord], rows$key[ord], rows$value[ord], sep = "\t"))

out_dir <- file.path(ROOT, "calc", "out")
dir.create(out_dir, showWarnings = FALSE, recursive = TRUE)
con <- file(file.path(out_dir, "seda-r.tsv"), open = "wb")
writeLines(lines, con, sep = "\n", useBytes = TRUE)
close(con)

cat(sprintf("facts: %d\nsha256: %s\n", length(lines) - 1,
            sha256(paste0(paste(lines, collapse = "\n"), "\n"))))
#> facts: 570
#> sha256: 3c66ba511b07dc26a5871f2195b9bdb835f7f47d9fc3415b3185ccdc7935e669
```

The TypeScript and Python implementations print the same hash.

#!/usr/bin/env python3
"""
The Python implementation of the SEDA calculation.

Pennsylvania and Alabama use different tests with different cut scores, so their
published proficiency rates are not comparable. The Stanford Education Data
Archive links every state's tests to one national scale, which is what makes a
West Philadelphia K-8 and a Tuscaloosa City school comparable at all. Both
measures here are deviations from the national average, in grade levels.

Third of three implementations; see calc/CONTRACT.md. Standard library only.

Run: npm run calc:py   (writes calc/out/seda-python.tsv)
"""

from __future__ import annotations

import csv
import json
import math
import os
import re
import sys

from contract import NA, fmt_int, fmt_real, fmt_text, serialize, sha256

csv.field_size_limit(sys.maxsize)

HERE = os.path.dirname(os.path.abspath(__file__))
ROOT = os.path.dirname(HERE)
SEDA = os.path.join(ROOT, "seda")
DEMOGRAPHICS = os.path.join(ROOT, "demographics")

# LEAs the current federal enrolment counts are read for.
CCD_LEAS = ["4218990", "0103360", "3600091", "3600092"]

# CCD codes: 99 is all students, 2 is Black students.
CCD_ALL, CCD_BLACK = 99, 2

# Where the low-income measure runs out of room; see calc/CONTRACT.md.
POVERTY_CEILING = 0.95

PHILA_LEA = "4218990"
TUSC_CITY_LEA = "103360"
TUSC_COUNTY_LEA = "103390"

DISTRICT_NAMES = {
    PHILA_LEA: "Philadelphia City SD",
    TUSC_CITY_LEA: "Tuscaloosa City",
    TUSC_COUNTY_LEA: "Tuscaloosa County",
}

K8 = [
    ("421899006528", "Harrington"),
    ("421899000619", "Penn Alexander"),
    ("421899006530", "Lea"),
    ("421899003782", "Comegys"),
    ("421899003783", "Powel"),
    ("421899004732", "Alain Locke"),
    ("421899003649", "Meredith"),
    ("421899003650", "Nebinger"),
    ("421899005186", "Greenfield"),
    ("421899003805", "Marian Anderson"),
    ("421899003618", "Bache-Martin"),
]

# SEDA files all of New York City under one synthetic citywide LEA rather than
# the 32 geographic-district LEAs the federal collections use. The five-digit
# school number is the same in both, so an id crosswalks by swapping the prefix.
NYC_SEDA_LEA = "3620580"

NYC = [
    ("360009102074", "PS 11 Purvis J. Behan"),
    ("360009102069", "PS 9 Teunis G. Bergen"),
    ("360009202818", "PS 321 William Penn"),
    ("360009202403", "PS 107 John W. Kimball"),
    ("360009202175", "PS 39 Henry Bristow"),
    ("360009202226", "MS 51 William Alexander"),
    ("360009202557", "PS 154 Windsor Terrace"),
    ("360009202072", "PS 10 Magnet School"),
    ("360009206406", "PS 118 Maurice Sendak"),
    ("360009102490", "PS 133 William A. Butler"),
    ("360009102793", "PS 282 Park Slope"),
]


def nyc_seda_id(nces):
    """A federal NCES school id as SEDA writes it for New York City."""
    if not re.match(r"^\d{12}$", nces):
        raise ValueError("nyc_seda_id: not a 12-digit NCES id: %r" % (nces,))
    return NYC_SEDA_LEA + nces[-5:]


SOURCES = [
    ("achievement", "seda_school_pool_cs_4.1.csv",
     ["sedasch", "sedaschname", "cs_mn_avg_eb", "cs_mn_grd_eb"]),
    ("covariates", "seda_cov_school_pool_4.1.csv",
     ["sedasch", "sedalea", "schcity", "schnam", "level", "perblk", "totenrl", "perecd"]),
    ("districts", "seda2022_dist.csv",
     ["sedaadmin", "sedaadminname", "subject", "subgroup", "ys_mn19_eb", "ys_mn22_eb", "ys_chg_eb"]),
]


def current_enrollment(rows):
    """
    Fold the long-format CCD extract into one record per school.

    Only the sex = 99 rows are read: the per-sex rows are the same students
    counted again. Suppressed cells are written as negative numbers and must
    not become a negative share.
    """
    total, black = {}, {}
    for r in rows:
        if r.get("ncessch") is None or r.get("sex") != 99:
            continue
        e = r.get("enrollment")
        if not isinstance(e, (int, float)) or e < 0:
            continue
        if r.get("race") == CCD_ALL:
            total[r["ncessch"]] = e
        elif r.get("race") == CCD_BLACK:
            black[r["ncessch"]] = e
    out = {}
    for sid in set(total) | set(black):
        out[sid] = (total.get(sid), black.get(sid))
    return out


def read(rel):
    path = os.path.join(SEDA, rel)
    if not os.path.exists(path):
        raise SystemExit("missing %s — run ./fetch-seda.sh" % path)
    with open(path, encoding="latin-1", newline="") as fh:
        return list(csv.DictReader(fh))


# A JavaScript-compatible numeric literal. Python's float() accepts things
# JavaScript's Number() does not — "1_000", "nan", "infinity" — so the shape is
# checked before conversion rather than after.
_NUMERIC = re.compile(r"^[+-]?(\d+\.?\d*|\.\d+)([eE][+-]?\d+)?$")


def num(v):
    """Parse a SEDA decimal. Values may be blank, 'NA', or written as '.075'."""
    if v is None:
        return None
    t = v.strip()
    if t == "" or t.upper() == "NA" or not _NUMERIC.match(t):
        return None
    n = float(t)
    return n if math.isfinite(n) else None


def js_round(x):
    """
    JavaScript's Math.round: halves go up.

    Python's round() and R's round() both round halves to even, so a percentile
    of exactly 46.5 would come out 46 there and 47 in the original. This is the
    single most likely place for three implementations to disagree by one.
    """
    return math.floor(x + 0.5)


def percentile_of(sorted_values, v):
    """Where `v` falls in an ascending distribution, 0-100; None if unrankable."""
    if v is None or not sorted_values:
        return None
    lo, hi = 0, len(sorted_values)
    while lo < hi:
        mid = (lo + hi) // 2
        if sorted_values[mid] < v:
            lo = mid + 1
        else:
            hi = mid
    return js_round((lo / len(sorted_values)) * 100)


def median(values):
    if not values:
        return None
    s = sorted(values)
    mid = len(s) // 2
    return s[mid] if len(s) % 2 == 1 else (s[mid - 1] + s[mid]) / 2


def pearson(xs, ys):
    """Accumulated left to right, so all three implementations land on the same bits."""
    n = min(len(xs), len(ys))
    if n < 2:
        return None
    sx = sy = 0.0
    for i in range(n):
        sx += xs[i]
        sy += ys[i]
    mx, my = sx / n, sy / n
    num_ = dx = dy = 0.0
    for i in range(n):
        a, b = xs[i] - mx, ys[i] - my
        num_ += a * b
        dx += a * a
        dy += b * b
    if dx == 0 or dy == 0:
        return None
    return num_ / math.sqrt(dx * dy)


def join_seda(achievement, covariates, keep, labels, levels, growths):
    """
    Join achievement to covariates, keeping schools that pass `keep`.

    Percentiles are against the whole national file, not the kept subset: the
    point is to place these schools among all US schools.
    """
    cov_by_id = {}
    for c in covariates:
        cid = c.get("sedasch")
        if cid is not None:
            cov_by_id[cid] = c

    out = []
    for a in achievement:
        sid = a.get("sedasch")
        if sid is None:
            continue
        level = num(a.get("cs_mn_avg_eb"))
        if level is None:
            continue
        cov = cov_by_id.get(sid)
        if cov is None or not keep(cov):
            continue
        growth = num(a.get("cs_mn_grd_eb"))
        # `??` in the original coalesces on absence, not on emptiness: a blank
        # school name stays blank rather than falling through to the next one.
        if sid in labels:
            name = labels[sid]
        elif cov.get("schnam") is not None:
            name = cov["schnam"].strip()
        elif a.get("sedaschname") is not None:
            name = a["sedaschname"].strip()
        else:
            name = sid
        out.append({
            "id": sid,
            "name": name,
            "district": (cov.get("sedalea") or "").strip(),
            "gradeLevel": (cov.get("level") or "").strip(),
            "level": level,
            "growth": growth,
            "levelPct": percentile_of(levels, level),
            "growthPct": percentile_of(growths, growth),
            "pctBlack": num(cov.get("perblk")),
            "enrollment": num(cov.get("totenrl")),
        })
    # Descending by level. Python's sort is stable, as the original's is, so
    # schools with the same level keep their file order.
    out.sort(key=lambda s: -s["level"])
    return out


def build():
    rows = []

    def add(metric, key, value):
        rows.append((metric, key, value))

    parsed = {}
    for name, rel, columns in SOURCES:
        parsed[name] = read(rel)
        add("input.rows", name, fmt_int(len(parsed[name])))
        dump = "\n".join("\x1f".join((r.get(c) or "") for c in columns) for r in parsed[name])
        add("input.digest", name, fmt_text(sha256(dump)))

    # Current demographics, so no decade-old racial composition is emitted alone.
    ccd_rows = []
    for lea in CCD_LEAS:
        path = os.path.join(DEMOGRAPHICS, "enrollment_%s.json" % lea)
        if not os.path.exists(path):
            raise SystemExit("missing %s — run ./fetch-demographics.sh" % path)
        with open(path, encoding="utf-8") as fh:
            body = fh.read()
        add("input.filedigest", "enrollment_%s" % lea, fmt_text(sha256(body)))
        ccd_rows.extend(json.loads(body).get("results", []))
    now_by_nces = current_enrollment(ccd_rows)
    add("input.rows", "ccd_enrollment", fmt_int(len(ccd_rows)))
    add("input.schools", "ccd_enrollment", fmt_int(len(now_by_nces)))

    achievement = parsed["achievement"]
    covariates = parsed["covariates"]
    dist_rows = parsed["districts"]

    levels = sorted(v for v in (num(a.get("cs_mn_avg_eb")) for a in achievement) if v is not None)
    growths = sorted(v for v in (num(a.get("cs_mn_grd_eb")) for a in achievement) if v is not None)

    labels = {nces: label for nces, label in K8}
    wanted = set(labels)
    nyc_labels = {nyc_seda_id(nces): label for nces, label in NYC}
    nyc_federal = {nyc_seda_id(nces): nces for nces, _ in NYC}
    nyc_wanted = set(nyc_labels)
    def is_brooklyn(c):
        return c.get("sedalea") == NYC_SEDA_LEA and (c.get("schcity") or "").upper() == "BROOKLYN"

    def now_of(sid):
        """
        The current federal counts for one SEDA school, or None.

        SEDA drops Alabama's leading zero and files New York under a synthetic
        citywide LEA; both have to be undone to reach the federal counts.
        """
        fed = nyc_federal.get(sid, ("0" + sid) if len(sid) == 11 else sid)
        return now_by_nces.get(fed)

    def school_dump(schools):
        """
        One line per school, carrying exactly the values the per-school facts do.

        Brooklyn is in the contract twice: the ten schools somebody named, and
        the whole borough they sit in. The borough is four hundred schools, and
        writing it out would add nearly five thousand lines and bury everything
        else, so it is checked by hashing this dump instead. One differing field
        in one school changes the hash.
        """
        lines = []
        for i, s in enumerate(schools):
            now = now_of(s["id"])
            lines.append("\x1f".join([
                s["id"], s["name"], s["district"], s["gradeLevel"],
                fmt_real(s["level"]), fmt_real(s["growth"]),
                NA if s["levelPct"] is None else fmt_int(s["levelPct"]),
                NA if s["growthPct"] is None else fmt_int(s["growthPct"]),
                fmt_real(s["pctBlack"]), fmt_real(s["enrollment"]),
                NA if now is None or now[0] is None else fmt_int(int(now[0])),
                NA if now is None or now[1] is None else fmt_int(int(now[1])),
                fmt_int(i),
            ]))
        return "\n".join(lines)

    groups = [
        ("phila", join_seda(achievement, covariates, lambda c: (c.get("sedasch") or "") in wanted, labels, levels, growths), True),
        ("tuscaloosa", join_seda(achievement, covariates, lambda c: c.get("sedalea") == TUSC_CITY_LEA, {}, levels, growths), True),
        ("brooklyn", join_seda(achievement, covariates, lambda c: (c.get("sedasch") or "") in nyc_wanted, nyc_labels, levels, growths), True),
        ("allBrooklyn", join_seda(achievement, covariates, is_brooklyn, {}, levels, growths), False),
    ]

    for group, schools, per_school in groups:
        add("seda.n", group, fmt_int(len(schools)))
        # The page draws a median rule on every row and plots growth for
        # whichever schools have it, so both are contract facts rather than
        # page arithmetic.
        add("seda.median", group, fmt_real(median([s["level"] for s in schools])))
        add("seda.withGrowth", group, fmt_int(sum(1 for s in schools if s["growth"] is not None)))
        add("seda.digest", group, fmt_text(sha256(school_dump(schools))))
        if not per_school:
            continue
        for i, s in enumerate(schools):
            key = "%s|%s" % (group, s["id"])
            add("seda.level", key, fmt_real(s["level"]))
            add("seda.growth", key, fmt_real(s["growth"]))
            add("seda.levelPct", key, NA if s["levelPct"] is None else fmt_int(s["levelPct"]))
            add("seda.growthPct", key, NA if s["growthPct"] is None else fmt_int(s["growthPct"]))
            add("seda.pctBlack", key, fmt_real(s["pctBlack"]))
            add("seda.enrollment", key, fmt_real(s["enrollment"]))
            add("seda.name", key, fmt_text(s["name"]))
            add("seda.district", key, fmt_text(s["district"]))
            add("seda.gradeLevel", key, fmt_text(s["gradeLevel"]))
            # The page lists schools by level, descending. Emitting the position
            # checks the ordering too, not just the values.
            add("seda.rank", key, fmt_int(i))
            now = now_of(s["id"])
            add("seda.nowEnrollment", key, NA if now is None or now[0] is None else fmt_int(int(now[0])))
            add("seda.nowBlack", key, NA if now is None or now[1] is None else fmt_int(int(now[1])))

    add("national.levels", "-", fmt_int(len(levels)))
    add("national.growths", "-", fmt_int(len(growths)))

    # How much of each district SEDA can say anything about. New York is the
    # reason this is in the contract: it has a growth estimate for about a third
    # of its schools, against most of Philadelphia's and all of Tuscaloosa's.
    for place, lea in (("Philadelphia", PHILA_LEA), ("Tuscaloosa_City", TUSC_CITY_LEA), ("New_York_City", NYC_SEDA_LEA)):
        ids = {c.get("sedasch") for c in covariates if c.get("sedalea") == lea}
        subset = [a for a in achievement if (a.get("sedasch") or "") in ids]
        add("coverage.schools", place, fmt_int(len(subset)))
        add("coverage.level", place, fmt_int(sum(1 for a in subset if num(a.get("cs_mn_avg_eb")) is not None)))
        add("coverage.growth", place, fmt_int(sum(1 for a in subset if num(a.get("cs_mn_grd_eb")) is not None)))

    # New York is absent from the 2022 district release entirely.
    add("district.newYorkRows", "-", fmt_int(sum(1 for r in dist_rows if r.get("stateabb") == "NY")))

    # ---------------------------------------------------------- the districts
    for r in dist_rows:
        rid = r.get("sedaadmin") or ""
        if r.get("subgroup") != "all" or rid not in DISTRICT_NAMES:
            continue
        subject = "math" if r.get("subject") == "mth" else "reading"
        add("district.level2019", "%s|%s" % (rid, subject), fmt_real(num(r.get("ys_mn19_eb"))))
        add("district.level2022", "%s|%s" % (rid, subject), fmt_real(num(r.get("ys_mn22_eb"))))
        add("district.change", "%s|%s" % (rid, subject), fmt_real(num(r.get("ys_chg_eb"))))
    for rid, name in DISTRICT_NAMES.items():
        add("district.name", rid, fmt_text(name))

    for subj in ("mth", "rla"):
        subject = "math" if subj == "mth" else "reading"
        for_subject = [r for r in dist_rows if r.get("subgroup") == "all" and r.get("subject") == subj]
        add("usMedian.level2019", subject,
            fmt_real(median([v for v in (num(r.get("ys_mn19_eb")) for r in for_subject) if v is not None])))
        add("usMedian.change", subject,
            fmt_real(median([v for v in (num(r.get("ys_chg_eb")) for r in for_subject) if v is not None])))
        # Every district row for the subject, including the ones with no
        # estimate: this is the denominator the page prints.
        add("usMedian.n", subject, fmt_int(len(for_subject)))

    # ------------------------------------------------- how much is just intake
    poverty = {}
    for c in covariates:
        cid = c.get("sedasch")
        p = num(c.get("perecd"))
        if cid is not None and p is not None and 0 <= p <= 1:
            poverty[cid] = p
    add("poverty.schools", "-", fmt_int(len(poverty)))

    # Where `perecd` stops measuring anything. It is built from free-and-reduced
    # -lunch eligibility, so a district on universal free meals books nearly
    # every school at the maximum whatever its families earn. The share of each
    # set at the ceiling is the only way to see which rows have run out of room,
    # so it is a contract fact rather than something the page works out.
    add("poverty.ceiling", "-", fmt_int(sum(1 for v in poverty.values() if v >= POVERTY_CEILING)))
    add("poverty.ceilingAt", "-", fmt_real(POVERTY_CEILING))

    for name, column in (("level", "cs_mn_avg_eb"), ("growth", "cs_mn_grd_eb")):
        xs, ys = [], []
        for a in achievement:
            sid = a.get("sedasch")
            v = num(a.get(column))
            pov = None if sid is None else poverty.get(sid)
            if v is None or pov is None:
                continue
            xs.append(pov)
            ys.append(v)
        add("poverty.%s.r" % name, "-", fmt_real(pearson(xs, ys)))
        add("poverty.%s.n" % name, "-", fmt_int(len(xs)))

    # ------------------------------------------- the sets the page compares
    #
    # The intake table reads five sets off one national line. Its numbers were
    # the last part of the page not covered here, which is exactly the part a
    # reader is most likely to want checked.
    level_of = {}
    for a in achievement:
        sid = a.get("sedasch")
        v = num(a.get("cs_mn_avg_eb"))
        if sid is not None and v is not None:
            level_of[sid] = v

    def set_stats(name, ids):
        levels = [level_of[i] for i in ids if i in level_of]
        povs = [poverty[i] for i in ids if i in poverty]
        add("set.n", name, fmt_int(len(levels)))
        add("set.level", name, fmt_real(median(levels)))
        add("set.poverty", name, fmt_real(median(povs)))
        # The denominator is the schools with a share at all — the same set the
        # median is taken over, not `set.n`, which counts level estimates.
        add("set.ceiling", name, fmt_int(sum(1 for v in povs if v >= POVERTY_CEILING)))
        add("set.ceilingOf", name, fmt_int(len(povs)))

    def cov_ids_where(keep):
        return [c["sedasch"] for c in covariates if c.get("sedasch") is not None and keep(c)]

    set_stats("philaEleven", [nces for nces, _ in K8])
    set_stats("philaAll", cov_ids_where(lambda c: c.get("sedalea") == PHILA_LEA))
    set_stats("brooklynTen", [nyc_seda_id(nces) for nces, _ in NYC])
    set_stats("brooklynAll", cov_ids_where(is_brooklyn))
    set_stats("tuscaloosaCity", cov_ids_where(lambda c: c.get("sedalea") == TUSC_CITY_LEA))

    # Least squares of level on the low-income share, all US schools — the line
    # the table's "predicted" column is read off.
    xs, ys = [], []
    for a in achievement:
        sid = a.get("sedasch")
        v = num(a.get("cs_mn_avg_eb"))
        pv = None if sid is None else poverty.get(sid)
        if v is None or pv is None:
            continue
        xs.append(pv)
        ys.append(v)
    n = len(xs)
    sx = sy = 0.0
    for i in range(n):
        sx += xs[i]
        sy += ys[i]
    mx, my = sx / n, sy / n
    sxy = sxx = 0.0
    for i in range(n):
        dx = xs[i] - mx
        sxy += dx * (ys[i] - my)
        sxx += dx * dx
    slope = sxy / sxx
    add("povertyFit.slope", "-", fmt_real(slope))
    add("povertyFit.intercept", "-", fmt_real(my - slope * mx))
    add("povertyFit.n", "-", fmt_int(n))

    return rows


def main():
    out_dir = os.path.join(ROOT, "calc", "out")
    os.makedirs(out_dir, exist_ok=True)
    tsv = serialize(build())
    with open(os.path.join(out_dir, "seda-python.tsv"), "w", encoding="utf-8", newline="\n") as fh:
        fh.write(tsv)
    print("wrote calc/out/seda-python.tsv (%d facts)" % (tsv.count("\n") - 1))


if __name__ == "__main__":
    main()

/**
 * The TypeScript implementation's side of the SEDA contract.
 *
 * As with the other two emitters, nothing is computed here that the site does
 * not already compute: `src/seda.ts` does the join and the percentiles, and this
 * writes the result out in the shared format.
 *
 * Run: npm run calc:ts   (writes calc/out/seda-typescript.tsv)
 */

import { createHash } from 'node:crypto'
import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'

import { int, NA, real, serialize, text, type Row } from '../src/contract.ts'
import { parseCsv } from '../src/csv.ts'
import { joinSeda, num, nycSedaId, NYC_SEDA_LEA, type SedaRow } from '../src/seda.ts'
import { K8_SCHOOLS, NYC_SCHOOLS } from '../src/schools.ts'
import { median, pearson } from '../src/stats.ts'
import { currentEnrollment, type CcdEnrollmentRow } from '../src/demographics.ts'

const ROOT = join(import.meta.dirname, '..')
const SEDA = join(ROOT, 'seda')
const DEMOGRAPHICS = join(ROOT, 'demographics')

/** LEAs the current federal enrolment counts are read for. */
const CCD_LEAS = ['4218990', '0103360', '3600091', '3600092']

const sha256 = (s: string): string => createHash('sha256').update(s, 'utf8').digest('hex')

/** Where the low-income measure runs out of room; see CONTRACT.md. */
const POVERTY_CEILING = 0.95

const PHILA_LEA = '4218990'
const TUSC_CITY_LEA = '103360'
const TUSC_COUNTY_LEA = '103390'

const DISTRICT_NAMES: Record<string, string> = {
  [PHILA_LEA]: 'Philadelphia City SD',
  [TUSC_CITY_LEA]: 'Tuscaloosa City',
  [TUSC_COUNTY_LEA]: 'Tuscaloosa County',
}

/** Each source, and the columns every implementation reads out of it. */
const SOURCES: { name: string; file: string; columns: string[] }[] = [
  {
    name: 'achievement',
    file: 'seda_school_pool_cs_4.1.csv',
    columns: ['sedasch', 'sedaschname', 'cs_mn_avg_eb', 'cs_mn_grd_eb'],
  },
  {
    name: 'covariates',
    file: 'seda_cov_school_pool_4.1.csv',
    columns: ['sedasch', 'sedalea', 'schcity', 'schnam', 'level', 'perblk', 'totenrl', 'perecd'],
  },
  {
    name: 'districts',
    file: 'seda2022_dist.csv',
    columns: ['sedaadmin', 'sedaadminname', 'subject', 'subgroup', 'ys_mn19_eb', 'ys_mn22_eb', 'ys_chg_eb'],
  },
]

const read = (file: string): SedaRow[] => {
  const path = join(SEDA, file)
  if (!existsSync(path)) throw new Error(`missing ${path} — run ./fetch-seda.sh`)
  return parseCsv(readFileSync(path, 'latin1'))
}

export function emitRows(): Row[] {
  const rows: Row[] = []
  const add = (metric: string, key: string, value: string): void => {
    rows.push({ metric, key, value })
  }

  const parsed = new Map<string, SedaRow[]>()
  for (const src of SOURCES) {
    const p = read(src.file)
    parsed.set(src.name, p)
    add('input.rows', src.name, int(p.length))
    add('input.digest', src.name, text(sha256(p.map((r) => src.columns.map((c) => r[c] ?? '').join('\x1f')).join('\n'))))
  }

  // Current demographics, so no decade-old racial composition is emitted alone.
  const ccdRows: CcdEnrollmentRow[] = []
  for (const lea of CCD_LEAS) {
    const path = join(DEMOGRAPHICS, `enrollment_${lea}.json`)
    if (!existsSync(path)) throw new Error(`missing ${path} — run ./fetch-demographics.sh`)
    const body = readFileSync(path, 'utf8')
    add('input.filedigest', `enrollment_${lea}`, text(sha256(body)))
    ccdRows.push(...((JSON.parse(body) as { results?: CcdEnrollmentRow[] }).results ?? []))
  }
  const nowByNces = currentEnrollment(ccdRows)
  add('input.rows', 'ccd_enrollment', int(ccdRows.length))
  add('input.schools', 'ccd_enrollment', int(nowByNces.size))

  const achievement = parsed.get('achievement') as SedaRow[]
  const covariates = parsed.get('covariates') as SedaRow[]
  const distRows = parsed.get('districts') as SedaRow[]

  // ------------------------------------------------------------ the schools
  const labels: Record<string, string> = {}
  for (const s of K8_SCHOOLS) if (s.nces) labels[s.nces] = s.label
  const wanted = new Set(K8_SCHOOLS.map((s) => s.nces).filter((v): v is string => v !== undefined))

  // SEDA files New York under one synthetic citywide LEA rather than the
  // geographic districts the federal collections use, so the ids crosswalk.
  const nycLabels: Record<string, string> = {}
  const nycWanted = new Set<string>()
  for (const s of NYC_SCHOOLS) {
    const id = nycSedaId(s.nces)
    nycLabels[id] = s.label
    nycWanted.add(id)
  }

  const nycFederalId = new Map(NYC_SCHOOLS.map((s) => [nycSedaId(s.nces), s.nces]))

  /**
   * The current federal counts for one SEDA school, or null.
   *
   * SEDA drops Alabama's leading zero and files New York under a synthetic
   * citywide LEA; both have to be undone to reach the federal counts.
   */
  const nowOf = (id: string): { enrollment: number | null; black: number | null } | null =>
    nowByNces.get(nycFederalId.get(id) ?? (id.length === 11 ? `0${id}` : id)) ?? null

  /**
   * One line per school, carrying exactly the values the per-school facts do.
   *
   * Brooklyn is in the contract twice: the ten schools somebody named, and the
   * whole borough they sit in. The borough is four hundred schools, and writing
   * it out would add nearly five thousand lines and bury everything else, so it
   * is checked by hashing this dump instead. One differing field in one school
   * changes the hash.
   */
  const schoolDump = (schools: ReturnType<typeof joinSeda>): string =>
    schools
      .map((s, i) => {
        const now = nowOf(s.id)
        return [
          s.id,
          s.name,
          s.district,
          s.gradeLevel,
          real(s.level),
          real(s.growth),
          s.levelPct === null ? NA : int(s.levelPct),
          s.growthPct === null ? NA : int(s.growthPct),
          real(s.pctBlack),
          real(s.enrollment),
          now === null || now.enrollment === null ? NA : int(now.enrollment),
          now === null || now.black === null ? NA : int(now.black),
          int(i),
        ].join('\x1f')
      })
      .join('\n')

  const isBrooklyn = (c: SedaRow): boolean =>
    c['sedalea'] === NYC_SEDA_LEA && (c['schcity'] ?? '').toUpperCase() === 'BROOKLYN'

  const groups: { group: string; schools: ReturnType<typeof joinSeda>; perSchool: boolean }[] = [
    { group: 'phila', schools: joinSeda(achievement, covariates, (c) => wanted.has(c['sedasch'] ?? ''), labels), perSchool: true },
    { group: 'tuscaloosa', schools: joinSeda(achievement, covariates, (c) => c['sedalea'] === TUSC_CITY_LEA), perSchool: true },
    { group: 'brooklyn', schools: joinSeda(achievement, covariates, (c) => nycWanted.has(c['sedasch'] ?? ''), nycLabels), perSchool: true },
    { group: 'allBrooklyn', schools: joinSeda(achievement, covariates, isBrooklyn), perSchool: false },
  ]

  for (const { group, schools, perSchool } of groups) {
    add('seda.n', group, int(schools.length))
    // The page draws a median rule on every row and plots growth for whichever
    // schools have it, so both are contract facts rather than page arithmetic.
    add('seda.median', group, real(median(schools.map((s) => s.level))))
    add('seda.withGrowth', group, int(schools.filter((s) => s.growth !== null).length))
    add('seda.digest', group, text(sha256(schoolDump(schools))))
    if (!perSchool) continue
    schools.forEach((s, i) => {
      const key = `${group}|${s.id}`
      add('seda.level', key, real(s.level))
      add('seda.growth', key, real(s.growth))
      add('seda.levelPct', key, s.levelPct === null ? 'NA' : int(s.levelPct))
      add('seda.growthPct', key, s.growthPct === null ? 'NA' : int(s.growthPct))
      add('seda.pctBlack', key, real(s.pctBlack))
      add('seda.enrollment', key, real(s.enrollment))
      add('seda.name', key, text(s.name))
      add('seda.district', key, text(s.district))
      add('seda.gradeLevel', key, text(s.gradeLevel))
      // The page lists schools by level, descending. Emitting the position
      // checks the ordering too, not just the values.
      add('seda.rank', key, int(i))
      const now = nowOf(s.id)
      add('seda.nowEnrollment', key, now === null || now.enrollment === null ? NA : int(now.enrollment))
      add('seda.nowBlack', key, now === null || now.black === null ? NA : int(now.black))
    })
  }

  // The two national distributions percentiles are taken against.
  const levels = achievement.map((a) => num(a['cs_mn_avg_eb'])).filter((v): v is number => v !== null)
  const growths = achievement.map((a) => num(a['cs_mn_grd_eb'])).filter((v): v is number => v !== null)
  add('national.levels', '-', int(levels.length))
  add('national.growths', '-', int(growths.length))

  // How much of each district SEDA can say anything about. New York is the
  // reason this is in the contract: it has a growth estimate for about a third
  // of its schools, against most of Philadelphia's and all of Tuscaloosa's.
  const covIds = (lea: string): Set<string> =>
    new Set(covariates.filter((c) => c['sedalea'] === lea).map((c) => c['sedasch'] as string))
  for (const [place, lea] of [
    ['Philadelphia', PHILA_LEA],
    ['Tuscaloosa_City', TUSC_CITY_LEA],
    ['New_York_City', NYC_SEDA_LEA],
  ] as [string, string][]) {
    const ids = covIds(lea)
    const rows2 = achievement.filter((a) => ids.has(a['sedasch'] ?? ''))
    add('coverage.schools', place, int(rows2.length))
    add('coverage.level', place, int(rows2.filter((a) => num(a['cs_mn_avg_eb']) !== null).length))
    add('coverage.growth', place, int(rows2.filter((a) => num(a['cs_mn_grd_eb']) !== null).length))
  }

  // New York is absent from the 2022 district release entirely.
  add('district.newYorkRows', '-', int(distRows.filter((r) => r['stateabb'] === 'NY').length))

  // ---------------------------------------------------------- the districts
  for (const r of distRows) {
    const id = r['sedaadmin'] ?? ''
    if (r['subgroup'] !== 'all' || !(id in DISTRICT_NAMES)) continue
    const subject = r['subject'] === 'mth' ? 'math' : 'reading'
    add('district.level2019', `${id}|${subject}`, real(num(r['ys_mn19_eb'])))
    add('district.level2022', `${id}|${subject}`, real(num(r['ys_mn22_eb'])))
    add('district.change', `${id}|${subject}`, real(num(r['ys_chg_eb'])))
  }
  for (const [id, name] of Object.entries(DISTRICT_NAMES)) add('district.name', id, text(name))

  for (const subj of ['mth', 'rla']) {
    const subject = subj === 'mth' ? 'math' : 'reading'
    const forSubject = distRows.filter((r) => r['subgroup'] === 'all' && r['subject'] === subj)
    add('usMedian.level2019', subject, real(median(forSubject.map((r) => num(r['ys_mn19_eb'])).filter((v): v is number => v !== null))))
    add('usMedian.change', subject, real(median(forSubject.map((r) => num(r['ys_chg_eb'])).filter((v): v is number => v !== null))))
    // Every district row for the subject, including the ones with no estimate:
    // this is the denominator the page prints.
    add('usMedian.n', subject, int(forSubject.length))
  }

  // ------------------------------------------------- how much is just intake
  const povertyOf = new Map<string, number>()
  for (const c of covariates) {
    const id = c['sedasch']
    const p = num(c['perecd'])
    if (id !== undefined && p !== null && p >= 0 && p <= 1) povertyOf.set(id, p)
  }
  add('poverty.schools', '-', int(povertyOf.size))

  // Where `perecd` stops measuring anything. It is built from free-and-reduced
  // -lunch eligibility, so a district on universal free meals books nearly
  // every school at the maximum whatever its families earn. The share of each
  // set at the ceiling is the only way to see which rows have run out of room,
  // so it is a contract fact rather than something the page works out.
  add('poverty.ceiling', '-', int([...povertyOf.values()].filter((p) => p >= POVERTY_CEILING).length))
  add('poverty.ceilingAt', '-', real(POVERTY_CEILING))

  for (const [name, column] of [
    ['level', 'cs_mn_avg_eb'],
    ['growth', 'cs_mn_grd_eb'],
  ] as [string, string][]) {
    const xs: number[] = []
    const ys: number[] = []
    for (const a of achievement) {
      const id = a['sedasch']
      const v = num(a[column])
      const pov = id === undefined ? undefined : povertyOf.get(id)
      if (v === null || pov === undefined) continue
      xs.push(pov)
      ys.push(v)
    }
    add(`poverty.${name}.r`, '-', real(pearson(xs, ys)))
    add(`poverty.${name}.n`, '-', int(xs.length))
  }

  // ------------------------------------------- the sets the page compares
  //
  // The intake table reads five sets off one national line. Its numbers were
  // the last part of the page not covered here, which is exactly the part a
  // reader is most likely to want checked.
  const levelOf = new Map<string, number>()
  for (const a of achievement) {
    const id = a['sedasch']
    const v = num(a['cs_mn_avg_eb'])
    if (id !== undefined && v !== null) levelOf.set(id, v)
  }

  const setStats = (set: string, ids: string[]): void => {
    const levels: number[] = []
    const povs: number[] = []
    for (const id of ids) {
      const l = levelOf.get(id)
      if (l !== undefined) levels.push(l)
      const p = povertyOf.get(id)
      if (p !== undefined) povs.push(p)
    }
    add('set.n', set, int(levels.length))
    add('set.level', set, real(median(levels)))
    add('set.poverty', set, real(median(povs)))
    // The denominator is the schools with a share at all — the same set the
    // median is taken over, not `set.n`, which counts level estimates.
    add('set.ceiling', set, int(povs.filter((p) => p >= POVERTY_CEILING).length))
    add('set.ceilingOf', set, int(povs.length))
  }

  const covIdsWhere = (keep: (c: SedaRow) => boolean): string[] =>
    covariates.filter(keep).map((c) => c['sedasch'] as string).filter((id) => id !== undefined)

  setStats('philaEleven', K8_SCHOOLS.map((s) => s.nces).filter((v): v is string => v !== undefined))
  setStats('philaAll', covIdsWhere((c) => c['sedalea'] === PHILA_LEA))
  setStats('brooklynTen', NYC_SCHOOLS.map((s) => nycSedaId(s.nces)))
  setStats('brooklynAll', covIdsWhere(isBrooklyn))
  setStats('tuscaloosaCity', covIdsWhere((c) => c['sedalea'] === TUSC_CITY_LEA))

  // Least squares of level on the low-income share, all US schools — the line
  // the table's "predicted" column is read off.
  {
    const xs: number[] = []
    const ys: number[] = []
    for (const a of achievement) {
      const id = a['sedasch']
      const v = num(a['cs_mn_avg_eb'])
      const p = id === undefined ? undefined : povertyOf.get(id)
      if (v === null || p === undefined) continue
      xs.push(p)
      ys.push(v)
    }
    const n = xs.length
    let sx = 0
    let sy = 0
    for (let i = 0; i < n; i++) {
      sx += xs[i] as number
      sy += ys[i] as number
    }
    const mx = sx / n
    const my = sy / n
    let sxy = 0
    let sxx = 0
    for (let i = 0; i < n; i++) {
      const dx = (xs[i] as number) - mx
      sxy += dx * ((ys[i] as number) - my)
      sxx += dx * dx
    }
    const slope = sxy / sxx
    add('povertyFit.slope', '-', real(slope))
    add('povertyFit.intercept', '-', real(my - slope * mx))
    add('povertyFit.n', '-', int(n))
  }

  return rows
}

if (import.meta.filename === process.argv[1]) {
  const outDir = join(ROOT, 'calc', 'out')
  mkdirSync(outDir, { recursive: true })
  const tsv = serialize(emitRows())
  writeFileSync(join(outDir, 'seda-typescript.tsv'), tsv)
  console.log(`wrote calc/out/seda-typescript.tsv (${tsv.split('\n').length - 2} facts)`)
}

metric	key	value
coverage.growth	New_York_City	432
coverage.growth	Philadelphia	241
coverage.growth	Tuscaloosa_City	18
coverage.level	New_York_City	1214
coverage.level	Philadelphia	279
coverage.level	Tuscaloosa_City	18
coverage.schools	New_York_City	1282
coverage.schools	Philadelphia	301
coverage.schools	Tuscaloosa_City	18
district.change	103360|math	-0.102000000000
district.change	103360|reading	-0.044300000000
district.change	103390|math	-0.033600000000
district.change	103390|reading	0.025600000000
district.change	4218990|math	-0.259000000000
district.change	4218990|reading	-0.084200000000
district.level2019	103360|math	-0.430000000000
district.level2019	103360|reading	-0.300000000000
district.level2019	103390|math	-0.382000000000
district.level2019	103390|reading	-0.274000000000
district.level2019	4218990|math	-0.598000000000
district.level2019	4218990|reading	-0.515000000000
district.level2022	103360|math	-0.532161990000
district.level2022	103360|reading	-0.343788910000
district.level2022	103390|math	-0.416051920000
district.level2022	103390|reading	-0.248064470000
district.level2022	4218990|math	-0.856155940000
district.level2022	4218990|reading	-0.598798480000
district.name	103360	Tuscaloosa City
district.name	103390	Tuscaloosa County
district.name	4218990	Philadelphia City SD
district.newYorkRows	-	0
input.digest	achievement	6cae2eb52490f94e943265c840d77b9b6ea757000d569a14e2a7c890d58d7dcc
input.digest	covariates	3fa93d47246b21c790620f1a7391568c53d57d8ac3cd5d9f257e2f598848f994
input.digest	districts	5d25751e4b32fd3e6609b5a165640447d8b4680e910feacc4af2f9d7361c2620
input.filedigest	enrollment_0103360	cf5174efcd9744bf9dd368f260b97f4536a19025c11037c449990158354ffe86
input.filedigest	enrollment_3600091	9a75833b6c9455e651f5fb23adc7126101f720f01cd37b6b4099fb51800fa863
input.filedigest	enrollment_3600092	62571c907088c5d4d2fa86856e2e97925ac88353c7e57ce8903a94e63938abe4
input.filedigest	enrollment_4218990	18908ff7b1ea04015a42891d47dc209c7b3e9a5793253fc6ba91c29fbb5c32a9
input.rows	achievement	82488
input.rows	ccd_enrollment	8910
input.rows	covariates	89350
input.rows	districts	32804
input.schools	ccd_enrollment	330
national.growths	-	51098
national.levels	-	72914
poverty.ceiling	-	7176
poverty.ceilingAt	-	0.950000000000
poverty.growth.n	-	51016
poverty.growth.r	-	-0.118066341620
poverty.level.n	-	72714
poverty.level.r	-	-0.814377936930
poverty.schools	-	86711
povertyFit.intercept	-	0.698923426303
povertyFit.n	-	72714
povertyFit.slope	-	-1.333664257200
seda.digest	allBrooklyn	ebe36ebc19ac95d8d9d244c4a3d414b3c78bffef1a54533568fd0cbe58f5a20b
seda.digest	brooklyn	bf2122c9e9009ec15b3fe233f499ae98bc106c54d074f96db8dba301df77dfb7
seda.digest	phila	5174c2fb102dfe660fcf587ed23f45597b3bffa7c58058abbc595c71db9b5d43
seda.digest	tuscaloosa	aad244b87219c06a202686c5c0d58be50b74644dfb6cced63ffbd9289b9e20cf
seda.district	brooklyn|362058002069	3620580
seda.district	brooklyn|362058002072	3620580
seda.district	brooklyn|362058002074	3620580
seda.district	brooklyn|362058002175	3620580
seda.district	brooklyn|362058002226	3620580
seda.district	brooklyn|362058002403	3620580
seda.district	brooklyn|362058002490	3620580
seda.district	brooklyn|362058002557	3620580
seda.district	brooklyn|362058002793	3620580
seda.district	brooklyn|362058002818	3620580
seda.district	phila|421899000619	4218990
seda.district	phila|421899003618	4218990
seda.district	phila|421899003649	4218990
seda.district	phila|421899003650	4218990
seda.district	phila|421899003782	4218990
seda.district	phila|421899003783	4218990
seda.district	phila|421899003805	4218990
seda.district	phila|421899004732	4218990
seda.district	phila|421899005186	4218990
seda.district	phila|421899006528	4218990
seda.district	phila|421899006530	4218990
seda.district	tuscaloosa|10336000584	103360
seda.district	tuscaloosa|10336001250	103360
seda.district	tuscaloosa|10336001251	103360
seda.district	tuscaloosa|10336001252	103360
seda.district	tuscaloosa|10336001255	103360
seda.district	tuscaloosa|10336001256	103360
seda.district	tuscaloosa|10336001257	103360
seda.district	tuscaloosa|10336001260	103360
seda.district	tuscaloosa|10336001266	103360
seda.district	tuscaloosa|10336001268	103360
seda.district	tuscaloosa|10336001269	103360
seda.district	tuscaloosa|10336001271	103360
seda.district	tuscaloosa|10336001632	103360
seda.district	tuscaloosa|10336001881	103360
seda.district	tuscaloosa|10336001889	103360
seda.district	tuscaloosa|10336002134	103360
seda.district	tuscaloosa|10336002143	103360
seda.district	tuscaloosa|10336002145	103360
seda.enrollment	brooklyn|362058002069	733.919490000000
seda.enrollment	brooklyn|362058002072	883.809690000000
seda.enrollment	brooklyn|362058002074	739.300230000000
seda.enrollment	brooklyn|362058002175	385.183350000000
seda.enrollment	brooklyn|362058002226	1094.393700000000
seda.enrollment	brooklyn|362058002403	551.194890000000
seda.enrollment	brooklyn|362058002490	551.038020000000
seda.enrollment	brooklyn|362058002557	497.922820000000
seda.enrollment	brooklyn|362058002793	929.940430000000
seda.enrollment	brooklyn|362058002818	1417.338100000000
seda.enrollment	phila|421899000619	574.012760000000
seda.enrollment	phila|421899003618	452.576080000000
seda.enrollment	phila|421899003649	535.566960000000
seda.enrollment	phila|421899003650	339.805660000000
seda.enrollment	phila|421899003782	530.450070000000
seda.enrollment	phila|421899003783	256.534940000000
seda.enrollment	phila|421899003805	236.835630000000
seda.enrollment	phila|421899004732	443.503170000000
seda.enrollment	phila|421899005186	558.125060000000
seda.enrollment	phila|421899006528	500.810180000000
seda.enrollment	phila|421899006530	518.397830000000
seda.enrollment	tuscaloosa|10336000584	544.940490000000
seda.enrollment	tuscaloosa|10336001250	425.035100000000
seda.enrollment	tuscaloosa|10336001251	337.246370000000
seda.enrollment	tuscaloosa|10336001252	332.295410000000
seda.enrollment	tuscaloosa|10336001255	670.716490000000
seda.enrollment	tuscaloosa|10336001256	417.140810000000
seda.enrollment	tuscaloosa|10336001257	288.318540000000
seda.enrollment	tuscaloosa|10336001260	350.812010000000
seda.enrollment	tuscaloosa|10336001266	399.849950000000
seda.enrollment	tuscaloosa|10336001268	443.080350000000
seda.enrollment	tuscaloosa|10336001269	448.333920000000
seda.enrollment	tuscaloosa|10336001271	413.397370000000
seda.enrollment	tuscaloosa|10336001632	568.598330000000
seda.enrollment	tuscaloosa|10336001881	145.422940000000
seda.enrollment	tuscaloosa|10336001889	495.487670000000
seda.enrollment	tuscaloosa|10336002134	510.468440000000
seda.enrollment	tuscaloosa|10336002143	160.412510000000
seda.enrollment	tuscaloosa|10336002145	294.626130000000
seda.gradeLevel	brooklyn|362058002069	Elementary
seda.gradeLevel	brooklyn|362058002072	Elementary
seda.gradeLevel	brooklyn|362058002074	Elementary
seda.gradeLevel	brooklyn|362058002175	Elementary
seda.gradeLevel	brooklyn|362058002226	Middle
seda.gradeLevel	brooklyn|362058002403	Elementary
seda.gradeLevel	brooklyn|362058002490	Elementary
seda.gradeLevel	brooklyn|362058002557	Elementary
seda.gradeLevel	brooklyn|362058002793	Elementary
seda.gradeLevel	brooklyn|362058002818	Elementary
seda.gradeLevel	phila|421899000619	Elementary
seda.gradeLevel	phila|421899003618	Elementary
seda.gradeLevel	phila|421899003649	Elementary
seda.gradeLevel	phila|421899003650	Elementary
seda.gradeLevel	phila|421899003782	Elementary
seda.gradeLevel	phila|421899003783	Elementary
seda.gradeLevel	phila|421899003805	Elementary
seda.gradeLevel	phila|421899004732	Elementary
seda.gradeLevel	phila|421899005186	Elementary
seda.gradeLevel	phila|421899006528	Elementary
seda.gradeLevel	phila|421899006530	Elementary
seda.gradeLevel	tuscaloosa|10336000584	Elementary
seda.gradeLevel	tuscaloosa|10336001250	Elementary
seda.gradeLevel	tuscaloosa|10336001251	Elementary
seda.gradeLevel	tuscaloosa|10336001252	Elementary
seda.gradeLevel	tuscaloosa|10336001255	Middle
seda.gradeLevel	tuscaloosa|10336001256	Elementary
seda.gradeLevel	tuscaloosa|10336001257	Elementary
seda.gradeLevel	tuscaloosa|10336001260	Elementary
seda.gradeLevel	tuscaloosa|10336001266	Elementary
seda.gradeLevel	tuscaloosa|10336001268	Middle
seda.gradeLevel	tuscaloosa|10336001269	Elementary
seda.gradeLevel	tuscaloosa|10336001271	Elementary
seda.gradeLevel	tuscaloosa|10336001632	Elementary
seda.gradeLevel	tuscaloosa|10336001881	Middle
seda.gradeLevel	tuscaloosa|10336001889	Middle
seda.gradeLevel	tuscaloosa|10336002134	Elementary
seda.gradeLevel	tuscaloosa|10336002143	Middle
seda.gradeLevel	tuscaloosa|10336002145	Elementary
seda.growth	brooklyn|362058002069	NA
seda.growth	brooklyn|362058002072	NA
seda.growth	brooklyn|362058002074	NA
seda.growth	brooklyn|362058002175	NA
seda.growth	brooklyn|362058002226	NA
seda.growth	brooklyn|362058002403	NA
seda.growth	brooklyn|362058002490	NA
seda.growth	brooklyn|362058002557	NA
seda.growth	brooklyn|362058002793	-0.013204446057
seda.growth	brooklyn|362058002818	NA
seda.growth	phila|421899000619	0.074640675684
seda.growth	phila|421899003618	-0.130599459072
seda.growth	phila|421899003649	-0.015879949546
seda.growth	phila|421899003650	-0.015943706604
seda.growth	phila|421899003782	-0.029730005782
seda.growth	phila|421899003783	NA
seda.growth	phila|421899003805	0.002622845809
seda.growth	phila|421899004732	-0.040009813872
seda.growth	phila|421899005186	-0.009919530861
seda.growth	phila|421899006528	-0.053472217529
seda.growth	phila|421899006530	0.037241028012
seda.growth	tuscaloosa|10336000584	0.053906243800
seda.growth	tuscaloosa|10336001250	0.099184361326
seda.growth	tuscaloosa|10336001251	0.144606237556
seda.growth	tuscaloosa|10336001252	-0.042549377918
seda.growth	tuscaloosa|10336001255	0.005899900850
seda.growth	tuscaloosa|10336001256	0.080510393625
seda.growth	tuscaloosa|10336001257	-0.083148383213
seda.growth	tuscaloosa|10336001260	0.024843163415
seda.growth	tuscaloosa|10336001266	0.026326298981
seda.growth	tuscaloosa|10336001268	-0.015381799326
seda.growth	tuscaloosa|10336001269	-0.021701793869
seda.growth	tuscaloosa|10336001271	-0.003411319961
seda.growth	tuscaloosa|10336001632	-0.023884187485
seda.growth	tuscaloosa|10336001881	-0.075701791797
seda.growth	tuscaloosa|10336001889	0.026863472238
seda.growth	tuscaloosa|10336002134	0.053953891176
seda.growth	tuscaloosa|10336002143	0.028615391663
seda.growth	tuscaloosa|10336002145	-0.018238163592
seda.growthPct	brooklyn|362058002069	NA
seda.growthPct	brooklyn|362058002072	NA
seda.growthPct	brooklyn|362058002074	NA
seda.growthPct	brooklyn|362058002175	NA
seda.growthPct	brooklyn|362058002226	NA
seda.growthPct	brooklyn|362058002403	NA
seda.growthPct	brooklyn|362058002490	NA
seda.growthPct	brooklyn|362058002557	NA
seda.growthPct	brooklyn|362058002793	31
seda.growthPct	brooklyn|362058002818	NA
seda.growthPct	phila|421899000619	88
seda.growthPct	phila|421899003618	1
seda.growthPct	phila|421899003649	29
seda.growthPct	phila|421899003650	29
seda.growthPct	phila|421899003782	21
seda.growthPct	phila|421899003783	NA
seda.growthPct	phila|421899003805	42
seda.growthPct	phila|421899004732	16
seda.growthPct	phila|421899005186	33
seda.growthPct	phila|421899006528	11
seda.growthPct	phila|421899006530	68
seda.growthPct	tuscaloosa|10336000584	78
seda.growthPct	tuscaloosa|10336001250	95
seda.growthPct	tuscaloosa|10336001251	99
seda.growthPct	tuscaloosa|10336001252	15
seda.growthPct	tuscaloosa|10336001255	45
seda.growthPct	tuscaloosa|10336001256	90
seda.growthPct	tuscaloosa|10336001257	4
seda.growthPct	tuscaloosa|10336001260	59
seda.growthPct	tuscaloosa|10336001266	61
seda.growthPct	tuscaloosa|10336001268	30
seda.growthPct	tuscaloosa|10336001269	25
seda.growthPct	tuscaloosa|10336001271	38
seda.growthPct	tuscaloosa|10336001632	24
seda.growthPct	tuscaloosa|10336001881	5
seda.growthPct	tuscaloosa|10336001889	61
seda.growthPct	tuscaloosa|10336002134	78
seda.growthPct	tuscaloosa|10336002143	62
seda.growthPct	tuscaloosa|10336002145	28
seda.level	brooklyn|362058002069	0.230938935180
seda.level	brooklyn|362058002072	0.602043384185
seda.level	brooklyn|362058002074	0.462316160595
seda.level	brooklyn|362058002175	0.779416239711
seda.level	brooklyn|362058002226	0.962595863718
seda.level	brooklyn|362058002403	0.926253547714
seda.level	brooklyn|362058002490	-0.206987233514
seda.level	brooklyn|362058002557	0.670484908402
seda.level	brooklyn|362058002793	0.099551128793
seda.level	brooklyn|362058002818	1.047354351739
seda.level	phila|421899000619	0.875694711629
seda.level	phila|421899003618	-0.527523861897
seda.level	phila|421899003649	0.586607305137
seda.level	phila|421899003650	-0.327989853407
seda.level	phila|421899003782	-0.958367186826
seda.level	phila|421899003783	-0.139980202330
seda.level	phila|421899003805	-0.549868067584
seda.level	phila|421899004732	-1.032088582117
seda.level	phila|421899005186	0.167242378667
seda.level	phila|421899006528	-0.648232398996
seda.level	phila|421899006530	-0.816776683460
seda.level	tuscaloosa|10336000584	0.505213428535
seda.level	tuscaloosa|10336001250	-0.686364399052
seda.level	tuscaloosa|10336001251	-0.361379269939
seda.level	tuscaloosa|10336001252	-0.685328533037
seda.level	tuscaloosa|10336001255	-0.673865822013
seda.level	tuscaloosa|10336001256	-0.695216256033
seda.level	tuscaloosa|10336001257	-0.593182832728
seda.level	tuscaloosa|10336001260	-0.726672147273
seda.level	tuscaloosa|10336001266	-0.502333827785
seda.level	tuscaloosa|10336001268	-0.848672265933
seda.level	tuscaloosa|10336001269	-0.386634943110
seda.level	tuscaloosa|10336001271	-0.726973945561
seda.level	tuscaloosa|10336001632	0.468757967816
seda.level	tuscaloosa|10336001881	-0.697400171310
seda.level	tuscaloosa|10336001889	0.334935795044
seda.level	tuscaloosa|10336002134	-0.621223121123
seda.level	tuscaloosa|10336002143	0.629598474341
seda.level	tuscaloosa|10336002145	0.575952615205
seda.levelPct	brooklyn|362058002069	74
seda.levelPct	brooklyn|362058002072	93
seda.levelPct	brooklyn|362058002074	88
seda.levelPct	brooklyn|362058002175	97
seda.levelPct	brooklyn|362058002226	99
seda.levelPct	brooklyn|362058002403	99
seda.levelPct	brooklyn|362058002490	34
seda.levelPct	brooklyn|362058002557	95
seda.levelPct	brooklyn|362058002793	63
seda.levelPct	brooklyn|362058002818	100
seda.levelPct	phila|421899000619	98
seda.levelPct	phila|421899003618	14
seda.levelPct	phila|421899003649	93
seda.levelPct	phila|421899003650	25
seda.levelPct	phila|421899003782	2
seda.levelPct	phila|421899003783	40
seda.levelPct	phila|421899003805	13
seda.levelPct	phila|421899004732	1
seda.levelPct	phila|421899005186	69
seda.levelPct	phila|421899006528	9
seda.levelPct	phila|421899006530	4
seda.levelPct	tuscaloosa|10336000584	90
seda.levelPct	tuscaloosa|10336001250	8
seda.levelPct	tuscaloosa|10336001251	23
seda.levelPct	tuscaloosa|10336001252	8
seda.levelPct	tuscaloosa|10336001255	8
seda.levelPct	tuscaloosa|10336001256	7
seda.levelPct	tuscaloosa|10336001257	11
seda.levelPct	tuscaloosa|10336001260	6
seda.levelPct	tuscaloosa|10336001266	15
seda.levelPct	tuscaloosa|10336001268	4
seda.levelPct	tuscaloosa|10336001269	21
seda.levelPct	tuscaloosa|10336001271	6
seda.levelPct	tuscaloosa|10336001632	89
seda.levelPct	tuscaloosa|10336001881	7
seda.levelPct	tuscaloosa|10336001889	82
seda.levelPct	tuscaloosa|10336002134	10
seda.levelPct	tuscaloosa|10336002143	94
seda.levelPct	tuscaloosa|10336002145	93
seda.median	allBrooklyn	-0.197355502802
seda.median	brooklyn	0.636264146293
seda.median	phila	-0.527523861897
seda.median	tuscaloosa	-0.607202976925
seda.n	allBrooklyn	400
seda.n	brooklyn	10
seda.n	phila	11
seda.n	tuscaloosa	18
seda.name	brooklyn|362058002069	PS 9 Teunis G. Bergen
seda.name	brooklyn|362058002072	PS 10 Magnet School
seda.name	brooklyn|362058002074	PS 11 Purvis J. Behan
seda.name	brooklyn|362058002175	PS 39 Henry Bristow
seda.name	brooklyn|362058002226	MS 51 William Alexander
seda.name	brooklyn|362058002403	PS 107 John W. Kimball
seda.name	brooklyn|362058002490	PS 133 William A. Butler
seda.name	brooklyn|362058002557	PS 154 Windsor Terrace
seda.name	brooklyn|362058002793	PS 282 Park Slope
seda.name	brooklyn|362058002818	PS 321 William Penn
seda.name	phila|421899000619	Penn Alexander
seda.name	phila|421899003618	Bache-Martin
seda.name	phila|421899003649	Meredith
seda.name	phila|421899003650	Nebinger
seda.name	phila|421899003782	Comegys
seda.name	phila|421899003783	Powel
seda.name	phila|421899003805	Marian Anderson
seda.name	phila|421899004732	Alain Locke
seda.name	phila|421899005186	Greenfield
seda.name	phila|421899006528	Harrington
seda.name	phila|421899006530	Lea
seda.name	tuscaloosa|10336000584	Rock Quarry Elementary School
seda.name	tuscaloosa|10336001250	The Alberta School of Performing Arts
seda.name	tuscaloosa|10336001251	Arcadia Elementary School
seda.name	tuscaloosa|10336001252	Central Elementary School
seda.name	tuscaloosa|10336001255	Eastwood Middle School
seda.name	tuscaloosa|10336001256	Northington Elementary School
seda.name	tuscaloosa|10336001257	Oakdale Elementary School
seda.name	tuscaloosa|10336001260	Skyland Elementary School
seda.name	tuscaloosa|10336001266	University Place Elementary School
seda.name	tuscaloosa|10336001268	Westlawn Middle School
seda.name	tuscaloosa|10336001269	Woodland Forrest Elementary School
seda.name	tuscaloosa|10336001271	Martin L King Jr Elementary School
seda.name	tuscaloosa|10336001632	Verner Elementary School
seda.name	tuscaloosa|10336001881	University Place Middle School
seda.name	tuscaloosa|10336001889	Northridge Middle School
seda.name	tuscaloosa|10336002134	Southview Elementary School
seda.name	tuscaloosa|10336002143	Tuscaloosa Magnet School  Middle
seda.name	tuscaloosa|10336002145	Tuscaloosa Magnet School  Elementary
seda.nowBlack	brooklyn|362058002069	244
seda.nowBlack	brooklyn|362058002072	54
seda.nowBlack	brooklyn|362058002074	439
seda.nowBlack	brooklyn|362058002175	19
seda.nowBlack	brooklyn|362058002226	79
seda.nowBlack	brooklyn|362058002403	12
seda.nowBlack	brooklyn|362058002490	166
seda.nowBlack	brooklyn|362058002557	18
seda.nowBlack	brooklyn|362058002793	165
seda.nowBlack	brooklyn|362058002818	32
seda.nowBlack	phila|421899000619	74
seda.nowBlack	phila|421899003618	243
seda.nowBlack	phila|421899003649	44
seda.nowBlack	phila|421899003650	145
seda.nowBlack	phila|421899003782	249
seda.nowBlack	phila|421899003783	184
seda.nowBlack	phila|421899003805	128
seda.nowBlack	phila|421899004732	202
seda.nowBlack	phila|421899005186	86
seda.nowBlack	phila|421899006528	263
seda.nowBlack	phila|421899006530	285
seda.nowBlack	tuscaloosa|10336000584	73
seda.nowBlack	tuscaloosa|10336001250	592
seda.nowBlack	tuscaloosa|10336001251	303
seda.nowBlack	tuscaloosa|10336001252	242
seda.nowBlack	tuscaloosa|10336001255	611
seda.nowBlack	tuscaloosa|10336001256	NA
seda.nowBlack	tuscaloosa|10336001257	267
seda.nowBlack	tuscaloosa|10336001260	376
seda.nowBlack	tuscaloosa|10336001266	482
seda.nowBlack	tuscaloosa|10336001268	477
seda.nowBlack	tuscaloosa|10336001269	414
seda.nowBlack	tuscaloosa|10336001271	426
seda.nowBlack	tuscaloosa|10336001632	132
seda.nowBlack	tuscaloosa|10336001881	NA
seda.nowBlack	tuscaloosa|10336001889	259
seda.nowBlack	tuscaloosa|10336002134	605
seda.nowBlack	tuscaloosa|10336002143	81
seda.nowBlack	tuscaloosa|10336002145	133
seda.nowEnrollment	brooklyn|362058002069	771
seda.nowEnrollment	brooklyn|362058002072	813
seda.nowEnrollment	brooklyn|362058002074	852
seda.nowEnrollment	brooklyn|362058002175	360
seda.nowEnrollment	brooklyn|362058002226	963
seda.nowEnrollment	brooklyn|362058002403	517
seda.nowEnrollment	brooklyn|362058002490	617
seda.nowEnrollment	brooklyn|362058002557	426
seda.nowEnrollment	brooklyn|362058002793	539
seda.nowEnrollment	brooklyn|362058002818	1234
seda.nowEnrollment	phila|421899000619	583
seda.nowEnrollment	phila|421899003618	567
seda.nowEnrollment	phila|421899003649	515
seda.nowEnrollment	phila|421899003650	404
seda.nowEnrollment	phila|421899003782	267
seda.nowEnrollment	phila|421899003783	306
seda.nowEnrollment	phila|421899003805	309
seda.nowEnrollment	phila|421899004732	261
seda.nowEnrollment	phila|421899005186	730
seda.nowEnrollment	phila|421899006528	289
seda.nowEnrollment	phila|421899006530	484
seda.nowEnrollment	tuscaloosa|10336000584	655
seda.nowEnrollment	tuscaloosa|10336001250	705
seda.nowEnrollment	tuscaloosa|10336001251	420
seda.nowEnrollment	tuscaloosa|10336001252	301
seda.nowEnrollment	tuscaloosa|10336001255	722
seda.nowEnrollment	tuscaloosa|10336001256	NA
seda.nowEnrollment	tuscaloosa|10336001257	289
seda.nowEnrollment	tuscaloosa|10336001260	456
seda.nowEnrollment	tuscaloosa|10336001266	612
seda.nowEnrollment	tuscaloosa|10336001268	539
seda.nowEnrollment	tuscaloosa|10336001269	506
seda.nowEnrollment	tuscaloosa|10336001271	475
seda.nowEnrollment	tuscaloosa|10336001632	621
seda.nowEnrollment	tuscaloosa|10336001881	NA
seda.nowEnrollment	tuscaloosa|10336001889	727
seda.nowEnrollment	tuscaloosa|10336002134	645
seda.nowEnrollment	tuscaloosa|10336002143	167
seda.nowEnrollment	tuscaloosa|10336002145	303
seda.pctBlack	brooklyn|362058002069	0.551583590000
seda.pctBlack	brooklyn|362058002072	0.109369240000
seda.pctBlack	brooklyn|362058002074	0.745225550000
seda.pctBlack	brooklyn|362058002175	0.100666830000
seda.pctBlack	brooklyn|362058002226	0.121868770000
seda.pctBlack	brooklyn|362058002403	0.061950397000
seda.pctBlack	brooklyn|362058002490	0.326685730000
seda.pctBlack	brooklyn|362058002557	0.051633999000
seda.pctBlack	brooklyn|362058002793	0.641921580000
seda.pctBlack	brooklyn|362058002818	0.076830767000
seda.pctBlack	phila|421899000619	0.332727490000
seda.pctBlack	phila|421899003618	0.766228020000
seda.pctBlack	phila|421899003649	0.191735090000
seda.pctBlack	phila|421899003650	0.538971540000
seda.pctBlack	phila|421899003782	0.972577750000
seda.pctBlack	phila|421899003783	0.795495810000
seda.pctBlack	phila|421899003805	0.864163220000
seda.pctBlack	phila|421899004732	0.923590480000
seda.pctBlack	phila|421899005186	0.411883680000
seda.pctBlack	phila|421899006528	0.965497970000
seda.pctBlack	phila|421899006530	0.871207360000
seda.pctBlack	tuscaloosa|10336000584	0.116713460000
seda.pctBlack	tuscaloosa|10336001250	0.901518230000
seda.pctBlack	tuscaloosa|10336001251	0.564161060000
seda.pctBlack	tuscaloosa|10336001252	0.980601190000
seda.pctBlack	tuscaloosa|10336001255	0.833068310000
seda.pctBlack	tuscaloosa|10336001256	0.805354180000
seda.pctBlack	tuscaloosa|10336001257	0.992155670000
seda.pctBlack	tuscaloosa|10336001260	0.933014810000
seda.pctBlack	tuscaloosa|10336001266	0.776885510000
seda.pctBlack	tuscaloosa|10336001268	0.990180910000
seda.pctBlack	tuscaloosa|10336001269	0.790377620000
seda.pctBlack	tuscaloosa|10336001271	0.985377250000
seda.pctBlack	tuscaloosa|10336001632	0.273181740000
seda.pctBlack	tuscaloosa|10336001881	0.846527930000
seda.pctBlack	tuscaloosa|10336001889	0.242520680000
seda.pctBlack	tuscaloosa|10336002134	0.935312090000
seda.pctBlack	tuscaloosa|10336002143	0.492854210000
seda.pctBlack	tuscaloosa|10336002145	0.456884710000
seda.rank	brooklyn|362058002069	7
seda.rank	brooklyn|362058002072	5
seda.rank	brooklyn|362058002074	6
seda.rank	brooklyn|362058002175	3
seda.rank	brooklyn|362058002226	1
seda.rank	brooklyn|362058002403	2
seda.rank	brooklyn|362058002490	9
seda.rank	brooklyn|362058002557	4
seda.rank	brooklyn|362058002793	8
seda.rank	brooklyn|362058002818	0
seda.rank	phila|421899000619	0
seda.rank	phila|421899003618	5
seda.rank	phila|421899003649	1
seda.rank	phila|421899003650	4
seda.rank	phila|421899003782	9
seda.rank	phila|421899003783	3
seda.rank	phila|421899003805	6
seda.rank	phila|421899004732	10
seda.rank	phila|421899005186	2
seda.rank	phila|421899006528	7
seda.rank	phila|421899006530	8
seda.rank	tuscaloosa|10336000584	2
seda.rank	tuscaloosa|10336001250	12
seda.rank	tuscaloosa|10336001251	5
seda.rank	tuscaloosa|10336001252	11
seda.rank	tuscaloosa|10336001255	10
seda.rank	tuscaloosa|10336001256	13
seda.rank	tuscaloosa|10336001257	8
seda.rank	tuscaloosa|10336001260	15
seda.rank	tuscaloosa|10336001266	7
seda.rank	tuscaloosa|10336001268	17
seda.rank	tuscaloosa|10336001269	6
seda.rank	tuscaloosa|10336001271	16
seda.rank	tuscaloosa|10336001632	3
seda.rank	tuscaloosa|10336001881	14
seda.rank	tuscaloosa|10336001889	4
seda.rank	tuscaloosa|10336002134	9
seda.rank	tuscaloosa|10336002143	0
seda.rank	tuscaloosa|10336002145	1
seda.withGrowth	allBrooklyn	132
seda.withGrowth	brooklyn	1
seda.withGrowth	phila	10
seda.withGrowth	tuscaloosa	18
set.ceiling	brooklynAll	71
set.ceiling	brooklynTen	0
set.ceiling	philaAll	171
set.ceiling	philaEleven	7
set.ceiling	tuscaloosaCity	1
set.ceilingOf	brooklynAll	471
set.ceilingOf	brooklynTen	11
set.ceilingOf	philaAll	310
set.ceilingOf	philaEleven	11
set.ceilingOf	tuscaloosaCity	19
set.level	brooklynAll	-0.197355502802
set.level	brooklynTen	0.636264146293
set.level	philaAll	-0.628922930895
set.level	philaEleven	-0.527523861897
set.level	tuscaloosaCity	-0.607202976925
set.n	brooklynAll	400
set.n	brooklynTen	10
set.n	philaAll	279
set.n	philaEleven	11
set.n	tuscaloosaCity	18
set.poverty	brooklynAll	0.876131470000
set.poverty	brooklynTen	0.304890600000
set.poverty	philaAll	0.972991260000
set.poverty	philaEleven	0.972183050000
set.poverty	tuscaloosaCity	0.796933350000
usMedian.change	math	-0.146000000000
usMedian.change	reading	-0.089800000000
usMedian.level2019	math	0.043600000000
usMedian.level2019	reading	0.037900000000
usMedian.n	math	5696
usMedian.n	reading	5601

"""
The contract format, for the Python implementations.

A port of `src/contract.ts`, kept in one place so the three Python
implementations cannot drift apart about how a number is written down. See
`calc/CONTRACT.md` for what the format is and why it is shaped this way.
"""

from __future__ import annotations

import hashlib
import math
import re

HEADER = "metric\tkey\tvalue"

#: How a value that does not exist is written — an empty cohort, an undefined rate.
NA = "NA"

#: Decimal places used for every real in the contract.
PRECISION = 12

#: Printable ASCII, excluding space so that a key never looks like two columns.
_SORTABLE = re.compile(r"^[\x21-\x7e]+$")


def fmt_int(n):
    if isinstance(n, bool) or not isinstance(n, int):
        raise ValueError("contract: %r is not an integer" % (n,))
    return str(n)


def fmt_real(x):
    if x is None:
        return NA
    if not math.isfinite(x):
        raise ValueError("contract: %r is not finite" % (x,))
    # -0.0 and 0.0 are the same number; only their printed forms differ.
    return "%.*f" % (PRECISION, 0.0 if x == 0 else x)


def fmt_text(s):
    if "\t" in s or "\n" in s or "\r" in s:
        raise ValueError("contract: value contains a tab or newline: %r" % (s,))
    return s


def serialize(rows):
    """Sort (metric, key) in byte order and write the file. Rows are 3-tuples."""
    seen = set()
    for metric, key, value in rows:
        if not _SORTABLE.match(metric):
            raise ValueError("contract: metric is not printable ASCII: %r" % (metric,))
        if not _SORTABLE.match(key):
            raise ValueError("contract: key is not printable ASCII: %r" % (key,))
        fmt_text(value)
        if (metric, key) in seen:
            raise ValueError("contract: duplicate row for %s/%s" % (metric, key))
        seen.add((metric, key))
    lines = ["%s\t%s\t%s" % row for row in sorted(rows, key=lambda row: row[0] + "\t" + row[1])]
    return HEADER + "\n" + "\n".join(lines) + "\n"


def sha256(s):
    return hashlib.sha256(s.encode("utf-8")).hexdigest()

# The calculation contract

This directory is a strangler fig. The original calculations are the TypeScript
in `../src/`; they still run, and they still produce the site. Around them, a
second implementation is growing as R Markdown documents, and a third as Python
scripts. This file is the seam they meet at.

Three calculations have been written three times each: **retention** (teacher
retention from the payroll extracts), **attendance** (chronic absence in
Philadelphia and Brooklyn, and the federal three-city comparison), and **seda**
(Philadelphia and Tuscaloosa on a common achievement scale).

The pattern's point is that nothing is replaced by faith. A new implementation
is only allowed to take over a part of the calculation once it demonstrably
produces the same answer for that part, and the demonstration is a test that
keeps running afterwards. The contract below is what "the same answer" means.

## The format

One file per calculation per implementation, in `out/`, named
`<contract>-<implementation>.tsv`:

| File | Written by |
| --- | --- |
| `out/<contract>-typescript.tsv` | `<contract>-emit.ts`, calling the existing `src/` modules |
| `out/<contract>-r.tsv` | `<contract>.Rmd`, knitted |
| `out/<contract>-python.tsv` | `<contract>.py` |

Each is a tab-separated file with the header `metric`, `key`, `value`, one fact
per line, sorted by metric then key, LF line endings, UTF-8, with a trailing
newline. If the calculation is the same, the three files are byte-identical.

Three rules make that achievable across three languages:

**Counts stay counts.** Every rate is published as its numerator and its
denominator, as integers, rather than as the quotient — `pooled.teacher.stayed`
and `pooled.teacher.start`, not `pooled.teacher.rate`. Integers cannot drift, so
most of the contract is exact by construction rather than by tolerance.

**Reals get exactly twelve decimal places.** What remains real is small in
magnitude — percentile ranks, correlations, medians of rates, attendance
percentages, achievement estimates in grade levels — so twelve decimals is far
finer than any disagreement worth having and coarse enough to absorb a last-bit
difference in how three languages accumulate a sum. A value that does not exist
is `NA`, never an empty cell and never a zero. Negative zero is written as zero.

**Sort keys are printable ASCII with no spaces.** Byte order, code-point order
and UTF-16 code-unit order are the same order for ASCII, so all three
implementations sort identically without anyone reasoning about locale
collation. Values may hold anything but a tab or a newline, because values are
never sorted on: `metric` and `key` together are unique.

## What is in it

### retention

| Metric family | What it covers |
| --- | --- |
| `input.*` | Rows, distinct identity keys, teacher and principal counts, and two SHA-256 digests per extract |
| `peer.*` | Peer-set size, median, and quartiles, per transition, level and staff definition |
| `series.*` | Stayers, starters and peer percentile for each school and transition, under each of the three staff definitions |
| `pooled.*` | The same, pooled over the window and over 2020 onwards |
| `alt.*` | PFT and student/contracted headcount per school-year |
| `principal.*` | Tenure runs and the number of changes |
| `era.*` | Retention pooled within each principal's tenure |
| `cohort2020.*` | The 2020 teaching cohort, and how much of it is left each year |
| `destination.*` | Where leavers went: another school, another role, or out of the district |
| `correlation.*` | Principal churn and longest tenure against pooled retention, plus both input series |

The two digests are the part that does the most work. `input.digest` hashes the
six fields each implementation reads out of an extract, in file order, so that
"three CSV readers appear to agree" becomes something a test asserts rather than
something anyone assumes. `input.keydigest` hashes the sorted identity keys, so
that the name-normalization rule is checked on its own rather than only through
its effect on the counts.

### attendance

| Metric family | What it covers |
| --- | --- |
| `input.*` | Rows and a digest per Philadelphia CSV; a raw-file digest per NYC and CRDC JSON |
| `phl.*` | Average daily attendance, chronic absence, the 95% series and teacher attendance, per school-year |
| `phlMedian.*`, `phlCount.*` | The districtwide median for each measure and year, and how many schools it is over |
| `nyc.*`, `nycMedian.*` | The same two measures per Brooklyn school-year, and the citywide medians |
| `federal.*` | The CRDC rate for every school in all three cities, for 2017-18 and 2022-23, with its city and name |
| `federalMedian.*` | Median, range and count per city per collection, on the one common definition |
| `correlation.*` | Chronic absence against teacher attendance, plus both input series |

The JSON sources are hashed as raw bytes rather than as parsed columns: there is
no cross-language question about how to read a JSON string, and what each
implementation makes of the numbers is checked by the facts themselves. The CRDC
files are where that matters most — they return two rows per school even with
every subgroup pinned to "all", and keeping the homeless-subgroup row instead of
the all-students row produces single-digit rates that look entirely plausible.

### seda

| Metric family | What it covers |
| --- | --- |
| `input.*` | Rows and a digest per SEDA file |
| `seda.*` | Level, growth, both national percentiles, demographics, name and rank for each of the thirty-nine schools in the three named sets |
| `seda.n`, `seda.median`, `seda.withGrowth`, `seda.digest` | Per group: how many schools, the median level the page draws as a rule, how many have a growth estimate, and a digest of every school in it |
| `coverage.*` | How many schools in each district SEDA has a level and a growth estimate for |
| `seda.now*` | Current federal enrolment and Black enrolment per school, so no decade-old composition is published alone |
| `national.*` | The size of the two national distributions percentiles are taken against |
| `district.*`, `usMedian.*` | The 2019 and 2022 district estimates, the change, and the national median district |
| `poverty.*` | How far level and growth track the share of students economically disadvantaged, across every school in the release, and how many schools sit at that measure's ceiling |
| `set.*`, `povertyFit.*` | The five sets the page's intake table compares — count, median level, median low-income share, and share at the ceiling — and the national line their predictions are read off |

### A note on `perecd`

`perecd` is the share of a school's students who are economically disadvantaged,
and it is the input to every `poverty.*`, `set.poverty` and `povertyFit.*` fact.
It is derived from free-and-reduced-lunch eligibility, which means a district
serving universal free meals books nearly every school at the maximum whatever
its families earn. Philadelphia has done that district-wide since 2014;
Tuscaloosa City has not. The measure therefore has a ceiling, and the ceiling is
not the same distance away for every district.

Nothing in the contract can fix that — there is no income measure in the
school-level release that is not built from meal eligibility. What the contract
can do is make it visible, so `set.ceiling` and `set.ceilingOf` are emitted
beside every median: how many of a set are recorded at or above
`poverty.ceilingAt` (0.95), out of how many have a share at all. `poverty.ceiling`
is the same count across the whole release, so any set can be read against the
national rate. A set that is mostly at the ceiling cannot be placed on
`povertyFit` at all, and the page says so where it draws the table.

## Definitions the three had to be told to share

These are places where the languages' defaults disagree, and where all three
implementations therefore say something more explicit than they otherwise would.

- **`\b` is not the same everywhere.** JavaScript's word boundary is ASCII-only;
  Python's is Unicode-aware by default. The teacher-title test is written as an
  explicit character test, and Python passes `re.ASCII`.
- **Uppercasing is not the same everywhere.** Only `A`–`Z` survive the identity
  key, so the three agree on today's extracts, where the only non-ASCII letter in
  any name is a single `É`. A future extract containing a character whose
  uppercase form *is* ASCII — `ı`, or the `fi` ligature — would make them differ.
  `input.keydigest` is what would catch it.
- **Sorting is not the same everywhere.** R sorts strings by the session's
  locale unless told otherwise, so every sort in the R implementation passes
  `method = "radix"`, which is C-locale byte order.
- **Summation order is not the same everywhere.** Pearson's r is written as an
  explicit loop in all three, rather than as `cor()` or a vectorized sum, so the
  terms are added in the same order.
- **A duplicated name resolves to the last row.** Building a person-to-school
  index by assignment in file order means the last row wins. R's named-vector
  lookup would take the first, so the R implementation deduplicates with
  `fromLast = TRUE` to match. The same applies to every id-keyed index in the
  attendance and SEDA calculations.
- **Rounding halves is not the same everywhere.** JavaScript's `Math.round`
  sends halves up; R's `round` and Python's `round` both send halves to the
  nearest even number. A SEDA percentile landing exactly on 46.5 would come out
  47 in the original and 46 in a naive port, so all three compute
  `floor(x + 0.5)` explicitly. This is the likeliest place for three
  implementations to disagree by one.
- **What counts as a number is not the same everywhere.** Python's `float` and
  R's `as.numeric` accept strings JavaScript's `Number` rejects, and vice versa.
  The SEDA reader checks the shape of the string against an explicit pattern
  before converting, so the three agree about what is *not* a number too.
- **A byte-order mark is not stripped everywhere.** One Philadelphia attendance
  file carries one. Left in place it becomes part of the first column's name, so
  every lookup on that column silently returns nothing — which produces an empty
  chart rather than an error. All three strip it from the raw bytes, before
  decoding.
- **New York's school ids are not the federal ones.** SEDA rolls the city's 32
  geographic districts into one synthetic citywide LEA (`3620580`), where CCD and
  CRDC keep them apart (`3600091` is District 13). The five-digit school number
  survives, so ids crosswalk by swapping the prefix; the contract emits
  `seda.district` for every school, so all three implementations have to agree
  that the crosswalk landed in that LEA.
- **A group too large to write out is hashed instead.** Three of the four SEDA
  groups are small enough to emit school by school. The fourth is every school in
  Brooklyn — four hundred of them, which at twelve facts each would be nine
  tenths of the contract and would bury the rest. All four groups therefore also
  carry `seda.digest`: the SHA-256 of a dump with one line per school, fields
  separated by `U+001F` and lines by `\n`, no trailing newline, in the group's own
  order. The fields are exactly the values the per-school facts carry — id, name,
  district, grade level, level, growth, both percentiles, percent Black,
  enrolment, current enrolment, current Black enrolment, rank — each written by
  the same formatter as the corresponding fact, so `NA` appears wherever the fact
  would say `NA`. One differing field in one school changes the hash. The cost is
  that a mismatch names the group rather than the school; `seda.n`,
  `seda.median` and `seda.withGrowth` are emitted beside it so the commonest
  causes are still named directly.
- **Coalescing on absence is not coalescing on emptiness.** The original uses
  `??`, which falls through only for a missing value, so a school with a blank
  name keeps the blank rather than falling back to the next candidate. Python's
  `or` and R's `if (!nzchar(x))` would both fall through; the ports test for
  presence instead.

## Migration ledger

Where the fig has actually reached. "Migrated" means the R Markdown and Python
implementations compute it and are checked against the original on every test
run; it does not yet mean the original has been removed.

| Part of the calculation | Status |
| --- | --- |
| Reading and parsing the October extracts | migrated — all three, verified by digest |
| Identity keys, teacher/alt/principal definitions | migrated — verified by digest |
| Year-over-year retention, all three staff definitions | migrated |
| Peer sets, medians, quartiles, percentile ranks | migrated |
| Pooled retention, including the 2020-onwards window | migrated |
| Principal tenures, changes, per-era retention | migrated |
| 2020 cohort survival | migrated |
| Leaver destinations | migrated |
| Churn/tenure correlations | migrated |
| Philadelphia attendance, all four series | migrated — all three |
| New York chronic absence and citywide medians | migrated |
| Federal CRDC three-city comparison | migrated |
| Chronic absence vs teacher attendance | migrated |
| SEDA level, growth and national percentiles | migrated — all three |
| Every school in Brooklyn, beside the ten that were named | migrated — verified by digest |
| The New York id crosswalk into SEDA's citywide LEA | migrated |
| Current demographics from CCD, joined to the SEDA schools | migrated |
| SEDA district estimates and the national median district | migrated |
| The poverty correlations across every US school | migrated |
| The intake table's five sets, and the line they are read off | migrated — all three |
| How much of each set sits at the low-income measure's ceiling | migrated |
| School labels, notes, display ordering | **not migrated** — presentation, stays in TypeScript |
| Catchment and zone lookups (`src/places.ts`) | **not migrated** — geography, stays in TypeScript |
| Rendering the site's `*-data.ts` modules | **not migrated** — the original still writes the site's data |

The last row is the one that matters for what "strangled" would eventually mean.
The site is still built by `npm run analyze`, which runs the TypeScript. What
these contracts establish is that it *could* be built by either of the other
two, and that the choice is now a routing decision rather than a rewrite.

## Running it

```sh
./calc/run.sh          # nine runs, then the comparison
npm run calc:ts        # just the TypeScript emitters
npm run calc:py        # just Python
npm run calc:r         # just the R Markdown (needs R + knitr, digest, jsonlite)
npm test               # asserts the committed outputs are identical
```

`npm test` compares the committed `out/*.tsv` and needs neither R nor Python
installed. `run.sh` regenerates them and needs both.

Cross-implementation equivalence check
========================================================================

Three implementations of each calculation — the original TypeScript, an R
Markdown document and a Python script — each writing the contract described
in calc/CONTRACT.md, then compared byte for byte.

RESULT: IDENTICAL — every implementation produced the same bytes.

  contracts:       3
  facts compared:  4135
  differing facts: 0
  max numeric difference:  0
  non-numeric differences: 0
  real precision:  12 decimal places

------------------------------------------------------------------------
RETENTION    identical   2697 facts   0 differing   max numeric difference 0

  typescript    2697 facts    109059 bytes
              sha256 5482b6dab98213dd934861682542b7823069ae4c8d3be32be97140ee5e0f5ddd
  r             2697 facts    109059 bytes
              sha256 5482b6dab98213dd934861682542b7823069ae4c8d3be32be97140ee5e0f5ddd
  python        2697 facts    109059 bytes
              sha256 5482b6dab98213dd934861682542b7823069ae4c8d3be32be97140ee5e0f5ddd

------------------------------------------------------------------------
ATTENDANCE   identical   868 facts   0 differing   max numeric difference 0

  typescript     868 facts     37778 bytes
              sha256 01130b46ec66dce7bc27ed085a5f3e6b94afe4ee5ca1a854b841d0883e36a174
  r              868 facts     37778 bytes
              sha256 01130b46ec66dce7bc27ed085a5f3e6b94afe4ee5ca1a854b841d0883e36a174
  python         868 facts     37778 bytes
              sha256 01130b46ec66dce7bc27ed085a5f3e6b94afe4ee5ca1a854b841d0883e36a174

------------------------------------------------------------------------
SEDA         identical   570 facts   0 differing   max numeric difference 0

  typescript     570 facts     25209 bytes
              sha256 3c66ba511b07dc26a5871f2195b9bdb835f7f47d9fc3415b3185ccdc7935e669
  r              570 facts     25209 bytes
              sha256 3c66ba511b07dc26a5871f2195b9bdb835f7f47d9fc3415b3185ccdc7935e669
  python         570 facts     25209 bytes
              sha256 3c66ba511b07dc26a5871f2195b9bdb835f7f47d9fc3415b3185ccdc7935e669

TAP version 13
# Subtest: normalizeUlcs strips padding so 0-prefixed codes join to plain ones
ok 1 - normalizeUlcs strips padding so 0-prefixed codes join to plain ones
  ---
  type: 'test'
  ...
# Subtest: normalizeUlcs keeps a genuine zero rather than emptying it
ok 2 - normalizeUlcs keeps a genuine zero rather than emptying it
  ---
  type: 'test'
  ...
# Subtest: seriesFromRows keys by school and year
ok 3 - seriesFromRows keys by school and year
  ---
  type: 'test'
  ...
# Subtest: seriesFromRows skips rows whose value will not parse
ok 4 - seriesFromRows skips rows whose value will not parse
  ---
  type: 'test'
  ...
# Subtest: seriesFromRows honours a row filter
ok 5 - seriesFromRows honours a row filter
  ---
  type: 'test'
  ...
# Subtest: chronicFromNyc converts the attending share into a missing share
ok 6 - chronicFromNyc converts the attending share into a missing share
  ---
  type: 'test'
  ...
# Subtest: chronicFromNyc drops rows with unusable numbers
ok 7 - chronicFromNyc drops rows with unusable numbers
  ---
  type: 'test'
  ...
# Subtest: a YearValue map reads back the years it was given
ok 8 - a YearValue map reads back the years it was given
  ---
  type: 'test'
  ...
# Subtest: federalRate converts CRDC counts to a percentage
ok 9 - federalRate converts CRDC counts to a percentage
  ---
  type: 'test'
  ...
# Subtest: federalRate rejects CRDC suppression codes rather than treating them as counts
ok 10 - federalRate rejects CRDC suppression codes rather than treating them as counts
  ---
  type: 'test'
  ...
# Subtest: federalRate rejects a missing or zero denominator
ok 11 - federalRate rejects a missing or zero denominator
  ---
  type: 'test'
  ...
# Subtest: federalRate does not return a rate above 100
ok 12 - federalRate does not return a rate above 100
  ---
  type: 'test'
  ...
# Subtest: integers are written without a decimal point
ok 13 - integers are written without a decimal point
  ---
  type: 'test'
  ...
# Subtest: int refuses anything that is not a whole number
ok 14 - int refuses anything that is not a whole number
  ---
  type: 'test'
  ...
# Subtest: reals get a fixed twelve decimal places
ok 15 - reals get a fixed twelve decimal places
  ---
  type: 'test'
  ...
# Subtest: negative zero is written as zero
ok 16 - negative zero is written as zero
  ---
  type: 'test'
  ...
# Subtest: a missing value is NA, not an empty cell or a zero
ok 17 - a missing value is NA, not an empty cell or a zero
  ---
  type: 'test'
  ...
# Subtest: real rejects a non-finite number
ok 18 - real rejects a non-finite number
  ---
  type: 'test'
  ...
# Subtest: text values may not contain a tab or a newline
ok 19 - text values may not contain a tab or a newline
  ---
  type: 'test'
  ...
# Subtest: serialize writes a header and a trailing newline
ok 20 - serialize writes a header and a trailing newline
  ---
  type: 'test'
  ...
# Subtest: serialize sorts by metric then key, in byte order
ok 21 - serialize sorts by metric then key, in byte order
  ---
  type: 'test'
  ...
# Subtest: serialize refuses two rows with the same metric and key
ok 22 - serialize refuses two rows with the same metric and key
  ---
  type: 'test'
  ...
# Subtest: metric and key must be plain printable ASCII
ok 23 - metric and key must be plain printable ASCII
  ---
  type: 'test'
  ...
# Subtest: a value may hold non-ASCII, because values are not sorted on
ok 24 - a value may hold non-ASCII, because values are not sorted on
  ---
  type: 'test'
  ...
# Subtest: parseContract round-trips what serialize wrote
ok 25 - parseContract round-trips what serialize wrote
  ---
  type: 'test'
  ...
# Subtest: parseContract rejects a file whose header is not the contract header
ok 26 - parseContract rejects a file whose header is not the contract header
  ---
  type: 'test'
  ...
# Subtest: parses a plain header + row
ok 27 - parses a plain header + row
  ---
  type: 'test'
  ...
# Subtest: strips surrounding double quotes
ok 28 - strips surrounding double quotes
  ---
  type: 'test'
  ...
# Subtest: keeps commas that live inside quoted fields
ok 29 - keeps commas that live inside quoted fields
  ---
  type: 'test'
  ...
# Subtest: unescapes doubled quotes inside a quoted field
ok 30 - unescapes doubled quotes inside a quoted field
  ---
  type: 'test'
  ...
# Subtest: handles CRLF line endings
ok 31 - handles CRLF line endings
  ---
  type: 'test'
  ...
# Subtest: ignores a trailing blank line
ok 32 - ignores a trailing blank line
  ---
  type: 'test'
  ...
# Subtest: preserves embedded newlines inside quoted fields
ok 33 - preserves embedded newlines inside quoted fields
  ---
  type: 'test'
  ...
# Subtest: pads short rows with empty strings
ok 34 - pads short rows with empty strings
  ---
  type: 'test'
  ...
# Subtest: strips a UTF-8 BOM so the first column key is usable
ok 35 - strips a UTF-8 BOM so the first column key is usable
  ---
  type: 'test'
  ...
# Subtest: strips a BOM that arrived through a latin-1 read
ok 36 - strips a BOM that arrived through a latin-1 read
  ---
  type: 'test'
  ...
# Subtest: a school is the total row and the Black row, both at sex 99
ok 37 - a school is the total row and the Black row, both at sex 99
  ---
  type: 'test'
  ...
# Subtest: other races are ignored rather than summed
ok 38 - other races are ignored rather than summed
  ---
  type: 'test'
  ...
# Subtest: a suppressed count is null, never a negative share
ok 39 - a suppressed count is null, never a negative share
  ---
  type: 'test'
  ...
# Subtest: a school with no total gets no share
ok 40 - a school with no total gets no share
  ---
  type: 'test'
  ...
# Subtest: a zero enrolment does not divide
ok 41 - a zero enrolment does not divide
  ---
  type: 'test'
  ...
# Subtest: rows with no school id are skipped
ok 42 - rows with no school id are skipped
  ---
  type: 'test'
  ...
# Subtest: every NCES id used for a join exists in the committed CCD directory
ok 43 - every NCES id used for a join exists in the committed CCD directory
  ---
  type: 'test'
  ...
# Subtest: the ids point at the schools they are labelled with
ok 44 - the ids point at the schools they are labelled with
  ---
  type: 'test'
  ...
# Subtest: Marian Anderson is filed federally under its former name
ok 45 - Marian Anderson is filed federally under its former name
  ---
  type: 'test'
  ...
# Subtest: every contract has an output from every implementation
ok 46 - every contract has an output from every implementation
  ---
  type: 'test'
  ...
# Subtest: every contract came out byte-identical across the three implementations
ok 47 - every contract came out byte-identical across the three implementations
  ---
  type: 'test'
  ...
# Subtest: the three contracts are three different calculations
ok 48 - the three contracts are three different calculations
  ---
  type: 'test'
  ...
# Subtest: each comparison covers its whole contract, not a handful of rows
ok 49 - each comparison covers its whole contract, not a handful of rows
  ---
  type: 'test'
  ...
# Subtest: every implementation read the same bytes off disk
ok 50 - every implementation read the same bytes off disk
  ---
  type: 'test'
  ...
# Subtest: the published equivalence report matches the committed outputs
ok 51 - the published equivalence report matches the committed outputs
  ---
  type: 'test'
  ...
# Subtest: each knitted R document reports the hash of the file it wrote
ok 52 - each knitted R document reports the hash of the file it wrote
  ---
  type: 'test'
  ...
# Subtest: re-running the TypeScript emitters reproduces their committed output
ok 53 - re-running the TypeScript emitters reproduces their committed output
  ---
  type: 'test'
  ...
# Subtest: isAltInstructional matches student and contracted teachers
ok 54 - isAltInstructional matches student and contracted teachers
  ---
  type: 'test'
  ...
# Subtest: isAltInstructional does not match regular PFT teachers or other staff
ok 55 - isAltInstructional does not match regular PFT teachers or other staff
  ---
  type: 'test'
  ...
# Subtest: destinations separates transfers, role changes and district exits
ok 56 - destinations separates transfers, role changes and district exits
  ---
  type: 'test'
  ...
# Subtest: destinations is all zeroes for a school with no teachers
ok 57 - destinations is all zeroes for a school with no teachers
  ---
  type: 'test'
  ...
# Subtest: eraRetention attributes each transition to whoever held the starting year
ok 58 - eraRetention attributes each transition to whoever held the starting year
  ---
  type: 'test'
  ...
# Subtest: eraRetention drops a run with no transition starting inside it
ok 59 - eraRetention drops a run with no transition starting inside it
  ---
  type: 'test'
  ...
# Subtest: a corpus reports where a teacher went and whether they stayed on the payroll
ok 60 - a corpus reports where a teacher went and whether they stayed on the payroll
  ---
  type: 'test'
  ...
# Subtest: pointInRing separates inside from outside
ok 61 - pointInRing separates inside from outside
  ---
  type: 'test'
  ...
# Subtest: pointInRing does not double-count a vertex at the test latitude
ok 62 - pointInRing does not double-count a vertex at the test latitude
  ---
  type: 'test'
  ...
# Subtest: pointInPolygon treats rings after the first as holes
ok 63 - pointInPolygon treats rings after the first as holes
  ---
  type: 'test'
  ...
# Subtest: featureAt finds the containing feature, Polygon or MultiPolygon
ok 64 - featureAt finds the containing feature, Polygon or MultiPolygon
  ---
  type: 'test'
  ...
# Subtest: featureAt returns null where no feature covers the point
ok 65 - featureAt returns null where no feature covers the point
  ---
  type: 'test'
  ...
# Subtest: featuresAt returns every match, so overlapping layers are visible
ok 66 - featuresAt returns every match, so overlapping layers are visible
  ---
  type: 'test'
  ...
# Subtest: geocoded reads a Census geocoder snapshot
ok 67 - geocoded reads a Census geocoder snapshot
  ---
  type: 'test'
  ...
# Subtest: geocoded rejects a snapshot with no match rather than guessing
ok 68 - geocoded rejects a snapshot with no match rather than guessing
  ---
  type: 'test'
  ...
# Subtest: 4914 Pentridge St resolves to the Harrington catchment
ok 69 - 4914 Pentridge St resolves to the Harrington catchment
  ---
  type: 'test'
  ...
# Subtest: the catchment is read from the committed layer, not asserted
ok 70 - the catchment is read from the committed layer, not asserted
  ---
  type: 'test'
  ...
# Subtest: the two sides of Grand Army Plaza are zoned to different elementary schools
ok 71 - the two sides of Grand Army Plaza are zoned to different elementary schools
  ---
  type: 'test'
  ...
# Subtest: PS 11's own building falls inside PS 11's zone
ok 72 - PS 11's own building falls inside PS 11's zone
  ---
  type: 'test'
  ...
# Subtest: District 13 has no zoned middle school, per the middle-school layer
ok 73 - District 13 has no zoned middle school, per the middle-school layer
  ---
  type: 'test'
  ...
# Subtest: high school is citywide choice, per the high-school layer
ok 74 - high school is citywide choice, per the high-school layer
  ---
  type: 'test'
  ...
# Subtest: compressTimeline collapses repeated holders into one run
ok 75 - compressTimeline collapses repeated holders into one run
  ---
  type: 'test'
  ...
# Subtest: compressTimeline keeps a single-year run
ok 76 - compressTimeline keeps a single-year run
  ---
  type: 'test'
  ...
# Subtest: compressTimeline returns nothing for an empty timeline
ok 77 - compressTimeline returns nothing for an empty timeline
  ---
  type: 'test'
  ...
# Subtest: cohortSurvival tracks the base year cohort forward
ok 78 - cohortSurvival tracks the base year cohort forward
  ---
  type: 'test'
  ...
# Subtest: cohortSurvival reports null shares when the base cohort was empty
ok 79 - cohortSurvival reports null shares when the base cohort was empty
  ---
  type: 'test'
  ...
# Subtest: peerOrgs keeps same-level buildings above the size floor
ok 80 - peerOrgs keeps same-level buildings above the size floor
  ---
  type: 'test'
  ...
# Subtest: peerOrgs excludes buildings at a different level
ok 81 - peerOrgs excludes buildings at a different level
  ---
  type: 'test'
  ...
# Subtest: buildSchoolReport pairs consecutive years and finds the principal
ok 82 - buildSchoolReport pairs consecutive years and finds the principal
  ---
  type: 'test'
  ...
# Subtest: buildSchoolReport marks years where the school is absent from the extract
ok 83 - buildSchoolReport marks years where the school is absent from the extract
  ---
  type: 'test'
  ...
# Subtest: employeeKey normalizes case and surrounding whitespace
ok 84 - employeeKey normalizes case and surrounding whitespace
  ---
  type: 'test'
  ...
# Subtest: employeeKey collapses internal whitespace so "VAN DYKE" matches "VAN  DYKE"
ok 85 - employeeKey collapses internal whitespace so "VAN DYKE" matches "VAN  DYKE"
  ---
  type: 'test'
  ...
# Subtest: employeeKey ignores punctuation differences in names
ok 86 - employeeKey ignores punctuation differences in names
  ---
  type: 'test'
  ...
# Subtest: isTeacher accepts classroom teacher titles
ok 87 - isTeacher accepts classroom teacher titles
  ---
  type: 'test'
  ...
# Subtest: isTeacher rejects non-teaching staff who merely mention teaching
ok 88 - isTeacher rejects non-teaching staff who merely mention teaching
  ---
  type: 'test'
  ...
# Subtest: isSchoolLeader matches principals but not assistant principals
ok 89 - isSchoolLeader matches principals but not assistant principals
  ---
  type: 'test'
  ...
# Subtest: retentionBetween counts staff present in both years
ok 90 - retentionBetween counts staff present in both years
  ---
  type: 'test'
  ...
# Subtest: retentionBetween reports a null rate when the school had no staff to start
ok 91 - retentionBetween reports a null rate when the school had no staff to start
  ---
  type: 'test'
  ...
# Subtest: retentionBetween deduplicates people listed twice in one snapshot
ok 92 - retentionBetween deduplicates people listed twice in one snapshot
  ---
  type: 'test'
  ...
# Subtest: num parses SEDA decimals, including the leading-dot form
ok 93 - num parses SEDA decimals, including the leading-dot form
  ---
  type: 'test'
  ...
# Subtest: num returns null for blanks and junk rather than NaN
ok 94 - num returns null for blanks and junk rather than NaN
  ---
  type: 'test'
  ...
# Subtest: percentileOf places a value in a sorted distribution
ok 95 - percentileOf places a value in a sorted distribution
  ---
  type: 'test'
  ...
# Subtest: percentileOf is null for a missing value or empty distribution
ok 96 - percentileOf is null for a missing value or empty distribution
  ---
  type: 'test'
  ...
# Subtest: joinSeda pairs achievement with covariates and ranks nationally
ok 97 - joinSeda pairs achievement with covariates and ranks nationally
  ---
  type: 'test'
  ...
# Subtest: joinSeda drops schools with no achievement estimate
ok 98 - joinSeda drops schools with no achievement estimate
  ---
  type: 'test'
  ...
# Subtest: joinSeda keeps a school whose growth is missing but level is present
ok 99 - joinSeda keeps a school whose growth is missing but level is present
  ---
  type: 'test'
  ...
# Subtest: joinSeda applies a label override so schools match the retention names
ok 100 - joinSeda applies a label override so schools match the retention names
  ---
  type: 'test'
  ...
# Subtest: New York school ids crosswalk to SEDA’s synthetic citywide LEA
ok 101 - New York school ids crosswalk to SEDA’s synthetic citywide LEA
  ---
  type: 'test'
  ...
# Subtest: the crosswalk refuses anything that is not a twelve-digit NCES id
ok 102 - the crosswalk refuses anything that is not a twelve-digit NCES id
  ---
  type: 'test'
  ...
# Subtest: median of an odd-length list is the middle value
ok 103 - median of an odd-length list is the middle value
  ---
  type: 'test'
  ...
# Subtest: median of an even-length list averages the two middle values
ok 104 - median of an even-length list averages the two middle values
  ---
  type: 'test'
  ...
# Subtest: median of an empty list is null
ok 105 - median of an empty list is null
  ---
  type: 'test'
  ...
# Subtest: percentileRank reports the share of values at or below the target
ok 106 - percentileRank reports the share of values at or below the target
  ---
  type: 'test'
  ...
# Subtest: percentileRank is 0 when nothing is at or below the target
ok 107 - percentileRank is 0 when nothing is at or below the target
  ---
  type: 'test'
  ...
# Subtest: percentileRank of an empty distribution is null
ok 108 - percentileRank of an empty distribution is null
  ---
  type: 'test'
  ...
# Subtest: pooledRate weights each year by its cohort size, unlike a mean of rates
ok 109 - pooledRate weights each year by its cohort size, unlike a mean of rates
  ---
  type: 'test'
  ...
# Subtest: pooledRate is null when there is no cohort at all
ok 110 - pooledRate is null when there is no cohort at all
  ---
  type: 'test'
  ...
# Subtest: ordinal uses st/nd/rd for 1, 2, 3
ok 111 - ordinal uses st/nd/rd for 1, 2, 3
  ---
  type: 'test'
  ...
# Subtest: ordinal uses th for the 11-13 exceptions
ok 112 - ordinal uses th for the 11-13 exceptions
  ---
  type: 'test'
  ...
# Subtest: ordinal uses th for everything else
ok 113 - ordinal uses th for everything else
  ---
  type: 'test'
  ...
# Subtest: pearson is +1 for a perfectly increasing relationship
ok 114 - pearson is +1 for a perfectly increasing relationship
  ---
  type: 'test'
  ...
# Subtest: pearson is -1 for a perfectly inverse relationship
ok 115 - pearson is -1 for a perfectly inverse relationship
  ---
  type: 'test'
  ...
# Subtest: pearson is null when a variable has no spread
ok 116 - pearson is null when a variable has no spread
  ---
  type: 'test'
  ...
# Subtest: pearson is null below two points
ok 117 - pearson is null below two points
  ---
  type: 'test'
  ...
# Subtest: three identical files compare as identical
ok 118 - three identical files compare as identical
  ---
  type: 'test'
  ...
# Subtest: a numeric difference is reported with its size
ok 119 - a numeric difference is reported with its size
  ---
  type: 'test'
  ...
# Subtest: a difference in a text value is reported but does not become a number
ok 120 - a difference in a text value is reported but does not become a number
  ---
  type: 'test'
  ...
# Subtest: a fact missing from one implementation is a difference, not a crash
ok 121 - a fact missing from one implementation is a difference, not a crash
  ---
  type: 'test'
  ...
# Subtest: NA compares equal to NA and unequal to a number
ok 122 - NA compares equal to NA and unequal to a number
  ---
  type: 'test'
  ...
# Subtest: comparing fewer than two implementations is refused
ok 123 - comparing fewer than two implementations is refused
  ---
  type: 'test'
  ...
# Subtest: the report states the verdict, the counts and every hash
ok 124 - the report states the verdict, the counts and every hash
  ---
  type: 'test'
  ...
# Subtest: the report lists the differing facts when there are any
ok 125 - the report lists the differing facts when there are any
  ---
  type: 'test'
  ...
# Subtest: the report is the same text every time it is rendered
ok 126 - the report is the same text every time it is rendered
  ---
  type: 'test'
  ...
# Subtest: each contract gets its own block, named and counted
ok 127 - each contract gets its own block, named and counted
  ---
  type: 'test'
  ...
# Subtest: one failing contract fails the whole report
ok 128 - one failing contract fails the whole report
  ---
  type: 'test'
  ...
# Subtest: a section names its contract and its counts
ok 129 - a section names its contract and its counts
  ---
  type: 'test'
  ...
1..129
# tests 129
# suites 0
# pass 129
# fail 0
# cancelled 0
# skipped 0
# todo 0