# HackerPapers data

This is the complete published **rule-labeled edition** of HackerPapers. No LLM calls were made. A 50-paper human quality gate remains pending; do not describe this edition as human-validated or as meeting a 90% acceptance target.

## Provenance

Source: [open-index/hacker-news](https://huggingface.co/datasets/open-index/hacker-news), locally downloaded snapshot `c54ced5973b95d80cd934b8921f44ec8d87c3cd1`, monthly `data/*/*.parquet` only. The actual source rows, shard count, last creation timestamp, and generation timestamp are in `manifest.json`. `today/` was excluded after existing local validation found timezone, overlap, and missing-item problems. These files are historical, not live data. Original raw files are never modified.

Metadata comes from the arXiv Atom API, Crossref REST API, or publisher citation metadata. Metadata failures remain in the local retry queue and do not become invented catalog entries. Some valid papers on unsupported hosts are outside this edition.

## Method

1. Select non-deleted, non-dead HN comments with a nonempty author. Story submissions do not count as mentions.
2. Parse HTML hrefs and bare URLs on a restricted scholarly-host list. Require at least 100 characters of the comment author's text after excluding anchors and bare links. No PDF downloads or arbitrary redirect crawling.
3. Normalize arXiv abs/pdf/html, ar5iv aliases, legacy identifiers, and version suffixes; lowercase DOIs; canonicalize publisher page/PDF variants where supported. One paper/comment pair counts once.
4. Retain identifiers mentioned by at least three distinct usernames across three distinct **UTC** calendar months. Each threshold is independent; three months need not each have three users.
5. Merge arXiv and DOI editions only where metadata provides an exact DOI. Other duplicates may remain; fuzzy title matching is deliberately absent.
6. Classify stance using the versioned lexical rules in `pipeline/core.py`. Check jokes and explicit criticism before recommendation phrases; default to neutral. These are comment-level labels and can misattribute multi-paper comparisons, quoted statements, negation, or irony. They are not scientific quality judgments. No spam classifier is claimed; short-link-only and dead/deleted comments are filtered.
7. Score **before** capping: number of distinct months with a recommendation or strong recommendation, plus 0.5 times the number of months with a strong recommendation. A month with a strong recommendation contributes 1.5 total. HN karma is not used.
8. Keep at most 20 attributed comments per paper, prioritizing different authors and explicit recommendations while retaining neutral and critical context when space permits. `mention_count` and `months` reflect the full set, not just exported excerpts.
9. Check scholarly metadata. Crossref types accepted: journal-article, proceedings-article, posted-content. Reject explicit book/monograph/lecture-note descriptions, book-review/textbook titles, editorial/correction titles, and Nature's `10.1038/d…` news/commentary DOI family. Hosting on arXiv or assignment of a DOI alone does not make a document a research paper. The remaining type gate can still admit ambiguous non-research preprints; manual review remains necessary. Unknown titles and unresolved metadata do not appear on the site. Rejected identifiers and reasons are recorded in `manifest.json`.
10. Assign one to three topics using documented title/abstract keyword rules. The small taxonomy reflects this first rule-based edition; it is not the proposed 100–150-topic LLM taxonomy. Descriptions are the first abstract sentence, shortened to 340 characters, or empty when unavailable.

## Files and fields

All JSONL files are UTF-8, one JSON object per newline. IDs are strings, even when they contain numeric identifiers. Unknown metadata uses null or an empty array/string, not invented values.

### papers.jsonl

| Field | Meaning |
| --- | --- |
| `id` | Stable canonical identifier: `arxiv:…`, `doi:…`, or `url:…` |
| `slug` | First 16 hex characters of SHA-256(id), used for detail routes |
| `title`, `authors`, `year`, `venue`, `url`, `doi` | Scholarly metadata and an external paper landing URL |
| `abstract` | Source abstract, if available; rights remain with its authors |
| `description`, `description_method` | Abstract excerpt or empty; never an invented summary |
| `topics`, `topic_method` | One to three rule-assigned topic names and provenance |
| `score` | Recommendation-month score, computed on all qualifying mentions |
| `mention_count`, `user_count`, `months` | Full distinct comment count, username count, sorted UTC months |
| `last_mentioned` | Latest qualifying comment timestamp, ISO 8601 UTC |
| `metadata_source`, `review_status` | Source adapter and recorded review state |

### mentions.jsonl

| Field | Meaning |
| --- | --- |
| `paper_id`, `hn_comment_id` | Paper foreign key and original HN comment ID |
| `story_id`, `story_title` | Resolved ancestor story; null when unavailable in the snapshot |
| `author`, `timestamp`, `month` | HN username and creation date/month in UTC |
| `excerpt` | Complete plain-text comment body from the snapshot; HTML is not rendered |
| `stance` | `strong_recommendation`, `recommendation`, `neutral`, `joke`, or `criticism` |
| `stance_method` | `rules-v1`, or an explicitly imported and recorded annotation method |
| `source_url` | Normalized scholarly URL found in the original comment |

### topics.json / manifest.json

Topics contain `id` (route slug), `name`, `paper_ids`, and `count`. The manifest records schema version, source and export counts, thresholds, pending metadata, merged aliases, annotation methods, review state, and content checksums. Do not infer full-archive completeness from metadata-resolved catalog size.

## Attribution and rights

The original archive declares [Open Data Commons Attribution (ODC-By)](https://opendatacommons.org/licenses/by/1-0/). Attribute open-index and Hacker News, preserve these method/provenance notes, and consult the original license. This does not relicense original comments, paper titles/abstracts, or publisher materials; their respective authors and rightsholders retain their rights. Each comment is attributed by username and original HN ID. No claim is made that comment text or paper abstracts are public-domain or available for unrestricted reuse.

Suggested database attribution: “HackerPapers, derived from the open-index Hacker News archive; original discussions and paper metadata attributed to their respective authors.”

Inspiration: [Karpathy's HackerTube](https://hackertube.org) and [HackerBooks](https://hackerbooks.org). This independent project is not affiliated with Hacker News, Y Combinator, or the paper authors.