record-diff

record-diff

Detect added, changed and removed records between API polls. Persist IDs and SHA-256 fingerprints instead of full payloads. Zero dependencies, Node.js 18+, ES modules and TypeScript declarations. No network calls, telemetry, filesystem access, lifecycle hooks or background workers.

Useful for ETL jobs, product feeds, ticket sync, inventory polling and scheduled automations that should only process new or changed records.

Install

lpm install @lpm.dev/nora-natio.record-diff

Quick start

import { diffRecords } from '@lpm.dev/nora-natio.record-diff';

const first = diffRecords(null, [{ id: 'a', price: 10 }, { id: 'b', price: 20 }]);
// first.added contains both records

const second = diffRecords(first.snapshot, [
  { price: 11, id: 'a' },
  { id: 'c', price: 30 },
]);
// second.added: [{ id: 'c', price: 30 }]
// second.changed: [{ price: 11, id: 'a' }]
// second.removed: ['b']
// second.unchanged: []

// Save second.snapshot with JSON.stringify using your own storage.
// Commit it only after your processing succeeds; this library does not
// provide transactional delivery, retries, locking or exactly-once effects.

Ignore volatile fields and use custom IDs

const options = {
  key: row => String(row.productId),
  select: row => ({ price: row.price, stock: row.stock }),
};
const result = diffRecords(previousSnapshot, completeProductList, options);

Use the same key and select rules on every poll. Reset the snapshot when changing those rules. Selection affects fingerprints, not returned records: added and changed contain the original objects, without cloning or mutation.

API

• diffRecords(previous, records, options?): compare a snapshot (or null for first use) with the complete current collection. Returns { added, changed, removed, unchanged, snapshot }.
• createSnapshot(records, options?): establish a baseline without notifications.
• fingerprint(value): SHA-256 hex digest of the deterministic JSON encoding.
• canonicalJson(value): deterministic JSON encoding with sorted object keys.

Record IDs must be unique, non-empty strings. By default they come from id; key may name another field or be a function. Results preserve current input order; removed IDs preserve previous snapshot order. Record reordering and JSON object key reordering do not count as changes. Array order does count.

Values must be JSON data: null, strings, booleans, finite numbers, dense arrays and plain objects. Cycles, non-finite numbers, undefined, BigInt, Date instances, accessors, sparse arrays, symbol keys and nesting beyond 100 levels are rejected with TypeError. Convert such values explicitly before calling. This encoding is intended for these snapshots; it is not an RFC 8785 implementation.

Operational notes

• Complete collections only. Missing IDs are removals. Finish all API pages before comparing; do not replace good state with a failed or partial fetch.
• Snapshot format: { version: 1, algorithm: 'sha256', entries: [{ id, hash }] }. Validate errors are thrown before a diff is returned. Keep state atomically and use one writer per feed, or supply your own transaction/lock.
• Snapshots retain IDs. Hashes are not encryption and can reveal low-entropy values by guessing. Apply your normal access controls; this is not an anonymization tool or a cryptographic signature scheme.
• Runtime and memory grow with the number and size of records. Bound your input sizes for large feeds. No external service or separate subscription is needed to run the code after you have obtained it.

License and support

MIT. Commercial use, modification and redistribution are allowed under the included license, without a seat limit; those rights do not expire with a registry subscription. Registry access and any subscription billing are managed separately by LPM.dev. No paid support or uptime service is included. For reproducible issues, contact nora@natio.re without including private data.