Proposal: new repo for general survey microdata calculator

[MaxGhenis]

Context

Marina Gindelsky at BEA uses PolicyEngine to calculate taxes and transfers on CPS ASEC microdata that she's scaled to NIPA totals. Currently she goes through policyengine-taxsim's TAXSIM format, but this is limiting — TAXSIM's input format doesn't cover transfer variables (SNAP, Medicaid, etc.) and requires lossy translation from CPS's native structure.

More broadly, many researchers and agencies have their own microdata (CPS, ACS, SCF, SIPP, custom surveys) and want to run PolicyEngine calculations on it. The current options are:

1. policyengine-taxsim — limited to TAXSIM's input/output format
2. policyengine-us Simulation API — powerful but requires manually mapping every variable to PE's entity structure
3. policyengine-us-data — builds our canonical dataset, not designed for user-supplied data

Proposal: policyengine-survey-calculator (or similar name)

A new repo/package that:

1. Accepts common survey microdata formats with prebuilt connectors:

   • CPS ASEC (Marina's immediate need)
   • ACS
   • SCF
   • SIPP
   • Flat CSV with a documented schema

2. Maps survey variables to PolicyEngine's input variables — each connector knows how to translate (e.g., CPS PEMLR -> PE employment_income, CPS household structure -> PE tax units/SPM units)

3. Runs PolicyEngine calculations and returns results merged back onto the original microdata

4. Exports results in the original survey format or flat files

Separation of concerns

Repo	Responsibility
policyengine-us	Tax/benefit rules only
policyengine-us-data	Building the best unified microdata file (Enhanced CPS)
policyengine-taxsim	TAXSIM format emulation specifically
New repo	"Bring your own survey data, get PE calculations back"

The CPS connector code could be shared with policyengine-us-data (e.g., as a dependency or shared utility), since both need to parse CPS ASEC structure into PE entities.

Open questions

• Repo name: policyengine-survey-calculator? policyengine-surveys? policyengine-microdata?
• Should policyengine-taxsim eventually become a thin wrapper around this + a TAXSIM format adapter?
• How much connector logic can be shared with policyengine-us-data's CPS processing?
• Should this support UK surveys too (FRS, SPI) or stay US-focused initially?

Immediate motivation

Marina at BEA needs transfer variable calculations (SNAP, Medicaid, SSI, etc.) on her CPS-based microdata. The TAXSIM format can't express these inputs/outputs. A CPS connector that accepts her data directly would solve this cleanly.

[feenberg]

I would prefer policyengine-microdata.

PE should certainly do a calculator that accepts csv files with all the variables PE supports, and for exactly the reasons given. It should also export as many intermediate values as are available. In my experience there are many surprising results in tax calculations, and having the intermediate results usually makes clear the reason for the result.

Nevertheless, there are 86 unclosed issues for the taxsim emulator, and there have been no replies for several months. Nor is the emulator fast enough to be useful. Could these be addressed before a large project is taken on? That is, could we put off building complex systems to translate the ASEC (and other surveys) into a PE.csv format that is isomorphic to the hdf5 format (but csv) and let the users use that for the time being? Maybe they would even contribute their code. Just accepting a PE variable set in csv format would be much less work that the total effort proposed here, and would let users do what they need to do.

Is there some reason h5py can't do this? Then all that would be necessary is a list of variable names for users to target. If the US and UK models have overlapping variable names, that would mean both were supported.

I am aware that if the hdf5 format isn't a flat file, but (for example) is hierarchical, then a flat csv file will have redundant data. This isn't a real problem that should be addressed. Also, converting person data to taxpayer unit data is complex, and would further divert effort from finishing the comparison project.