Publish Raw Data

[leftouterjoins]

Where can we find a dump of the raw map data?

[kaylai]

I came here looking for this as well. I'm interested in making a quick overlay of flock locations on my city's map of council districts to use in communications with my council member. The city actually provides geoJSON files of administrative boundaries. I'm not familiar with how Open Street Map is integrated into a stack like this, but is it possible to provide a geoJSON or similar open standard file(s) with locations of markers and associated metadata?

[Themis3000]

As it relates to the data specifically served by deflock:

The data is split into multiple region files. There isn't a dump of what you're looking for contained in a single file unfortunately.

You can get an index of these files at https://cdn.deflock.me/regions/index.json. That contains a list of all the regions. Use the title_url value to build the url for the region you wish to fetch. For example if you want to fetch 40/0 and the title url is https://cdn.deflock.me/regions/{lat}/{lon}.json?v=1768638893, you'd build https://cdn.deflock.me/regions/40/0.json?v=1768638893. The data here is updated hourly.

Where you can get a more complete dump:

As mentioned by @kaylai, the data is sourced from Open Street Maps. https://overpass-turbo.eu/ is a great place to query OSM data. Overpass turbo allows you to export queried data in many formats, including geoJSON. The following query gets all ALPR's in the world:

[out:json];
node["man_made"="surveillance"]["surveillance:type"="ALPR"];
out body;

This query might take a while to complete. There's quite a lot of entries. (over 30MB)

If you only care about flock cameras, this gets only the flock ones:

[out:json];
node["man_made"="surveillance"]["surveillance:type"="ALPR"]["manufacturer"="Flock Safety"];
out body;

This one might also take a while.

And if you only need a specific area:

[out:json];
{{geocodeArea:Minneapolis}}->.searchArea;
node["man_made"="surveillance"]["surveillance:type"="ALPR"](area.searchArea);
out body;

After you run the query, press the "export" button to save the data. The overpass turbo api will rate limit you very quickly when making large queries, so be efficient about the amount of queries you make.

Hope that's helpful :)

[Themis3000]

And by the way, if you're curious about how deflock fetches it's data, you can read the related source code here: https://github.com/FoggedLens/deflock/blob/master/serverless/alpr_cache/src/alpr_cache.py

[frillweeman]

Thanks @Themis3000 for helping out with the raw data.

I do plan to have an export button sometime and will probably just update the above script to save it as a separate file in S3.

[Themis3000]

Thanks @Themis3000 for helping out with the raw data.

I do plan to have an export button sometime and will probably just update the above script to save it as a separate file in S3.

Sounds good, I may be able to take care of that if you'd like.

[Themis3000]

@frillweeman I'm actually freshening up the alpr_cache.py script right now by chance. One of the roadblocks I hit when making #78 was local only development/testing. I found myself temporarily changing hard coded in values and making temporary changes to alpr_cache.py so I could make a locally built copy of the region files.

I'm working on https://github.com/Themis3000/deflock/tree/alpr_cache_rework to try and improve developer experience in regards to that. It's going to provide an easy way to build region files saved & served on your local system, add an environment variable that controls which endpoint the front end points at so no temporary hard coding is needed, and add simple tests to alpr_cache.py to make sure updates to that file don't cause unexpected breakages.

Eventually I'll make a PR when that's ready for review. I'll be sure to throw in a unsegmented dump upload while I'm at it too in alpr_cache.py

[tsbischof]

If it is useful, I maintain a minified extract here: https://alprwatch.org/pub/extract/flock/planet/planet-latest.geojson.gz. That updates within about a minute of any published changes to OSM. There are also more general extracts for surveillance or any ALPR