Merge pull request #56 from TaskarCenterAtUW/feature-3246

[0.3.5 BUG 3246] Fix OSW filename-based schema selection

Author: sujata-m

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Change log
 
+### 0.3.5 - 2026-03-16
+- Fixed filename-based schema selection to use exact dataset suffixes such as `.nodes.geojson`, `.edges.geojson`, and the legacy `.nodes.OSW.geojson` form instead of loose substring matching.
+- Prevented false schema selection for filenames with misleading prefixes such as `gs_metaline_falls_uga.nodes.geojson` and `gs_yarrow_point.edges.geojson`.
+- Updated extracted dataset filename validation to enforce the same suffix-based rules and reject glued names such as `roadEdges.geojson`.
+- Added regression coverage for suffix-based filename matching and refreshed the README to document current ZIP input, validation output, and supported filename patterns.
+
 ### 0.3.4 - 2026-02-04
 - Update leaf_cycle enums in 0.3 lines/points/polygons schemas and add coverage for the new allowed values.
 - Add unit tests that validate acceptance/rejection of leaf_cycle values with the 0.3 schemas.

README.md

@@ -1,20 +1,22 @@
 # TDEI python lib OSW validation package
 
-This package validates the OSW geojson file. Package requires a OSW zip file path
+This package validates OSW GeoJSON datasets packaged as a ZIP file.
 
 ## System requirements
 
 | Software | Version |
 |----------|---------|
-| Python   | 3.10.x  |
+| Python   | >= 3.10 |
 
 ## What this package does?
 
-- It unzip the provided zip files
-- Check for the required nodes and edges geojson files inside the unzipped folder
-- Validate each file (edges, lines, nodes, points, polygons and zones) against the matching schema (0.3 defaults live in `src/python_osw_validation/schema`)
-- Return true or false according to validation
-- you can check the error if it returned false.
+- Extracts the provided ZIP file
+- Finds supported OSW dataset files inside the extracted directory
+- Validates each file (`edges`, `lines`, `nodes`, `points`, `polygons`, and `zones`) against the matching schema
+- Runs cross-file integrity checks such as duplicate `_id` detection and edge or zone references back to nodes
+- Returns a `ValidationResult` object with `is_valid`, `errors`, and `issues`
+
+Any subset of the six supported dataset files may be present. By default, no individual dataset file is required.
 
 ## Starting a new project with template
 
@@ -28,103 +30,105 @@ This package validates the OSW geojson file. Package requires a OSW zip file pat
 from python_osw_validation import OSWValidation
 
 validator = OSWValidation(zipfile_path='<Zip file path>')
-result = validator.validate()  
+result = validator.validate()
 print(result.is_valid)
-print(result.errors) # will return first 20 errors by default if there are errors
+print(result.errors)  # returns up to the first 20 high-level errors by default
+print(result.issues)  # per-file or per-feature issues
 
-result = validator.validate(max_errors=10)  
+result = validator.validate(max_errors=10)
 print(result.is_valid)
-print(result.errors) # will return first 10 errors depending on the max_errors parameter
+print(result.errors)  # returns up to the first 10 high-level errors
+```
+
+You can also override schemas:
+
+```python
+from python_osw_validation import OSWValidation
 
+validator = OSWValidation(
+    zipfile_path='<Zip file path>',
+    schema_paths={
+        'nodes': 'path/to/opensidewalks.nodes.schema-0.3.json',
+        'edges': 'path/to/opensidewalks.edges.schema-0.3.json',
+    },
+)
 ```
 
+## Supported filenames
+
+The validator accepts dataset files whose names end with one of these exact suffixes:
+
+- `.edges.geojson`
+- `.lines.geojson`
+- `.nodes.geojson`
+- `.points.geojson`
+- `.polygons.geojson`
+- `.zones.geojson`
+
+It also accepts the legacy form:
+
+- `.edges.OSW.geojson`
+- `.lines.OSW.geojson`
+- `.nodes.OSW.geojson`
+- `.points.OSW.geojson`
+- `.polygons.OSW.geojson`
+- `.zones.OSW.geojson`
+
+Examples:
+
+- `gs_metaline_falls_uga.nodes.geojson` is valid
+- `gs_yarrow_point.edges.geojson` is valid
+- `roadEdges.geojson` is invalid
+
+If a dataset uses canonical OSW 0.3 names that start with `opensidewalks.`, then only these exact names are allowed:
+
+- `opensidewalks.edges.geojson`
+- `opensidewalks.lines.geojson`
+- `opensidewalks.nodes.geojson`
+- `opensidewalks.points.geojson`
+- `opensidewalks.polygons.geojson`
+- `opensidewalks.zones.geojson`
+
 ### Testing
 
-The project is configured with `python` to figure out the coverage of the unit tests. All the tests are in `tests`
-folder.
+All unit tests are under `tests/unit_tests`.
 
-- To execute the tests, please follow the commands:
+- To execute the tests:
 
   `pip install -r requirements.txt`
 
   `python -m unittest discover -v tests/unit_tests`
 
-- To execute the code coverage, please follow the commands:
+- To execute code coverage:
 
   `coverage run --source=src/python_osw_validation -m unittest discover -v tests/unit_tests`
 
-  `coverage html` // Can be run after 1st command
-
-  `coverage report` // Can be run after 1st command
-
-- After the commands are run, you can check the coverage report in `htmlcov/index.html`. Open the file in any browser,
-  and it shows complete coverage details
-- The terminal will show the output of coverage like this
-
-```shell
-
->  coverage run --source=src/python_osw_validation -m unittest discover -v tests/unit_tests
-test_duplicate_files (test_extracted_data_validator.TestExtractedDataValidator) ... ok
-test_empty_directory (test_extracted_data_validator.TestExtractedDataValidator) ... ok
-test_invalid_directory (test_extracted_data_validator.TestExtractedDataValidator) ... ok
-test_missing_optional_file (test_extracted_data_validator.TestExtractedDataValidator) ... ok
-test_no_geojson_files (test_extracted_data_validator.TestExtractedDataValidator) ... ok
-test_valid_data_at_root (test_extracted_data_validator.TestExtractedDataValidator) ... ok
-test_valid_data_inside_folder (test_extracted_data_validator.TestExtractedDataValidator) ... ok
-test_edges_invalid_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_edges_invalid_zipfile_with_invalid_schema (test_osw_validation.TestOSWValidation) ... ok
-test_edges_invalid_zipfile_with_schema (test_osw_validation.TestOSWValidation) ... ok
-test_external_extension_file_inside_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_external_extension_file_inside_zipfile_with_invalid_schema (test_osw_validation.TestOSWValidation) ... ok
-test_external_extension_file_inside_zipfile_with_schema (test_osw_validation.TestOSWValidation) ... ok
-test_extra_field_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_id_missing_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_invalid_geometry_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_invalid_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_invalid_zipfile_with_invalid_schema (test_osw_validation.TestOSWValidation) ... ok
-test_invalid_zipfile_with_schema (test_osw_validation.TestOSWValidation) ... ok
-test_minimal_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_minimal_zipfile_with_invalid_schema (test_osw_validation.TestOSWValidation) ... ok
-test_minimal_zipfile_with_schema (test_osw_validation.TestOSWValidation) ... ok
-test_missing_identifier_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_no_entity_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_nodes_invalid_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_nodes_invalid_zipfile_with_invalid_schema (test_osw_validation.TestOSWValidation) ... ok
-test_nodes_invalid_zipfile_with_schema (test_osw_validation.TestOSWValidation) ... ok
-test_points_invalid_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_points_invalid_zipfile_with_invalid_schema (test_osw_validation.TestOSWValidation) ... ok
-test_points_invalid_zipfile_with_schema (test_osw_validation.TestOSWValidation) ... ok
-test_valid_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_valid_zipfile_with_invalid_schema (test_osw_validation.TestOSWValidation) ... ok
-test_valid_zipfile_with_schema (test_osw_validation.TestOSWValidation) ... ok
-test_wrong_datatypes_zipfile (test_osw_validation.TestOSWValidation) ... ok
-test_extract_invalid_zip (test_zipfile_handler.TestZipFileHandler) ... ok
-test_extract_valid_zip (test_zipfile_handler.TestZipFileHandler) ... ok
-test_remove_extracted_files (test_zipfile_handler.TestZipFileHandler) ... ok
-
-----------------------------------------------------------------------
-Ran 37 tests in 1284.068s
-
-OK
-```
+  `coverage html`
 
-## Use locally:
+  `coverage report`
+
+After running coverage, open `htmlcov/index.html` to inspect the report in a browser.
+
+## Use locally
 To use the library locally, use the [example.py](./src/example.py) code
 
-## Deployment:
+## Deployment
+
+- The library can be pushed to [TestPyPI](https://test.pypi.org/project/python-osw-validation/) or [PyPI](https://pypi.org/project/python-osw-validation/)
 
-- The library can be pushed to [TestPy](https://test.pypi.org/project/python-osw-validation/) or [PYPI](https://pypi.org/project/python-osw-validation/)
-### Deploy to TestPy
-- On every push to `dev` branch, a workflow is triggered which publishes the updated version to TestPy
+### Deploy to TestPyPI
+
+- On every push to `dev` branch, a workflow is triggered which publishes the updated version to TestPyPI
 
 ### Deploy to PyPI
-- This happens whenever a tag/release is created with `*.*.*` notation (eg. 0.0.8)
-- To change the version, change the version at [version.py](./src/python_osw_validation/version.py)
+
+- This happens whenever a tag or release is created with `*.*.*` notation, for example `0.0.8`
+- To change the version, update [version.py](./src/python_osw_validation/version.py)
 - To release a new version:
-  - Go to Github link of this repository
+  - Go to the GitHub repository
   - Under [releases](https://github.com/TaskarCenterAtUW/TDEI-python-lib-osw-validation/releases), click on `Draft a new release`
-  - Under `choose a new tag`, add a new tag `v*.*.*` , Generate Release notes
+  - Under `choose a new tag`, add a new tag `v*.*.*`, then generate release notes
   - Choose `main` branch for release
   - Publish the release.
-- This release triggers a workflow to generate the new version of the Package.
+- This release triggers a workflow to generate the new package version.
 - The new package will be available at https://pypi.org/project/python-osw-validation/

src/python_osw_validation/__init__.py

@@ -112,21 +112,21 @@ def _get_colset(self, gdf: Optional[gpd.GeoDataFrame], col: str, filekey: str) -
                 return set()
 
     def _schema_key_from_text(self, text: Optional[str]) -> Optional[str]:
-        """Return dataset key (edges/nodes/points/lines/polygons/zones) if mentioned in text."""
+        """Return dataset key from exact filename suffixes only."""
         if not text:
             return None
-        lower = text.lower()
-        aliases = {
-            "edges": ("edge", "edges"),
-            "lines": ("line", "lines", "linestring"),
-            "nodes": ("node", "nodes"),
-            "points": ("point", "points"),
-            "polygons": ("polygon", "polygons", "area"),
-            "zones": ("zone", "zones"),
-        }
-        for key, variants in aliases.items():
-            if any(alias in lower for alias in variants):
+
+        basename = os.path.basename(text).lower()
+        stem, _ = os.path.splitext(basename)
+        for key in self.dataset_schema_paths:
+            if (
+                    stem == key
+                    or stem == f"{key}.osw"
+                    or stem.endswith(f".{key}")
+                    or stem.endswith(f".{key}.osw")
+            ):
                 return key
+
         return None
 
     def _contains_disallowed_features_for_02(self, geojson_data: Dict[str, Any]) -> set:

src/python_osw_validation/extracted_data_validator.py

@@ -48,6 +48,20 @@
 }
 
 
+def _matches_dataset_filename(basename: str, dataset_key: str) -> bool:
+    lower_name = basename.lower()
+    if not lower_name.endswith(".geojson"):
+        return False
+
+    stem = lower_name[:-len(".geojson")]
+    return (
+        stem == dataset_key
+        or stem == f"{dataset_key}.osw"
+        or stem.endswith(f".{dataset_key}")
+        or stem.endswith(f".{dataset_key}.osw")
+    )
+
+
 class ExtractedDataValidator:
     def __init__(self, extracted_dir: str):
         self.extracted_dir = extracted_dir
@@ -100,7 +114,7 @@ def is_valid(self) -> bool:
 
         allowed_keys = tuple(OSW_DATASET_FILES.keys())
         unsupported_files = sorted(
-            {bn for bn in basenames if not any(key in bn for key in allowed_keys)}
+            {bn for bn in basenames if not any(_matches_dataset_filename(bn, key) for key in allowed_keys)}
         )
         if unsupported_files:
             allowed_fmt = ", ".join(allowed_keys)
@@ -121,7 +135,7 @@ def is_valid(self) -> bool:
                 file_count = 0
                 for filename in geojson_files:
                     base_name = os.path.basename(filename)
-                    if required_file in base_name and base_name.endswith('.geojson'):
+                    if _matches_dataset_filename(base_name, required_file):
                         file_count += 1
                         save_filename = filename
                 if file_count == 0:
@@ -138,7 +152,7 @@ def is_valid(self) -> bool:
                 file_count = 0
                 for filename in geojson_files:
                     base_name = os.path.basename(filename)
-                    if optional_file in base_name and base_name.endswith('.geojson'):
+                    if _matches_dataset_filename(base_name, optional_file):
                         file_count += 1
                         save_filename = filename
                 if file_count == 1:

src/python_osw_validation/version.py

@@ -1 +1 @@
-__version__ = '0.3.4'
+__version__ = '0.3.5'

tests/unit_tests/test_extracted_data_validator.py

@@ -69,6 +69,12 @@ def test_valid_subset_of_allowed_files(self):
         self.assertTrue(validator.is_valid())
         self.assertEqual(len(validator.files), 1)
 
+    def test_valid_legacy_osw_suffix_files(self):
+        validator = ExtractedDataValidator(self.test_dir)
+        self.create_files(['wa.microsoft.graph.nodes.OSW.geojson', 'wa.microsoft.graph.edges.OSW.geojson'])
+        self.assertTrue(validator.is_valid())
+        self.assertEqual(len(validator.files), 2)
+
     def test_non_standard_filenames_raise_error(self):
         validator = ExtractedDataValidator(self.test_dir)
         self.create_files(['custom.nodes.geojson', 'opensidewalks.nodes.geojson'])
@@ -109,6 +115,16 @@ def test_unsupported_files_are_rejected(self):
             'Allowed file names are *.{edges, nodes, points, lines, zones, polygons}.geojson'
         )
 
+    def test_glued_dataset_names_are_rejected(self):
+        validator = ExtractedDataValidator(self.test_dir)
+        self.create_files(['roadedges.geojson', 'roadnodes.geojson'])
+        self.assertFalse(validator.is_valid())
+        self.assertEqual(
+            validator.error,
+            'Unsupported .geojson files present: roadedges.geojson, roadnodes.geojson. '
+            'Allowed file names are *.{edges, nodes, points, lines, zones, polygons}.geojson'
+        )
+
 
 if __name__ == '__main__':
     unittest.main()

tests/unit_tests/test_osw_validation_extras.py

@@ -867,6 +867,29 @@ def test_pick_schema_filename_fallback(self):
             v.line_schema_path,
         )
 
+    def test_pick_schema_ignores_prefix_substrings(self):
+        v = OSWValidation(zipfile_path="dummy.zip")
+        self.assertEqual(
+            v.pick_schema_for_file("/tmp/gs_metaline_falls_uga.nodes.geojson", {"features": []}),
+            v.dataset_schema_paths["nodes"],
+        )
+        self.assertEqual(
+            v.pick_schema_for_file("/tmp/gs_yarrow_point.edges.geojson", {"features": []}),
+            v.dataset_schema_paths["edges"],
+        )
+        self.assertEqual(
+            v.pick_schema_for_file("/tmp/wa.microsoft.graph.nodes.OSW.geojson", {"features": []}),
+            v.dataset_schema_paths["nodes"],
+        )
+        self.assertEqual(
+            v.pick_schema_for_file("/tmp/baseline.nodes.geojson", {"features": []}),
+            v.dataset_schema_paths["nodes"],
+        )
+        self.assertEqual(
+            v.pick_schema_for_file("/tmp/roadEdges.geojson", {"features": []}),
+            v.line_schema_path,
+        )
+
     def test_pick_schema_force_single_schema_override(self):
         force = "/forced/opensidewalks.schema-0.3.json"
         v = OSWValidation(zipfile_path="dummy.zip", schema_file_path=force)