DP+ 3.0 tracking issue

[jqnatividad]

DP+ 3.0 will make the Data Resource Upload Flow (DRUF) that was developed for the Texas Water Data Hub generally available.

The DRUF is interactive FAIRification, featuring:

• ckanext-scheming 3.0 powered Dataset Form Pages with a standard Upload page in the beginning, and a Review page at the end.
• metadata formulas - Excel like formula DSL using jinja2 that can calculate/suggest metadata. They can either be pre-calculated and assigned automatically or offered as suggestions.
• Extensible formula helper library - since the formulas are written in jinja2, additional helpers can be easily added. The formula helpers will prioritize precalculating DCAT-US 3 recommended optional properties that are otherwise too hard to calculate manually (DCAT-US 3 in DP+ tracking issue #218)
• suggestion UX/UI - a Suggestion UI using Bootstrap popovers that show the Suggestion and the Formula.

---

• apply scheming 3.0 changes
• reorganize project for maintainability Feature/reorganize project structure #264
  (DO THIS FIRST, BEFORE ITEMS BELOW)
• Commit after truncate to reduce lock contention #263
• incorporate TX Form Pages implementation - specifically, the Registration Formpage and the Upload formpage in the beginning, and the Review Formpage at the end.
• incorporate AI-powered suggestions for Description and Data Dictionary using qsv describegpt
• extend qsv describegpt so that it can use a controlled vocabulary for suggested tags. The controlled vocabulary will be configurable and point to the specified CKAN resource with the following columns tag, description, AI guidance. AI guidance will be customizable instructions when/how to use the associated tag.
• leverage qsv's delimiter inferencing (see Failing to detect separator for semicolon separated file #256 )
• Automatic Empty Date Range handling #261
• make safenames configurable #262
• Stash existing data dictionary before doing a DP+ job. If DP+job fails, the old Data Dictionary should be restored. #265

[tino097]

with #264 changes, we will reorganize the project structure which will help in better maintainability and testability. It will help the whole project to move into better direction and help to identify what can be improved in future.

In the upcoming period we will merge this into main branch so please sync the forks which will help with the merging of all PR's from the users

cc @jqnatividad

[tino097]

Two development branches have been created:

• Dev 2.0
• Dev 3.0

For the next 2–3 months, dev-v2.0 will be kept in sync with main. After that, dev-v3.0 will be merged into main, becoming the primary codebase. The dev-v2.0 branch will remain available for a limited transition period.

My intention is to move Datapusher Plus to the refactored code (v3.0) within the next 2 months, then allow some time for users to migrate to the new version. After the transition period, v1.0 and v2.0 will be deprecated and dropped.

@jqnatividad

Merry Christmas 🎄