Skip to content

feat: bulk data example#74

Open
TordAreStromsnes wants to merge 10 commits intomainfrom
feat/bulk-data-example
Open

feat: bulk data example#74
TordAreStromsnes wants to merge 10 commits intomainfrom
feat/bulk-data-example

Conversation

@TordAreStromsnes
Copy link
Copy Markdown
Contributor

Added couple of new notebooks with example of use

Documentation improvements:

  • Renamed the "Fetch Polygons" example to "Polygons" and added "SurfaceGrid" and "Horizon" example notebooks to the Examples section in the mkdocs.yml navigation.

Copy link
Copy Markdown
Contributor

@jucc jucc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In horizons the example uses data from SNORRE_TORDIS_VIGDIS and in Surfaces the example is from Grane.
We are absolutely not allowed to use data from Equinor internal projects as examples, as they are restricted data. Even on internal repos they are not allowed, as users have differnt access levels nd must request it per country or, in the case of externals, per project. But in this case this is even more critical as this is a public repo.
Please replace the examples to use the same project as was used in the polygons notebook, VOLVE_PUBLIC - the Volve data was made public by Equinor and can safely be used.

Additionally, in the horizon plot, it seems the x and y axis start at 0, instead of representing the actual coordinates? That may be confusing to the user, as they are expecting to see the actual x and y values. Is it possible to plot with the real coordinates, as in the polygons and surfaces notebooks?

@TordAreStromsnes
Copy link
Copy Markdown
Contributor Author

Changed the project, but does not exist any data for the horizon schema for VOLVE_PUBLIC

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the code hasn't been changed in the polygons file, please remove this change from this PR.

"id": "a175788c",
"metadata": {},
"source": [
"We need to specify the name of the model we plan to use, as the `dsis-client` library requires this when building the `DSISConfig` object."
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Applies to both horizon and surface notebook]
I think this is an artifact from the previous version of my polygons example. This is no longer true, the model name is not necessary for DSISConfig, and it can be provided in the QueryBuilder section to avoid confusion.

"id": "9c3feab2",
"metadata": {},
"source": [
"### Fetch bulk (binary) data\n",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Applies to both horizon and surface notebook]
We should demonstrate how to get the full binary file with the two methods before proceeding to visualization part.
It might be that since the visualization requires transforming the data and calculating additional information, it does not belong to the dsis-client library. It at least should be tested and compared with the transformations implemented in our current code/pipelines.

"id": "16aafced",
"metadata": {},
"source": [
"#### Method 2: `get_bulk_data_stream()` — stream in chunks\n",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be wrapped into a library method that optionally takes input from the user regarding the chunk size, but other than that just downloads in chunks and appends them and outputs the full file if this is what the end user expects as a final result?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image The chunk_size is already optional from the user in the api, will look into the possibility of appending to a full file for final result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants