[Java] How dictionaries work - roundtrip Java-Python by vibhatha · Pull Request #327 · apache/arrow-cookbook

vibhatha · 2023-09-05T18:18:31Z

This PR closes #213

davisusanibar · 2023-09-06T02:00:23Z

Hi @vibhatha, just reviewing error logs and error mention that arrow-memory-netty is not available on the nightly packages, and that is true, as you can see at https://nightlies.apache.org/arrow/java/org/apache/arrow/arrow-memory-netty/, which is very rare, but it needs to be reviewed.

Alternatively, you could start using the 14.0.0-SNAPSHOT version which can be configured in the arrow-cookbook/java/source/conf.py file as follows:

if arrow_nightly and arrow_nightly != '0':
    version = "14.0.0-SNAPSHOT"
else:
    version = "13.0.0"

vibhatha · 2023-09-06T02:40:12Z

Hi @vibhatha, just reviewing error logs and error mention that arrow-memory-netty is not available on the nightly packages, and that is true, as you can see at https://nightlies.apache.org/arrow/java/org/apache/arrow/arrow-memory-netty/, which is very rare, but it needs to be reviewed.

Alternatively, you could start using the 14.0.0-SNAPSHOT version which can be configured in the arrow-cookbook/java/source/conf.py file as follows:
if arrow_nightly and arrow_nightly != '0':
    version = "14.0.0-SNAPSHOT"
else:
    version = "13.0.0"

If I understand correctly, should I make that change in this PR itself? Or should we create a separate PR for that?

davisusanibar · 2023-09-06T02:51:34Z

Hi @vibhatha, just reviewing error logs and error mention that arrow-memory-netty is not available on the nightly packages, and that is true, as you can see at https://nightlies.apache.org/arrow/java/org/apache/arrow/arrow-memory-netty/, which is very rare, but it needs to be reviewed.
Alternatively, you could start using the 14.0.0-SNAPSHOT version which can be configured in the arrow-cookbook/java/source/conf.py file as follows:
if arrow_nightly and arrow_nightly != '0':
    version = "14.0.0-SNAPSHOT"
else:
    version = "13.0.0"
If I understand correctly, should I make that change in this PR itself? Or should we create a separate PR for that?

This change is being added to #320 as well, but it is true, this must have been created independently.

vibhatha · 2023-09-06T02:52:42Z

@davisusanibar sounds good and I will rebase once this PR is merged. Thank you 👍

vibhatha · 2023-09-11T17:24:02Z

@danepitkin @davisusanibar This PR is ready for review, please take a look.

danepitkin

Thanks @vibhatha ! This will be a great addition to the cook book. I'm not super familiar with the C Data interface / jpype so reviewing this was a good learning opportunity for me.

danepitkin · 2023-09-12T15:20:41Z

   data
   avro
   jdbc
+   python_java


Would "c data interface" potentially be a better name? The section below uses pyarrow/jpype as a python example, but in the future we could also add other language interfaces.

@danepitkin
Thank you for the quick review and suggestions. And I agree with your comment. Will make the changed.

davisusanibar · 2023-09-12T17:55:14Z

+    In the Java component, the MapValuesConsumer class receives data from the Python component through C Data. 
+    It then updates the data and sends it back to the Python component.
+
+.. code-block:: java


Java component is tested by custom directives created with testcode and testoutput.

During internal works, Java code is executed by Jshell and output defined by system.out.print on the testcode is compared to the value defined on the test output, such as:

.. testcode:: System.out.print("testme"); .. testoutput:: testme

davisusanibar · 2023-09-12T17:58:29Z

+            return this.vector;
+        }
+
+        public void update(long c_array_ptr, long c_schema_ptr) {


Could be an option to also validate Python call as as part of Java testing using .. testcode:: and .. testoutput:: directives?

Something like this:

import org.apache.arrow.c.ArrowArray; import org.apache.arrow.c.ArrowSchema; import org.apache.arrow.c.CDataDictionaryProvider; import org.apache.arrow.c.Data; import org.apache.arrow.memory.BufferAllocator; import org.apache.arrow.memory.RootAllocator; import org.apache.arrow.vector.BigIntVector; import org.apache.arrow.vector.FieldVector; public class MapValuesConsumer { private final static BufferAllocator allocator = new RootAllocator(); private final CDataDictionaryProvider provider; private FieldVector vector; public MapValuesConsumer(CDataDictionaryProvider provider) { this.provider = provider; } public MapValuesConsumer() { this.provider = null; } public static BufferAllocator getAllocatorForJavaConsumer() { return allocator; } public FieldVector getVector() { return this.vector; } public void update(long c_array_ptr, long c_schema_ptr) { ArrowArray arrow_array = ArrowArray.wrap(c_array_ptr); ArrowSchema arrow_schema = ArrowSchema.wrap(c_schema_ptr); this.vector = Data.importVector(allocator, arrow_array, arrow_schema, this.provider); this.doWorkInJava(vector); } public void update2(long c_array_ptr, long c_schema_ptr) { ArrowArray arrow_array = ArrowArray.wrap(c_array_ptr); ArrowSchema arrow_schema = ArrowSchema.wrap(c_schema_ptr); this.vector = Data.importVector(allocator, arrow_array, arrow_schema, null); this.doWorkInJava(vector); } private void doWorkInJava(FieldVector vector) { System.out.println("Doing work in Java"); BigIntVector bigIntVector = (BigIntVector)vector; bigIntVector.setSafe(0, 2); } public static void main(String[] args) { simulateAsAJavaConsumers(); } final static BigIntVector intVector = new BigIntVector("internal_test", allocator); public static BigIntVector getIntVectorForJavaConsumers() { intVector.allocateNew(3); intVector.set(0, 1); intVector.set(1, 7); intVector.set(2, 93); intVector.setValueCount(3); return intVector; } public static void simulateAsAJavaConsumers() { MapValuesConsumer mvc = new MapValuesConsumer();//FIXME! Use constructor with dictionary provider try ( ArrowArray arrowArray = ArrowArray.allocateNew(allocator); ArrowSchema arrowSchema = ArrowSchema.allocateNew(allocator) ) { //FIXME! Add custo logic to emulate a dictionary provider adding Data.exportVector(allocator, getIntVectorForJavaConsumers(), null, arrowArray, arrowSchema); mvc.update2(arrowArray.memoryAddress(), arrowSchema.memoryAddress()); try (FieldVector valueVectors = Data.importVector(allocator, arrowArray, arrowSchema, null);) { System.out.print(valueVectors); //FIXME! Validate on .. testoutput:: } } intVector.close(); //FIXME! Expose this method also to be called by end python program allocator.close(); } }

Thank you for the insight, @davisusanibar. I will work on this.

@davisusanibar this code doesn't seem to be working, but I get your idea.

vibhatha · 2023-09-13T06:51:09Z

@danepitkin I am addressing the reviews and thank you for reviewing this PR.

danepitkin

Nice work @vibhatha ! A few small comments, but overall LGTM otherwise.

vibhatha · 2023-09-14T02:45:07Z

@danepitkin I am working on updating this PR for Java doctests. I will address the reviews as well.

lidavidm · 2023-09-20T15:32:05Z

+    This section demonstrates a data roundtrip, where a dictionary array is created in Python, accessed and updated in Java,
+    and finally re-accessed and validated in Python for data consistency.


Suggested change

This section demonstrates a data roundtrip, where a dictionary array is created in Python, accessed and updated in Java,

and finally re-accessed and validated in Python for data consistency.

This section demonstrates a data roundtrip, where a dictionary array is created in Python, accessed and updated in Java,

and finally re-accessed and validated in Python for data consistency.

This description is misleading. You cannot mutate data in the C Data Interface.

I changed the wording.

This section demonstrates a data roundtrip where C Data interface is being used to provide the seamless access to data across language boundaries.

lidavidm · 2023-09-20T15:37:55Z

+    In the Java component, the MapValuesConsumer class receives data from the Python component through C Data. 
+    It then updates the data and sends it back to the Python component.
+
+.. testcode::


Format the code here as well.

lidavidm · 2023-09-20T15:38:06Z

+    In the Java component, the MapValuesConsumer class receives data from the Python component through C Data. 
+    It then updates the data and sends it back to the Python component.


Suggested change

In the Java component, the MapValuesConsumer class receives data from the Python component through C Data.

It then updates the data and sends it back to the Python component.

In the Java component, the MapValuesConsumer class receives data from the Python component through C Data.

It then updates the data and sends it back to the Python component.

Also, misleading wording

Changed the wording. Thanks for catching this.

vibhatha · 2023-09-20T15:53:07Z

@lidavidm thanks a lot for your feedback. I will address these.

lidavidm · 2023-09-20T18:15:27Z

Given we have no roundtrip examples at all, I don't see why we are starting with dictionaries vs a simpler type

davisusanibar · 2023-09-20T19:11:37Z

Given we have no roundtrip examples at all, I don't see why we are starting with dictionaries vs a simpler type

Could it be consider as a simpler type #325?

vibhatha · 2023-09-21T02:30:12Z

Given we have no roundtrip examples at all, I don't see why we are starting with dictionaries vs a simpler type

Good point. Although, I am merely picking up an existing issue.

vibhatha added 6 commits September 11, 2023 06:02

feat: initial draft

3704302

fix: format

71877c8

fix: text formatting

0b029e4

fix: python code cleanup

740a139

fix: adding python output

d2b0491

fix: minor wording issue

c35eefd

vibhatha force-pushed the feat-dictionary-python-java-rndtrp branch from 7befe38 to c35eefd Compare September 11, 2023 00:33

vibhatha marked this pull request as ready for review September 11, 2023 17:12

danepitkin reviewed Sep 12, 2023

View reviewed changes

davisusanibar reviewed Sep 12, 2023

View reviewed changes

fix: address reviews v1

46bba74

vibhatha force-pushed the feat-dictionary-python-java-rndtrp branch from 68d140b to 46bba74 Compare September 13, 2023 04:40

fix: minor

8508059

danepitkin reviewed Sep 13, 2023

View reviewed changes

Comment thread java/source/python_java.rst Outdated

Comment thread java/source/python_java.rst Outdated

Comment thread java/source/python_java.rst Outdated

Comment thread java/source/python_java.rst Outdated

Comment thread java/source/python_java.rst Outdated

fix: addressing reviews

9329cda

lidavidm requested changes Sep 20, 2023

View reviewed changes

vibhatha added 2 commits September 21, 2023 08:09

fix: address reviews v2

23617f0

fix: format

59ac6bb

fix: indentation

33e2cbf

vibhatha requested a review from lidavidm September 25, 2023 14:07

		This section demonstrates a data roundtrip, where a dictionary array is created in Python, accessed and updated in Java,
		and finally re-accessed and validated in Python for data consistency.

		In the Java component, the MapValuesConsumer class receives data from the Python component through C Data.
		It then updates the data and sends it back to the Python component.

Conversation

vibhatha commented Sep 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davisusanibar commented Sep 6, 2023

Uh oh!

vibhatha commented Sep 6, 2023

Uh oh!

davisusanibar commented Sep 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vibhatha commented Sep 6, 2023

Uh oh!

vibhatha commented Sep 11, 2023

Uh oh!

danepitkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vibhatha commented Sep 13, 2023

Uh oh!

danepitkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vibhatha commented Sep 14, 2023

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vibhatha commented Sep 20, 2023

Uh oh!

lidavidm commented Sep 20, 2023

Uh oh!

davisusanibar commented Sep 20, 2023

Uh oh!

vibhatha commented Sep 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

vibhatha commented Sep 5, 2023 •

edited

Loading

davisusanibar commented Sep 6, 2023 •

edited

Loading