Can we use array extensions to speed up string processing with dask

[birdsarah]

Much of the analysis done on this dataset uses dask. Dask is excellent for distributed numerical computing but seems to struggle with strings.

Array extensions unfortunately can't be serialized pandas-dev/pandas#20612

https://github.com/xhochy/fletcher is an array extension that adds string processing functionality.

Doing some standard tasks for this dataset like: collecting all domains, or number of script domains per location domain compare the performance of spark, dask, and dask with extension arrays.

[Muskan284]

@birdsarah ,I am an outreachy applicant.Can i work on this issue?
I have done text pre-processing using pyhton,pandas for my earlier projects and would love to contribute on this issue.

[birdsarah]

Of course! No need to ask. I will upload a notebook tomorrow that contains the kind of text processing I'm trying to speed up.

[birdsarah]

I have added notebook here: https://github.com/mozilla/overscripted/blob/master/analyses/issue_36.ipynb with some examples of string processing that I often find myself doing.

[Muskan284]

Thanks @birdsarah looking into it.

[shruthi0898]

Hi,
Can I contribute to this as well?
Shruthi

[srujana121]

@birdsarah dask is performing much worse than numpy. Any idea why?

[srujana121]

@birdsarah also where should I send you the notebooks?

[birdsarah]

@srujana121

@birdsarah also where should I send you the notebooks?

instructions for submitting a PR are on the landing page: https://github.com/mozilla/overscripted - please ask questions like this on the chat and keep issue questions

@birdsarah dask is performing much worse than numpy. Any idea why?

I could speculate wildly but that wouldn't be productive. Suggest a PR with a concrete examples documenting your observation of "dask performing much worse than numpy."

[birdsarah]

@shruthi0898 there's no need to ask permission to work on an issue.