[improve] Upgrade pulsar-client-python to 3.8.0 in Docker image

[lhotari]

Motivation

pulsar-client-python 3.8.0 has been released with fixes and it would be useful to upgrade pulsar-client-python used in Pulsar docker image.

Modifications

• upgrade pulsar-client-python version to 3.8.0
• add script for re-generating python protobuf and grpc stubs used in Pulsar Functions
• update python protobuf and grpc stubs
• pin python pip protobuf to 6.31.1 and grpc to 1.73.1 since these were used to generate the stubs

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[lhotari]

test failure in python related test:

  Error:  org.apache.pulsar.tests.integration.functions.PulsarStateTest.testPythonWordCountFunction  Time elapsed: 300.007 s  <<< FAILURE!
  org.testng.internal.thread.ThreadTimeoutException: Method org.apache.pulsar.tests.integration.functions.PulsarStateTest.testPythonWordCountFunction() didn't finish within the time-out 300000
  	at java.base/jdk.internal.misc.Unsafe.park(Native Method)
  	at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
  	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
  	at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
  	at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
  	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1712)
  	at org.apache.pulsar.common.util.collections.GrowableArrayBlockingQueue.take(GrowableArrayBlockingQueue.java:197)
  	at org.apache.pulsar.client.impl.ConsumerImpl.internalReceive(ConsumerImpl.java:517)
  	at org.apache.pulsar.client.impl.ConsumerBase.receive(ConsumerBase.java:264)
  	at org.apache.pulsar.tests.integration.functions.PulsarStateTest.publishAndConsumeMessages(PulsarStateTest.java:609)
  	at org.apache.pulsar.tests.integration.functions.PulsarStateTest.doTestPythonWordCountFunction(PulsarStateTest.java:95)
  	at org.apache.pulsar.tests.integration.functions.PulsarStateTest.testPythonWordCountFunction(PulsarStateTest.java:75)
  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  	at java.base/java.lang.Thread.run(Thread.java:1583)

[lhotari]

Found the python related error in logs:

2025-07-22T13:50:15.843245326Z 2025-07-22T13:50:15,843+0000 [function-timer-thread-104-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Creating function log directory logs//functions/public/default/test-wordcount-py-fn-dvjvzzau
2025-07-22T13:50:15.843381144Z 2025-07-22T13:50:15,843+0000 [function-timer-thread-104-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Created or found function log directory logs//functions/public/default/test-wordcount-py-fn-dvjvzzau
2025-07-22T13:50:15.843488732Z 2025-07-22T13:50:15,843+0000 [function-timer-thread-104-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - ProcessBuilder starting the process with args python3 /pulsar/instances/python-instance/python_instance_main.py --py /pulsar/download/pulsar_functions/public/default/test-wordcount-py-fn-dvjvzzau/0/wordcount_function.py --logging_directory logs//functions --logging_file test-wordcount-py-fn-dvjvzzau --logging_config_file /pulsar/conf/functions-logging/logging_config.ini --instance_id 0 --function_id 9c4218dc-01be-4cc8-a752-be47853460ab --function_version 0c43a98b-8731-4a6e-9350-71c1567c2fa3 --function_details '{"tenant":"public","namespace":"default","name":"test-wordcount-py-fn-dvjvzzau","className":"wordcount_function.WordCountFunction","runtime":"PYTHON","autoAck":true,"parallelism":1,"source":{"inputSpecs":{"test-wordcount-py-input-xwrapdis":{}},"cleanupSubscription":true},"sink":{"topic":"test-wordcount-py-output-ovmhbmug","forwardSourceMessageProperty":true},"resources":{"cpu":1.0,"ram":"1073741824","disk":"10737418240"},"componentType":"FUNCTION"}' --pulsar_serviceurl pulsar://localhost:6650 --max_buffered_tuples 1024 --port 39523 --metrics_port 37195 --state_storage_serviceurl bk://127.0.0.1:4181 --expected_healthcheck_interval 30 --secrets_provider secretsprovider.ClearTextSecretsProvider --cluster_name standalone
2025-07-22T13:50:15.846003070Z 2025-07-22T13:50:15,845+0000 [function-timer-thread-104-1] INFO  org.apache.pulsar.functions.runtime.process.ProcessRuntime - Started process successfully
2025-07-22T13:50:16.097383975Z Traceback (most recent call last):
2025-07-22T13:50:16.097406687Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 636, in _ConvertFieldValuePair
2025-07-22T13:50:16.097586414Z     self._ConvertMapFieldValue(
2025-07-22T13:50:16.097592856Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 833, in _ConvertMapFieldValue
2025-07-22T13:50:16.097839397Z     getattr(message, field.name)[key_value],
2025-07-22T13:50:16.097907004Z     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
2025-07-22T13:50:16.097911182Z   File "/usr/lib/python3.12/site-packages/google/protobuf/internal/containers.py", line 70, in __getitem__
2025-07-22T13:50:16.098044201Z     return self._values[key]
2025-07-22T13:50:16.098142318Z            ~~~~~~~~~~~~^^^^^
2025-07-22T13:50:16.098150653Z TypeError: list indices must be integers or slices, not str
2025-07-22T13:50:16.098155092Z 
2025-07-22T13:50:16.098158698Z The above exception was the direct cause of the following exception:
2025-07-22T13:50:16.098162526Z 
2025-07-22T13:50:16.098166102Z Traceback (most recent call last):
2025-07-22T13:50:16.098169368Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 684, in _ConvertFieldValuePair
2025-07-22T13:50:16.098404749Z     self.ConvertMessage(value, sub_message, '{0}.{1}'.format(path, name))
2025-07-22T13:50:16.098480150Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 540, in ConvertMessage
2025-07-22T13:50:16.098699912Z     self._ConvertFieldValuePair(value, message, path)
2025-07-22T13:50:16.098759733Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 702, in _ConvertFieldValuePair
2025-07-22T13:50:16.098994023Z     raise ParseError(
2025-07-22T13:50:16.099001526Z google.protobuf.json_format.ParseError: Failed to parse inputSpecs field: list indices must be integers or slices, not str.
2025-07-22T13:50:16.099005444Z 
2025-07-22T13:50:16.099008860Z The above exception was the direct cause of the following exception:
2025-07-22T13:50:16.099012457Z 
2025-07-22T13:50:16.099015392Z Traceback (most recent call last):
2025-07-22T13:50:16.099018438Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 318, in <module>
2025-07-22T13:50:16.099214756Z     main()
2025-07-22T13:50:16.099265337Z   File "/pulsar/instances/python-instance/python_instance_main.py", line 165, in main
2025-07-22T13:50:16.099395865Z     json_format.Parse(args.function_details, function_details)
2025-07-22T13:50:16.099449836Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 464, in Parse
2025-07-22T13:50:16.099647907Z     raise e
2025-07-22T13:50:16.099654700Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 460, in Parse
2025-07-22T13:50:16.099860645Z     return ParseDict(
2025-07-22T13:50:16.099867588Z            ^^^^^^^^^^
2025-07-22T13:50:16.099870924Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 494, in ParseDict
2025-07-22T13:50:16.100043718Z     parser.ConvertMessage(js_dict, message, '')
2025-07-22T13:50:16.100107978Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 540, in ConvertMessage
2025-07-22T13:50:16.100319200Z     self._ConvertFieldValuePair(value, message, path)
2025-07-22T13:50:16.100377910Z   File "/usr/lib/python3.12/site-packages/google/protobuf/json_format.py", line 692, in _ConvertFieldValuePair
2025-07-22T13:50:16.100588585Z     raise ParseError(
2025-07-22T13:50:16.100593835Z google.protobuf.json_format.ParseError: Failed to parse source field: Failed to parse inputSpecs field: list indices must be integers or slices, not str..

@BewareMyPower Any recommendation about how to fix this?

seems to originate to this line of code:

pulsar/pulsar-functions/instance/src/main/python/python_instance_main.py

Line 165 in de33f3b

json_format.Parse(args.function_details, function_details)

[BewareMyPower]

This might be caused by the upgrade of protobuf in apache/pulsar-client-python#260 for CVE. The python protobuf API might have changed. I guess that we need to regenerate the Function_pb2.py with latest protobuf. I will take a look tomorrow.

[BewareMyPower]

BTW, it also reminds me of the discussion I started long days ago: https://lists.apache.org/thread/5s2l9vgt5f973psb6xlttpmy4rpym7zh

The Python function related code and tests are maintained in the apache/pulsar repo, while the extra dependencies are still maintained in https://github.com/apache/pulsar-client-python/blob/main/setup.py. If we can separate the [function] dependencies, it could be easier to maintain and the core repo won't have to wait for a new Python client release.

[lhotari]

This might be caused by the upgrade of protobuf in apache/pulsar-client-python#260 for CVE. The python protobuf API might have changed. I guess that we need to regenerate the Function_pb2.py with latest protobuf. I will take a look tomorrow.

Yes, it seems to be caused by that change. There are multiple *_pb2.py files in the apache/pulsar repository:

bin/proto/MLDataFormats_pb2.py
pulsar-functions/instance/src/main/python/Function_pb2.py
pulsar-functions/instance/src/main/python/InstanceCommunication_pb2.py
pulsar-functions/instance/src/main/python/InstanceCommunication_pb2_grpc.py

I guess all of them would have to be re-generated.
I didn't find the instructions or a script in the code base other than this:

pulsar/bin/pulsar-managed-ledger-admin

Lines 33 to 35 in de53c29

	sys.exit("Incompatible proto/MLDataFormats_pb2.py. Regenerate using: "+
	"protoc -I=${PULSAR_PATH}/managed-ledger/src/main/proto --python_out=${PULSAR_PATH}/bin/proto/ "+
	"${PULSAR_PATH}/managed-ledger/src/main/proto/MLDataFormats.proto")

[lhotari]

Let's see if the BookKeeper stream storage protobuf and grpc stubs are compatible. The stubs at https://github.com/apache/bookkeeper/tree/master/stream/clients/python/bookkeeper/proto haven't been updated for ages.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.31%. Comparing base (bbc6224) to head (d9cda77).
⚠️ Report is 1393 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #24544      +/-   ##
============================================
+ Coverage     73.57%   74.31%   +0.74%     
+ Complexity    32624    32575      -49     
============================================
  Files          1877     1874       -3     
  Lines        139502   146231    +6729     
  Branches      15299    16772    +1473     
============================================
+ Hits         102638   108673    +6035     
- Misses        28908    28922      +14     
- Partials       7956     8636     +680     

Flag	Coverage Δ
inttests	26.70% <ø> (+2.12%)	⬆️
systests	23.30% <ø> (-1.03%)	⬇️
unittests	73.79% <ø> (+0.95%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1111 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[BewareMyPower]

Are these new scripts documented or integrated into Maven build?

[lhotari]

Are these new scripts documented or integrated into Maven build?

@BewareMyPower I've added docs now. There's no intention to integrate into the Maven build since the stubs would only need to be updated when the grpc version changes since grpcio-tools is used to generate stubs. grpcio-tools includes a specific libprotoc library so the protobuf version should match this version. I've documented this too. PTAL