-
Notifications
You must be signed in to change notification settings - Fork 0
fsm python Tute 1 Installation
The fsm-python package is based on Apache Spark (PySpark) and is designed to process big data using parallel computing. Therefore, you will need to install Spark to use the fsm tools
You can install Apache Spark form the official download page
curl -O https://downloads.apache.org/spark/spark-3.1.1/spark-3.1.1-bin-hadoop2.7.tgz
Unzip and move to /opt/spark
tar xvf spark-3.1.1-bin-hadoop2.7.tgz
sudo mv spark-3.1.1-bin-hadoop2.7 /opt/spark
Add Spark to your Environment variables:
Eg: for linux
echo '#Add Spark to PATH' >> $HOME/.bashrc && \
echo 'export SPARK_HOME=/opt/spark' >> $HOME/.bashrc && \
echo 'export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/bin' >> $HOME/.bashrc && \
echo 'export PYSPARK_PYTHON=python3' >> $HOME/.bashrc && \
source ~/.bashrc
You can check that pySpark is installed by running:
$ pyspark
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.0.0
/_/
Using Python version 3.8.5 (default, Jul 28 2020 12:59:40)
SparkSession available as 'spark'.
>>> sc
<SparkContext master=local[*] appName=PySparkShell>You can install it easily with pip.
Make sure your public ssh key has been added to this repo, then:
Note: to add your key, contact kip.crossing@gmail.com
pip install git+ssh://git@github.com/soiltechproject/fsm-python.git
To get the latest version:
pip install --upgrade git+ssh://git@github.com/soiltechproject/fsm-python.git
Start a standalone master server:
start-master.sh
Will return the location of a log file. You can look there for find out the URL of your master server.
start-slave.sh spark://ubuntu:7077
Tip You can stop master and slave spark running via:
$ SPARK_HOME/sbin/stop-slave.sh
$ SPARK_HOME/sbin/stop-master.sh
You can then use these nodes when configuring your spark application SparkContext
When starting your new FSM based application, you SHOULD configure your spark context. This will tell the application where and how to deploy.
For example, to run locally:
import fsm
from pyspark import SparkContext, SparkConf
from pyspark.serializers import MarshalSerializer
conf = SparkConf().setMaster("local[*]") \
.setAppName("FarmSoilMapping")\
.set("serializer", MarshalSerializer)\
.set("spark.scheduler.mode", "FAIR")
sc = SparkContext.getOrCreate(conf)
print(sc.uiWebUrl)To learn more about how to configure your spark context, see:
https://spark.apache.org/docs/latest/configuration.html
When it's time to deploy to the cloud, you can use a program such as flintrock
Paper: A conditioned Latin hypercube method for sampling in the presence of ancillary information
Minasny, Budiman, and Alex B. McBratney. 2006. “A Conditioned Latin Hypercube Method for Sampling in the Presence of Ancillary Information.” Computers & Geosciences 32 (9): 1378–88.
Paper: Predicting and Mapping the Soil Available Water Capacity of Australian Wheatbelt.
Padarian, J., B. Minasny, A. B. McBratney, and N. Dalgliesh. 2014. “Predicting and Mapping the Soil Available Water Capacity of Australian Wheatbelt.” Geoderma Regional 2-3 (November): 110–18.
Paper (non-journaled thesis): Provision of soil water retention information for biophysical modelling: an example for Australia
Padarian, J. 2014. "Provision of soil water retention information for biophysical modelling: an example for Australia". Provision of soil information for biophysical modelling: 19-50. Faculty of Agriculture and Environment. The University of Sydney
Paper: Inverse meta-modelling to estimate soil available water capacity at high spatial resolution across a farm
Florin, M. J., A. B. McBratney, B. M. Whelan, and B. Minasny. 2011. “Inverse Meta-Modelling to Estimate Soil Available Water Capacity at High Spatial Resolution across a Farm.” Precision Agriculture 12 (3): 421–38.
Paper: Modelling soil attribute depth functions with equal-area quadratic smoothing splines
Bishop, T. F. A., A. B. McBratney, and G. M. Laslett. 1999. “Modelling Soil Attribute Depth Functions with Equal-Area Quadratic Smoothing Splines.” Geoderma 91 (1): 27–45.
Paper: Validation of digital soil maps at different spatial supports
Bishop, T. F. A., A. Horta, and S. B. Karunaratne. 2015. “Validation of Digital Soil Maps at Different Spatial Supports.” Geoderma 241-242 (March): 238–49.
Paper: Boundaryline analysis of fieldscale yield response to soil properties
Shatar, T. M., and A. B. McBratney. 2004. “Boundary-Line Analysis of Field-Scale Yield Response to Soil Properties.” The Journal of Agricultural Science 142: 553.
Paper: A segmentation algorithm for the delineation of agricultural management zones
Pedroso, Moacir, James Taylor, Bruno Tisseyre, Brigitte Charnomordic, and Serge Guillaume. 2010. “A Segmentation Algorithm for the Delineation of Agricultural Management Zones.” Computers and Electronics in Agriculture 70 (1): 199–208.
Paper: Spatially explicit seasonal forecasting using fuzzy spatiotemporal clustering of long-term daily rainfall and temperature data
Plain, M. B., B. Minasny, A. B. McBratney, and R. W. Vervoort. 2008. “Spatially Explicit Seasonal Forecasting Using Fuzzy Spatiotemporal Clustering of Long-Term Daily Rainfall and Temperature Data.” https://hal.archives-ouvertes.fr/hal-00298949/.
Paper: Spatiotemporal monthly rainfall forecasts for south-eastern and eastern Australia using climatic indices
Montazerolghaem, Maryam, Willem Vervoort, Budiman Minasny, and Alex McBratney. 2016. “Spatiotemporal Monthly Rainfall Forecasts for South-Eastern and Eastern Australia Using Climatic Indices.” Theoretical and Applied Climatology 124 (3): 1045–63.