From 9735700ab7c817d9e58a4c90157e5a3bfc3734c9 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Fri, 30 Jan 2026 18:47:24 +0100 Subject: [PATCH 01/27] test scripts for changed code --- _episodes/08-justin-job-submission.md | 502 +++++++++++++++++++++++ _extras/short_submission.md | 96 +++++ _includes/makercds.sh | 10 + _includes/setup-grid | 129 ++++++ _includes/setup-local | 129 ++++++ _includes/setup_before_submit.sh | 25 ++ _includes/setup_local_test.sh | 23 ++ _includes/submit_local_code.jobscript.sh | 118 ++++++ _includes/submit_workflow.sh | 7 + 9 files changed, 1039 insertions(+) create mode 100644 _episodes/08-justin-job-submission.md create mode 100644 _extras/short_submission.md create mode 100644 _includes/makercds.sh create mode 100644 _includes/setup-grid create mode 100644 _includes/setup-local create mode 100644 _includes/setup_before_submit.sh create mode 100644 _includes/setup_local_test.sh create mode 100644 _includes/submit_local_code.jobscript.sh create mode 100644 _includes/submit_workflow.sh diff --git a/_episodes/08-justin-job-submission.md b/_episodes/08-justin-job-submission.md new file mode 100644 index 0000000..86249b0 --- /dev/null +++ b/_episodes/08-justin-job-submission.md @@ -0,0 +1,502 @@ +--- +title: Jobsub Grid Job Submission and Common Errors - justIn Version +teaching: 65 +exercises: 0 +questions: +- How to submit grid jobs? +objectives: +- Submit a basic batchjob and understand what's happening behind the scenes +- Monitor the job and look at its outputs +- Review best practices for submitting jobs (including what NOT to do) +keypoints: +- When in doubt, ask! Understand that policies and procedures that seem annoying, overly complicated, or unnecessary (especially when compared to running an interactive test) are there to ensure efficient operation and scalability. They are also often the result of someone breaking something in the past, or of simpler approaches not scaling well. +- Send test jobs after creating new workflows or making changes to existing ones. If things don't work, don't blindly resubmit and expect things to magically work the next time. +- Only copy what you need in input tar files. In particular, avoid copying log files, .git directories, temporary files, etc. from interactive areas. +- Take care to follow best practices when setting up input and output file locations. +- Always, always, always prestage input datasets. No exceptions. +--- + + + +#### Session Video + +This session will be captured on video a placed here after the workshop for asynchronous study. + +The video from the two day version of this training in May 2022 is provided [here](https://www.youtube.com/embed/QuDxkhq64Og) as a reference. + + + +#### Live Notes + + + + + + + + + +## Submit a job + +Go to [The Justin Tutorial](https://dunejustin.fnal.gov/docs/tutorials.dune.md) + +and work up to ["run some hello world jobs"](https://dunejustin.fnal.gov/docs/tutorials.dune.md#run-some-hello-world-jobs) + +> ## Quiz +> +> 1. What is your workflow ID? +> +{: .solution} + +Then work through + +- [View your workflow on the justIN web dashboard](https://dunejustin.fnal.gov/docs/tutorials.dune.md#view-your-workflow-on-the-justin-web-dashboard) +- [Jobs with inputs and outputs](https://dunejustin.fnal.gov/docs/tutorials.dune.md#jobs-with-inputs-and-outputs) +- [Fetching files from Rucio managed storage](https://dunejustin.fnal.gov/docs/tutorials.dune.md#fetching-files-from-rucio-managed-storage) +- (skip for now) Jobs using GPUs +- [Jobs writing to scratch](https://dunejustin.fnal.gov/docs/tutorials.dune.md#jobs-writing-to-scratch) + + + + + +## Submit a job using the tarball containing custom code + +First off, a very important point: for running analysis jobs, **you may not actually need to pass an input tarball**, especially if you are just using code from the base release and you don't actually modify any of it. In that case, it is much more efficient to use everything from the release and refrain from using a tarball. +All you need to do is set up any required software from CVMFS (e.g. dunetpc and/or protoduneana), and you are ready to go. +If you're just modifying a fcl file, for example, but no code, it's actually more efficient to copy just the fcl(s) you're changing to the scratch directory within the job, and edit them as part of your job script (copies of a fcl file in the current working directory have priority over others by default). + +Sometimes, though, we need to run some custom code that isn't in a release. +We need a way to efficiently get code into jobs without overwhelming our data transfer systems. +We have to make a few minor changes to the scripts you made in the previous tutorial section, generate a tarball, and invoke the proper jobsub options to get that into your job. +There are many ways of doing this but by far the best is to use the Rapid Code Distribution Service (RCDS), as shown in our example. + +If you have finished up the LArSoft follow-up and want to use your own code for this next attempt, feel free to tar it up (you won't need anything besides the localProducts* and work directories) and use your own tar ball in lieu of the one in this example. +You will have to change the last line with your own submit file instead of the pre-made one. + +First, we should make a tarball. Here is what we can do + +~~~ +cd mybuild +tar --exclude '.git' -czf mybuild.tar.gz * +~~~ + +Follow the instructions at: + +[rapid code distribution to jobs via cvmfs](https://dunejustin.fnal.gov/docs/tutorials.dune.md#rapid-code-distribution-to-jobs-via-cvmfs) + + + + + + +Before we continue, let's examine these files a bit. We will source the first one in our job script, and it will set up the environment for us. + +~~~ +#!/bin/bash + +DIRECTORY=mybuild +# we cannot rely on "whoami" in a grid job. We have no idea what the local username will be. +# Use the GRID_USER environment variable instead (set automatically by jobsub). +USERNAME=${GRID_USER} + +source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh +export WORKDIR=${_CONDOR_JOB_SCRATCH} # if we use the RCDS the our tarball will be placed in $INPUT_TAR_DIR_LOCAL. +if [ ! -d "$WORKDIR" ]; then + export WORKDIR=`echo .` +fi + +source ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts*/setup-grid +mrbslp +~~~ +{: .source} + + + +As you can see, we have switched from the hard-coded directories to directories defined by environment variables; the `INPUT_TAR_DIR_LOCAL` variable will be set for us (see below). +Now, let's actually create our tar file. + +Note how we have excluded the contents of ".git" directories in the various packages, since we don't need any of that in our jobs. It turns out that the .git directory can sometimes account for a substantial fraction of a package's size on disk! + +Then submit another job (in the following we keep the same submit file as above): + +```bash +jobsub_submit -G dune --mail_always -N 1 --memory=2500MB --disk=2GB --expected-lifetime=3h --cpu=1 --tar_file_name=dropbox:///exp/dune/app/users//sep2025tutorial.tar.gz --singularity-image /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest --append_condor_requirements='(TARGET.HAS_Singularity==true&&TARGET.HAS_CVMFS_dune_opensciencegrid_org==true&&TARGET.HAS_CVMFS_larsoft_opensciencegrid_org==true&&TARGET.CVMFS_dune_opensciencegrid_org_REVISION>=1105&&TARGET.HAS_CVMFS_fifeuser1_opensciencegrid_org==true&&TARGET.HAS_CVMFS_fifeuser2_opensciencegrid_org==true&&TARGET.HAS_CVMFS_fifeuser3_opensciencegrid_org==true&&TARGET.HAS_CVMFS_fifeuser4_opensciencegrid_org==true)' -e GFAL_PLUGIN_DIR=/usr/lib64/gfal2-plugins -e GFAL_CONFIG_DIR=/etc/gfal2.d file:///exp/dune/app/users/kherner/run_sep2025tutorial.sh +``` + +You'll see this is very similar to the previous case, but there are some new options: + +* `--tar_file_name=dropbox://` automatically **copies and untars** the given tarball into a directory on the worker node, accessed via the INPUT_TAR_DIR_LOCAL environment variable in the job. The value of INPUT_TAR_DIR_LOCAL is by default $CONDOR_DIR_INPUT/name_of_tar_file_without_extension, so if you have a tar file named e.g. sep2025tutorial.tar.gz, it would be $CONDOR_DIR_INPUT/sep2025tutorial. +* Notice that the `--append_condor_requirements` line is longer now, because we also check for the fifeuser[1-4]. opensciencegrid.org CVMFS repositories. + +The submission output will look something like this: + +~~~ +Attempting to get token from https://htvaultprod.fnal.gov:8200 ... succeeded +Storing bearer token in /tmp/bt_token_dune_Analysis_11469 +Using bearer token located at /tmp/bt_token_dune_Analysis_11469 to authenticate to RCDS +Checking to see if uploaded file is published on RCDS +Could not locate uploaded file on RCDS. Will retry in 30 seconds. +Could not locate uploaded file on RCDS. Will retry in 30 seconds. +Could not locate uploaded file on RCDS. Will retry in 30 seconds. +Found uploaded file on RCDS. +Transferring files to web sandbox... +Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/run_sep2025tutorial.sh [DONE] after 0s +Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/simple.cmd [DONE] after 0s +Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/simple.sh [DONE] after 0s +Submitting job(s). +1 job(s) submitted to cluster 62007523. + +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- + +Use job id 62007523.0@jobsub01.fnal.gov to retrieve output +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- +~~~ + +Note that the job submission will pause while it uploads the tarball to RCDS, and then it continues normally. + +Now, there's a very small gotcha when using the RCDS, and that is when your job runs, the files in the unzipped tarball are actually placed in your work area as symlinks from the CVMFS version of the file (which is what you want since the whole point is not to have N different copies of everything). +The catch is that if your job script expected to be able to edit one or more of those files within the job, it won't work because the link is to a read-only area. +Fortunately there's a very simple trick you can do in your script before trying to edit any such files: + +~~~ +cp ${INPUT_TAR_DIR_LOCAL}/file_I_want_to_edit mytmpfile # do a cp, not mv +rm ${INPUT_TAR_DIR_LOCAL}file_I_want_to_edit # This really just removes the link +mv mytmpfile file_I_want_to_edit # now it's available as an editable regular file. +~~~ + +You certainly don't want to do this for every file, but for a handful of small text files this is perfectly acceptable and the overall benefits of copying in code via the RCDS far outweigh this small cost. +This can get a little complicated when trying to do it for things several directories down, so it's easiest to have such files in the top level of your tar file. + + + + + +## Monitor your jobs +For all links below, log in with your FNAL Services credentials (FNAL email, not Kerberos password). + +* What DUNE is doing overall: +[https://fifemon.fnal.gov/monitor/d/000000053/experiment-batch-details?orgId=1&var-experiment=dune](https://fifemon.fnal.gov/monitor/d/000000053/experiment-batch-details?orgId=1&var-experiment=dune) + + +* What's going on with only your jobs: +Remember to change the url with your own username and adjust the time range to cover the region of interest. +[https://fifemon.fnal.gov/monitor/d/000000116/user-batch-details?orgId=1&var-cluster=fifebatch&var-user=kherner](https://fifemon.fnal.gov/monitor/d/000000116/user-batch-details?orgId=1&var-cluster=fifebatch&var-user=kherner) + +* Why your jobs are held: +Remember to choose your username in the upper left. +[https://fifemon.fnal.gov/monitor/d/000000146/why-are-my-jobs-held?orgId=1](https://fifemon.fnal.gov/monitor/d/000000146/why-are-my-jobs-held?orgId=1) + +## View the stdout/stderr of our jobs +Here's the link for the history page of the example job: [link](https://fifemon.fnal.gov/monitor/d/000000115/job-cluster-summary?orgId=1&var-cluster=40351757&var-schedd=jobsub01.fnal.gov&from=1611098894726&to=1611271694726). + +Feel free to sub in the link for your own jobs. + +Once there, click "View Sandbox files (job logs)". +In general you want the .out and .err files for stdout and stderr. +The .cmd file can sometimes be useful to see exactly what got passed in to your job. + +[Kibana][kibana] can also provide a lot of information. + +You can also download the job logs from the command line with jobsub_fetchlog: + +```bash +jobsub_fetchlog --jobid=12345678.0@jobsub0N.fnal.gov --unzipdir=some_appropriately_named_directory +``` + +That will download them as a tarball and unzip it into the directory specified by the --unzipdir option. +Of course replace 12345678.0@jobsub0N.fnal.gov with your own job ID. + +> ## Quiz +> +> Download the log of your last submission via jobsub_fetchlog or look it up on the monitoring pages. Then answer the following questions (all should be available in the .out or .err files): +> 1. On what site did your job run? +> 2. How much memory did it use? +> 3. Did it exit abnormally? If so, what was the exit code? +> +{: .solution} + +## Review of best practices in grid jobs (and a bit on the interactive machines) + +* When creating a new workflow or making changes to an existing one, **ALWAYS test with a single job first**. Then go up to 10, etc. Don't submit thousands of jobs immediately and expect things to work. +* **ALWAYS** be sure to prestage your input datasets before launching large sets of jobs. This may become less necesaary in the future as we move to distributed storage locations. +* **Use RCDS**; do not copy tarballs from places like scratch dCache. There's a finite amount of transfer bandwidth available from each dCache pool. If you absolutely cannot use RCDS for a given file, it's better to put it in resilient (but be sure to remove it when you're done!). The same goes for copying files from within your own job script: if you have a large number of jobs looking for a same file, get it from resilient. Remove the copy when no longer needed. Files in resilient dCache that go unaccessed for 45 days are automatically removed. +* Be careful about placing your output files. **NEVER place more than a few thousand files into any one directory inside dCache. That goes for all type of dCache (scratch, persistent, resilient, etc). Subdirectories also count against the total for these purposes, so don't put too many subdirectories at any one level. +* **AVOID** commands like `ifdh ls /some/path` inside grid jobs unless it is absolutely necessary. That is an expensive operation and can cause a lot of pain for many users, especially when a directory has large number of files in it. Remote listings take much, much longer than the corresponding op on a machine where the directory is mounted via NFS. If you just need to verify a directory exists, there are much better ways than ifdh ls, for example the gfal-stat command. Note also that ifdh cp will now, by default, create an output directory if it does not exist (so be careful that you've specified your output string properly). +* Use xrootd when opening files interactively; this is much more stable than simply doing `root /pnfs/dune/... (and in general, do NOT do that...)` +* **NEVER** copy job outputs to a directory in resilient dCache. Remember that they are replicated by a factor of 20! **Any such files are subject to deletion without warning**. +* **NEVER** do hadd on files in `/pnfs` areas unless you're using `xrootd`. I.e. do NOT do hadd out.root `/pnfs/dune/file1 /pnfs/dune/file2 ...` This can cause severe performance degradations. +* Generally aim for output file sizes of 1 GB or greater. dCache is really not a fan of small files. You may need to process multiple input files to get to that size (and we generally encourage that anyway!) +* Very short jobs (measured in minutes) are quite inefficient, especially if you have an input tarball. In general you want to run for at least a few hours, and 8-12 is probably ideal (and of course longer jobs are allowed). Again you may need to process multiple input files, depending on what you're doing, or run multiple workflow stages in the same job. See the POMS section of the tutorial for more details. + +**Side note:** Some people will pass file lists to their jobs instead of using a SAM dataset. We do not recommend that for two reasons: 1) Lists do not protect you from cases where files fall out of cache at the location(s) in your list. When that happens your jobs sit idle waiting for the files to be fetched from tape, which kills your efficiency and blocks resources for others. 2) You miss out on cases where there might be a local copy of the file at the site you're running on, or at least at closer one to your list. So you may end up unecessarily streaming across oceans, whereas using SAM (or later Rucio) will find you closer, local copies when they exist. + +**Another important side note:** If you are used to using other programs for your work such as project.py (which is **NOT** officially supported by DUNE or the Fermilab Scientific Computing Division), there is a helpful tool called [Project-py][project-py-guide] that you can use to convert existing xml into POMS configs, so you don't need to start from scratch! Then you can just switch to using POMS from that point forward. As a reminder, if you use unsupported tools, you are own your own and will receive NO SUPPORT WHATSOEVER. You are still responsible for making sure that your jobs satisfy Fermilab's policy for job efficiency: https://cd-docdb.fnal.gov/cgi-bin/sso/RetrieveFile?docid=7045&filename=FIFE_User_activity_mitigation_policy_20200625.pdf&version=1 + +## The cost of getting it wrong: a cautionary tale + +Earlier in May 2023 there was a fairly significant disruption to FNAL dCache, which resulted in at least five different tickets across four different experiments complaining of poor performance (resulting in jobs going held of exceeding time), timeouts, or other storage-related failures. It's unclear exactly how many jobs were affcted but it was likely in the many thousands. The root cause was a DUNE user running `ifdh ls $OUTDIR 0` to check the existence of a given directory. That command, though it only spits out the directory name, was indeed doing a full internal listing of the contents of $OUTDIR. Normally that's not the end of the world (see the comment in the best practices section), but this directory had over 100,000 files in it! The user was writing all job outputs into the same directory from what we could tell. + +Since the workflow was causing a systemwide disruption we immediately held all of the user's jobs and blocked new submissions until the workflow was re-engineered. Fortunately dCache performance recovered very quickly after that. The user's jobs are running again and they are also much more CPU-efficient than they were before the changes. + +*The bottom line: one single user not following best practices can disrupt the entire system if they get enough jobs running.* EVERYONE is responsible for following best practices. Getting it wrong affects not only you, but your collaborators! + +## A word on the DUNE Global Pool + +DUNE has also created a a global glideinWMS pool similar to the CMS Global Pool that is intended to serve as a single point through which multiple job submission systems (e.g. HTCondor schedulers at sites outside of Fermilab) can have access to the same resources. Jobs using the global pool still run in the exactly the same way as those that don't. We plan to move more and more work over to the global pool in 2023 and priority access to the FermiGrid quota will eventually be given to jobs submitted to the global pool. To switch to the global pool with jobsub, it's simply a matter of adding `--global-pool dune` as an option to your submission command. The only practical difference is that your jobs will come back with IDs of the form NNNNNNN.N@dunegpschedd0X.fnal.gov instead of NNNNNNN.N@jobsub0X.fnal.gov. Again, everything else is identical, so feel free to test it out. + + +## Making subsets of metacat datasets + +Running across very large number of files puts you at risk of system issues. It is often much nicer to run over several smaller subsets. +Many official metacat definitions are large data collections defined only by their properties and not really suitable for a single job. + +You can do the following. Submit your jobs using the skip and limit commands. Here 'namespace:official_dataset' describes the official dataset. + +See [the basics tutorial](https://dune.github.io/computing-basics/03-data-management/index.html#official-datasets-) for information on official datasets. + +~~~ +query="files from namespace:official_dataset skip 0 limit 1000" +query="files from namespace:official_dataset skip 1000 limit 1000" +query="files from namespace:official_dataset skip 2000 limit 1000" +.... +~~~ +{: ..language-bash} + + + + + + +## Verify Your Learning: + +> ## Question 01 +> +> What are the differences in environment between Fermilab worker nodes and those at other sites (assuming the site supports Singularity)? +>
    +>
  1. Fermilab workers have additional libraries available.
    2. +>
  2. Worker nodes at other sites have additional libraries installed.
    3. +>
  3. No difference.
    4. +>
+> +> > ## Answer +> > The correct answer is C - No difference. +> > {: .output} +> > Comment: +> {: .solution} +{: .challenge} + +> ## Question 02 +> +> After setting up a new workflow or preparing to run one that has not been exercised in a while, what is an that has not been exercised in a while, what is an appropriate number of test jobs to initially submit? +>
    +>
  1. 1.
    2. +>
  2. 10.
    3. +>
  3. 100.
    4. +>
  4. As many as needed.
    5. +>
+> +> > ## Answer +> > The correct answer is A - 1. +> > {: .output} +> > Comment: +> {: .solution} +{: .challenge} + + +> ## Question 03 +> +> project.py is supported by the Fermilab Scientific Computing Division +>
    +>
  1. True.
    2. +>
  2. False.
    3. +>
+> +> > ## Answer +> > The correct answer is B - False. +> > {: .output} +> > Comment: +> {: .solution} +{: .challenge} + + +> ## Question 04 +> +> What is generally the best way to read in a .root file for analysis within a grid job? +>
    +>
  1. Open with an xrootd URI (root://).
    2. +>
  2. Copy the entire file at the beginning of the job.
    3. +>
  3. Both A and B.
    4. +>
+> +> > ## Answer +> > The correct answer is A - Open with an xrootd URI (root://). +> > {: .output} +> > Comment: +> {: .solution} +{: .challenge} + + +> ## Question 05 +> +> What is the best way to specify your desired operating system and environment in a grid job? +>
    +>
  1. Use the --OS option in jobsub.
    2. +>
  2. Do not specify any OS, but control it with the SingularityImage classad.
    3. +>
  3. Don’t specify anything. The grid does it
    4. +>
  4. None of the Above
    5. +>
+> +> > ## Answer +> > The correct answer is B - Do not specify any OS, but control it with the SingularityImage classad. +> > {: .output} +> > Comment: +> {: .solution} +{: .challenge} + + +> ## Question 06 +> +> What is the best way to copy custom code into a grid job? +>
    +>
  1. Use the RCDS (i.e. --tar_file_name=dropbox://foo/bar/) and stage the file in via CVMFS.
    2. +>
  2. Copy a tarball to /pnfs/dune/scratch/users/username.
    3. +>
  3. Copy a tarball to /pnfs/dune/persistent/users/username.
    4. +>
  4. Copy a tarball to /pnfs/dune/resilient/users/username.
    5. +>
  5. None of the Above
    6. +>
+> +> > ## Answer +> > The correct answer is A - Use the RCDS (i.e. --tar_file_name=dropbox://foo/bar/) and stage the file in via CVMFS. +> > {: .output} +> > Comment: +> {: .solution} +{: .challenge} + + + +## Further Reading +Some more background material on these topics (including some examples of why certain things are bad) is in these links: + + +[December 2022 jobsub_lite demo and information session](https://indico.fnal.gov/event/57514/) + +[January 2023 additional experiment feedback session on jobsub_lite]( ) + +[Wiki page listing differences between jobsub_lite and legacy jobsub](https://fifewiki.fnal.gov/wiki/Differences_between_jobsub_lite_and_legacy_jobsub_client/server) + +[2021 Intensity Frontier Summer School](https://indico.fnal.gov/event/49414) + +[The Glidein-based Workflow Management System]( https://glideinwms.fnal.gov/doc.prd/index.html ) + +[Introduction to Docker](https://hsf-training.github.io/hsf-training-docker/index.html) + +[job-autorelease]: https://cdcvs.fnal.gov/redmine/projects/fife/wiki/Job_autorelease + +[redmine-wiki-jobsub]: https://cdcvs.fnal.gov/redmine/projects/jobsub/wiki + +[redmine-wiki-using-the-client]: https://cdcvs.fnal.gov/redmine/projects/jobsub/wiki/Using_the_Client + +[fifemon-dune]: https://fifemon.fnal.gov/monitor/d/000000053/experiment-batch-details?orgId=1&var-experiment=dune + +[fifemon-userjobs]: https://fifemon.fnal.gov/monitor/d/000000116/user-batch-details?orgId=1&var-cluster=fifebatch&var-user=kherner + +[fifemon-whyheld]: https://fifemon.fnal.gov/monitor/d/000000146/why-are-my-jobs-held?orgId=1 + +[kibana]: https://fifemon.fnal.gov/kibana/goto/8f432d2e4a40cbf81d3072d9c9d688a6 + +[poms-page-ana]: https://pomsgpvm01.fnal.gov/poms/index/dune/analysis/ + +[poms-user-doc]: https://cdcvs.fnal.gov/redmine/projects/prod_mgmt_db/wiki/POMS_User_Documentation + +[fife-launch-ref]: https://cdcvs.fnal.gov/redmine/projects/fife_utils/wiki/Fife_launch_Reference + +[poms-campaign-stage-info]: https://pomsgpvm01.fnal.gov/poms/campaign_stage_info/dune/analysis?campaign_stage_id=9023 + +[project-py-guide]: https://cdcvs.fnal.gov/redmine/projects/project-py/wiki/Project-py_guide + +[DUNE_computing_tutorial_advanced_topics_20200129]: https://indico.fnal.gov/event/20144/contributions/55932/attachments/34945/42690/DUNE_computing_tutorial_advanced_topics_and_best_practices_20200129.pdf + + +{%include links.md%} diff --git a/_extras/short_submission.md b/_extras/short_submission.md new file mode 100644 index 0000000..fad089f --- /dev/null +++ b/_extras/short_submission.md @@ -0,0 +1,96 @@ +--- +title: Short Submission +--- + +## this collects the sequence of steps for a batch submission with local code + +### in your top level mrb directory + +For example `/exp/dune/app/users/$USER/myworkarea` + +need to have a name for it as you will be making a tarball + +~~~ +export DIRECTORY=myworkarea +~~~ + +copy these scripts into that top level directory + +- setup-grid + +~~~ +{% include setup-grid %} +~~~ + +- setup-local + +~~~ +{% include setup-local %} +~~~ + +- makercds + +~~~ +{% include makercds.sh %} +~~~ + +- submit_workflow.sh + +~~~ +{% include submit_workflow.sh %} +~~~ + + +- submit_local_code.jobscript.sh + +~~~ +{% include submit_local_code.jobscript.sh %} +~~~ + +### modify this script (should not need to change the others) + + +- choose your code version and fcl file +- make certain the fcl file is either in the fcl path or in `$DIRECTORY` +- add a string `PROCESS_TYPE` that will go in your filename + + +#### setup_before_submit.sh + +~~~ +{% include setup_before_submit.sh %} +~~~ + +Then run it to set things up + +~~~ +source setup_before_submit.sh +~~~ + +### from DIRECTORY make a tarball and put in rcds + +If you have changed any scripts or code, you must redo this. + +~~~ +./makercds.sh $DIRECTORY +~~~ + +will take a while, produce a tarball on /exp/dune/data and put the cvmfs location in cvmfs.location + +### Submit the job + +~~~ +./submit_workflow.sh +~~~ + +should get a workflow number back + +go to [justin](https://dunejustin.fnal.gov/dashboard/?method=list-workflows) + +to track your job. + + + + + + diff --git a/_includes/makercds.sh b/_includes/makercds.sh new file mode 100644 index 0000000..36e5704 --- /dev/null +++ b/_includes/makercds.sh @@ -0,0 +1,10 @@ +# give me the directory name as argument +export HERE=`pwd` +# put the tar file on a bigger disk +export THERE=/exp/dune/data/users/$USER/ +cd .. # go up one +tar --exclude '.git' --exclude build_slf7.x86_64 -czf $THERE/$1.tar.gz $1 +export INPUT_TAR_DIR_LOCAL=`justin-cvmfs-upload $THERE/$1.tar.gz` +echo $INPUT_TAR_DIR_LOCAL +echo $INPUT_TAR_DIR_LOCAL > $1/cvmfs.location +cd $HERE \ No newline at end of file diff --git a/_includes/setup-grid b/_includes/setup-grid new file mode 100644 index 0000000..264d7ea --- /dev/null +++ b/_includes/setup-grid @@ -0,0 +1,129 @@ +# No magic #!, this script must be sourced! +# this script is part of mrb and gets renamed to setup when copied to the localProducts area + +# NOTICE: this script is not relocatable + +# +# Begin boilerplate. +# + +# Note: All the following special tricks for $_ must continue +# relaying the value to the next rule. Be careful! +# Special trick to nail the value of $_ down in a variety of shells. +echo $_ >& /dev/null +# Special trick for tcsh which is one-off on the command history stack. +: $_ +# Special trick to capture the value of $_ in zsh and bash +test $?shell$_ != 1$_ >& /dev/null && \ + dollar_underscore="$_" && \ + dollar_underscore=`expr "${dollar_underscore}" : ".\(.*\)"` +# Special trick to capture the value of $_ in tcsh +test $?shell = 1 && set dollar_underscore=`echo $_` + +# need to be able to check for mrb +test $?shell = 1 && set ss="csh" || ss="sh" +test "$ss" = "csh" && alias return exit + +test "$ss" = "csh" && \ + alias tnotnull "eval '"'test $?'"\!* -eq 1' && eval '"'test -n "$'"\!*"'"'"'" +test "$ss" = "sh" && \ + eval 'tnotnull() { eval "test -n \"\${$1-}\"" ;}' + +# check for mrb +tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS!" ; echo "ERROR:" ) +tnotnull UPS_DIR || unset ss +tnotnull UPS_DIR || return 1 +tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first setup mrb!"; echo "ERROR:" ) +tnotnull MRB_DIR || unset ss +tnotnull MRB_DIR || return 1 +test -f "$MRB_DIR/libexec/shell_independence" || \ + ( echo "ERROR:" ; echo "ERROR: this mrb area expects mrb >= v5_00_00 (found $MRB_VERSION)!" ; echo "ERROR:" ) +test -f "$MRB_DIR/libexec/shell_independence" || unset ss +test -f "$MRB_DIR/libexec/shell_independence" || return 1 + +# Get the shell independence aliases and functions. +source "$MRB_DIR/libexec/shell_independence" + +# Capture the value of $0 +set_ dollar_zed=`echo "${0}" | sed -e 's/^-//'` + +# Special tricks to figure out if this script has been sourced. +# Works for bash, tcsh, and in some cases for zsh. +set_ is_sourced=false +ifcsh_ + # Note: It is unfortunate that we must hard-code the name + # of this script here, but there is no other way + # that works, tcsh is brain-dead. + set base=`basename "${dollar_zed}"` + test "${base}" != "setup" && \ + set is_sourced=true +else + # Special trick for zsh. + test "${ZSH_NAME}" && test "${dollar_underscore}" = "${dollar_zed}" && \ + is_sourced=true + # If there were arguments then there is no safe way to find out + # whether or not the script was sourced in zsh. Pretend it was. + test "${ZSH_NAME}" && test "${#argv}" != "0" && \ + is_sourced=true + # Special trick for bash. + test "${BASH}" && test "${BASH_SOURCE}" != "${dollar_zed}" && \ + is_sourced=true +# Warning, this must be here because the tcsh parser is brain-dead. +endif +endifcsh_ + +# +# End of boilerplate. Begin of real work. +# + +tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS" ; echo "ERROR:" ) +tnotnull UPS_DIR || source "$MRB_DIR/libexec/unset_shell_independence" +tnotnull UPS_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav +tnotnull UPS_DIR || return 1 + + +tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first set up mrb!"; echo "ERROR:" ) +tnotnull MRB_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav +tnotnull MRB_DIR || return 1 + +setenv MRB_PROJECT "larsoft" +setenv MRB_PROJECT_VERSION "v09_91_02d01" +setenv MRB_QUALS "e26:prof" +setenv MRB_QUALSTRING = "e26_prof" +setenv MRB_TOP "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}" +setenv MRB_TOP_BUILD "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}" +setenv MRB_SOURCE "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/srcs" +setenv MRB_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_v09_91_02d01_e26_prof" +setenv PRODUCTS "${MRB_INSTALL}:${PRODUCTS}" +setenv CETPKG_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_v09_91_02d01_e26_prof" + +#--- begin middle boilerplate + +set_ flav=`get-directory-name subdir` +set_ buildDirName="build_${flav}" + +test "$ss" = sh && test -n "${MRB_BUILDDIR}" && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" +test "$ss" = csh && tnotnull MRB_BUILDDIR && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" +setenv MRB_BUILDDIR ${MRB_TOP_BUILD}/${buildDirName} + +unset me dollar_underscore dollar_zed is_sourced base msg1 flav + +#--- end middle boilerplate +# report the environment +echo +echo MRB_PROJECT=$MRB_PROJECT +echo MRB_PROJECT_VERSION=$MRB_PROJECT_VERSION +echo MRB_QUALS=$MRB_QUALS +echo MRB_TOP=$MRB_TOP +echo MRB_SOURCE=$MRB_SOURCE +echo MRB_BUILDDIR=$MRB_BUILDDIR +echo MRB_INSTALL=$MRB_INSTALL +echo MRB_DIR=$MRB_DIR +echo +echo PRODUCTS=$PRODUCTS +echo CETPKG_INSTALL=$CETPKG_INSTALL +echo + +source "$MRB_DIR/libexec/unset_shell_independence" +unset db buildDirName + diff --git a/_includes/setup-local b/_includes/setup-local new file mode 100644 index 0000000..205ff92 --- /dev/null +++ b/_includes/setup-local @@ -0,0 +1,129 @@ +# No magic #!, this script must be sourced! +# this script is part of mrb and gets renamed to setup when copied to the localProducts area + +# NOTICE: this script is not relocatable + +# +# Begin boilerplate. +# + +# Note: All the following special tricks for $_ must continue +# relaying the value to the next rule. Be careful! +# Special trick to nail the value of $_ down in a variety of shells. +echo $_ >& /dev/null +# Special trick for tcsh which is one-off on the command history stack. +: $_ +# Special trick to capture the value of $_ in zsh and bash +test $?shell$_ != 1$_ >& /dev/null && \ + dollar_underscore="$_" && \ + dollar_underscore=`expr "${dollar_underscore}" : ".\(.*\)"` +# Special trick to capture the value of $_ in tcsh +test $?shell = 1 && set dollar_underscore=`echo $_` + +# need to be able to check for mrb +test $?shell = 1 && set ss="csh" || ss="sh" +test "$ss" = "csh" && alias return exit + +test "$ss" = "csh" && \ + alias tnotnull "eval '"'test $?'"\!* -eq 1' && eval '"'test -n "$'"\!*"'"'"'" +test "$ss" = "sh" && \ + eval 'tnotnull() { eval "test -n \"\${$1-}\"" ;}' + +# check for mrb +tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS!" ; echo "ERROR:" ) +tnotnull UPS_DIR || unset ss +tnotnull UPS_DIR || return 1 +tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first setup mrb!"; echo "ERROR:" ) +tnotnull MRB_DIR || unset ss +tnotnull MRB_DIR || return 1 +test -f "$MRB_DIR/libexec/shell_independence" || \ + ( echo "ERROR:" ; echo "ERROR: this mrb area expects mrb >= v5_00_00 (found $MRB_VERSION)!" ; echo "ERROR:" ) +test -f "$MRB_DIR/libexec/shell_independence" || unset ss +test -f "$MRB_DIR/libexec/shell_independence" || return 1 + +# Get the shell independence aliases and functions. +source "$MRB_DIR/libexec/shell_independence" + +# Capture the value of $0 +set_ dollar_zed=`echo "${0}" | sed -e 's/^-//'` + +# Special tricks to figure out if this script has been sourced. +# Works for bash, tcsh, and in some cases for zsh. +set_ is_sourced=false +ifcsh_ + # Note: It is unfortunate that we must hard-code the name + # of this script here, but there is no other way + # that works, tcsh is brain-dead. + set base=`basename "${dollar_zed}"` + test "${base}" != "setup" && \ + set is_sourced=true +else + # Special trick for zsh. + test "${ZSH_NAME}" && test "${dollar_underscore}" = "${dollar_zed}" && \ + is_sourced=true + # If there were arguments then there is no safe way to find out + # whether or not the script was sourced in zsh. Pretend it was. + test "${ZSH_NAME}" && test "${#argv}" != "0" && \ + is_sourced=true + # Special trick for bash. + test "${BASH}" && test "${BASH_SOURCE}" != "${dollar_zed}" && \ + is_sourced=true +# Warning, this must be here because the tcsh parser is brain-dead. +endif +endifcsh_ + +# +# End of boilerplate. Begin of real work. +# + +tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS" ; echo "ERROR:" ) +tnotnull UPS_DIR || source "$MRB_DIR/libexec/unset_shell_independence" +tnotnull UPS_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav +tnotnull UPS_DIR || return 1 + + +tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first set up mrb!"; echo "ERROR:" ) +tnotnull MRB_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav +tnotnull MRB_DIR || return 1 + +setenv MRB_PROJECT "larsoft" +setenv MRB_PROJECT_VERSION "v09_91_02d01" +setenv MRB_QUALS "e26:prof" +setenv MRB_TOP "${DIRECTORY}" +setenv MRB_TOP_BUILD "${DIRECTORY}" +setenv MRB_SOURCE "${DIRECTORY}/srcs" +setenv MRB_INSTALL "${localProductsdir}" +setenv PRODUCTS "${MRB_INSTALL}:${PRODUCTS}" +setenv CETPKG_INSTALL "${localProductsdir}" + +echo "MRB_INSTALL is ${MRB_INSTALL}" +#--- begin middle boilerplate + +set_ flav=`get-directory-name subdir` +set_ buildDirName="build_${flav}" + +test "$ss" = sh && test -n "${MRB_BUILDDIR}" && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" +test "$ss" = csh && tnotnull MRB_BUILDDIR && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" +setenv MRB_BUILDDIR ${MRB_TOP_BUILD}/${buildDirName} + +unset me dollar_underscore dollar_zed is_sourced base msg1 flav + +#--- end middle boilerplate +# report the environment +echo +echo MRB_PROJECT=$MRB_PROJECT +echo MRB_PROJECT_VERSION=$MRB_PROJECT_VERSION +echo MRB_QUALS=$MRB_QUALS +echo MRB_TOP=$MRB_TOP +echo MRB_SOURCE=$MRB_SOURCE +echo MRB_BUILDDIR=$MRB_BUILDDIR +echo MRB_INSTALL=$MRB_INSTALL +echo +echo PRODUCTS=$PRODUCTS +echo CETPKG_INSTALL=$CETPKG_INSTALL + +echo " got here" + +source "${MRB_DIR}/libexec/unset_shell_independence" +unset db buildDirName + diff --git a/_includes/setup_before_submit.sh b/_includes/setup_before_submit.sh new file mode 100644 index 0000000..24412cc --- /dev/null +++ b/_includes/setup_before_submit.sh @@ -0,0 +1,25 @@ +# setup_local_test.sh for example +export DIRECTORY="$(basename "${PWD}")" +export DUNE_VERSION=v09_91_02d01 +export DUNE_QUALIFIER=e26:prof +export FCL_FILE=run_analyseEvents.fcl +export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 2 ordered " +export PROCESS_TYPE=analyze +export USERF=$USER +export NUM_EVENTS=1 +export FNALURL='https://fndcadoor.fnal.gov:2880/dune/scratch/users' + +## set up locally just as a check +source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh +setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +setup justin +justin time +justin get-token + +# make certain the setup script in localProducts is what you want +export localProductsdir=`ls -c1d $PWD/localProducts*` +cp setup-grid $localProductsdir/setup + + + diff --git a/_includes/setup_local_test.sh b/_includes/setup_local_test.sh new file mode 100644 index 0000000..b2b648f --- /dev/null +++ b/_includes/setup_local_test.sh @@ -0,0 +1,23 @@ +# setup_local_code.txt for example +export DUNE_VERSION=v09_91_02d01 +export DUNE_QUALIFIER=e26:prof +export FCL_FILE=run_analyseEvents.fcl +export INPUT_TAR_DIR_LOCAL=$PWD +export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 1 ordered " +export PROCESS_TYPE=analyze +export USERF=$USER + +# try doing a local setup for testing +echo "I am here now: `pwd`" +export localProductsdir=`ls -c1d $PWD/localProducts*` +echo "Local products directory is ${localProductsdir}" +source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh +export PRODUCTS="${localProductsdir}/:$PRODUCTS" +mv ${localProductsdir}/setup ${localProductsdir}/setup.bak +cp setup-local ${localProductsdir}/setup +# Then we can set up our local products +setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +mrbslp +source ${localProductsdir}/setup + diff --git a/_includes/submit_local_code.jobscript.sh b/_includes/submit_local_code.jobscript.sh new file mode 100644 index 0000000..db979ac --- /dev/null +++ b/_includes/submit_local_code.jobscript.sh @@ -0,0 +1,118 @@ +#!/bin/bash +:<<'EOF' + +To use this jobscript to process 5 files from the dataset fardet-hd__fd_mc_2023a_reco2__full-reconstructed__v09_81_00d02__standard_reco2_dune10kt_nu_1x2x6__prodgenie_nu_dune10kt_1x2x6__out1__validation +data and put the output logs in the `usertests` namespace and saves the output in /scratch + +Use these commands to set up ahead of time: + +export DUNE_VERSION= +export DUNE_QUALIFIER= +export FCL_FILE= +export INPUT_TAR_DIR_LOCAL= +export MQL= +export DIRECTORY= + +Use this command to create the workflow: + +justin simple-workflow \ +--mql "$MQL" \ +--jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ + --output-pattern "*.root:${FNALURL}/${USERF}" --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 + + +The following optional environment variables can be set when creating the +workflow/stage: FCL_FILE, PROCESS_TYPE, NUM_EVENTS, DUNE_VERSION, DUNE_QUALIFIER + +EOF + +# fcl file and DUNE software version/qualifier to be used +FCL_FILE=${FCL_FILE:-${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/my_code/fcls/my_reco.fcl} +PROCESS_TYPE=${PROCESS_TYPE:-reco2} +DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} +DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} + +cd ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY} + +echo "Current working directory is `pwd`" + + +# number of events to process from the input file +if [ "$NUM_EVENTS" != "" ] ; then + events_option="-n $NUM_EVENTS" +fi + +# First get an unprocessed file from this stage +did_pfn_rse=`$JUSTIN_PATH/justin-get-file` + +if [ "$did_pfn_rse" = "" ] ; then + echo "Nothing to process - exit jobscript" + exit 0 +fi + +# Keep a record of all input DIDs, for pdjson2meta file -> DID mapping +echo "$did_pfn_rse" | cut -f1 -d' ' >>all-input-dids.txt + +# pfn is also needed when creating justin-processed-pfns.txt +pfn=`echo $did_pfn_rse | cut -f2 -d' '` +echo "Input PFN = $pfn" + +echo "TARDIR ${INPUT_TAR_DIR_LOCAL}" +echo "CODE DIR ${DIRECTORY}" + +# Setup DUNE environment +localProductsdir=`ls -c1d ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts*` + +echo "localProductsdir ${localProductsdir}" + + +# seems to require the right name for the setup script + +echo " check that there is a setup in ${localProductsdir}" +ls -lrt ${localProductsdir}/setup +source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh +export PRODUCTS="${localProductsdir}/:$PRODUCTS" + +# Then we can set up our local products +setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +source ${localProductsdir}/setup +mrbslp + +# Construct outFile from input $pfn +now=$(date -u +"%Y-%m-%dT_%H%M%SZ") +Ffname=`echo $pfn | awk -F/ '{print $NF}'` +fname=`echo $Ffname | awk -F. '{print $1}'` +outFile=${fname}_${PROCESS_TYPE}_${now}.root + +campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" + +# Here is where the LArSoft command is call it +( +# Do the scary preload stuff in a subshell! +export LD_PRELOAD=${XROOTD_LIB}/libXrdPosixPreload.so +echo "$LD_PRELOAD" + +lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o $outFile "$pfn" > ${fname}_${PROCESS_TYPE}_${now}.log 2>&1 +) + +echo '=== Start last 1000 lines of lar log file ===' +tail -1000 ${fname}_ana_${now}.log +echo '=== End last 1000 lines of lar log file ===' + +# Subshell exits with exit code of last command +larExit=$? +echo "lar exit code $larExit" + +if [ $larExit -eq 0 ] ; then + # Success ! + echo "$pfn" > justin-processed-pfns.txt + jobscriptExit=0 +else + # Oh ! + jobscriptExit=1 +fi + +# Create compressed tar file with all log files +tar zcf `echo "$JUSTIN_JOBSUB_ID.logs.tgz" | sed 's/@/_/g'` *.log +exit $jobscriptExit diff --git a/_includes/submit_workflow.sh b/_includes/submit_workflow.sh new file mode 100644 index 0000000..09a087e --- /dev/null +++ b/_includes/submit_workflow.sh @@ -0,0 +1,7 @@ +# actual submission +export INPUT_TAR_DIR_LOCAL=`cat cvmfs.location` +echo "tardir $INPUT_TAR_DIR_LOCAL" +justin simple-workflow \ +--mql "$MQL" \ +--jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ + --output-pattern "*.root:${FNALURL}/${USERF}" --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} \ No newline at end of file From 9559ece0d6d8ebc72cda34ddfedf1b5c36b8b0ee Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 31 Jan 2026 09:34:54 +0100 Subject: [PATCH 02/27] more fun changes --- _includes/makercds.sh | 3 +++ _includes/setup_before_submit.sh | 1 + _includes/submit_local_code.jobscript.sh | 24 ++++++++++++++++++++---- _includes/submit_workflow.sh | 15 ++++++++++++++- 4 files changed, 38 insertions(+), 5 deletions(-) diff --git a/_includes/makercds.sh b/_includes/makercds.sh index 36e5704..9d0ef09 100644 --- a/_includes/makercds.sh +++ b/_includes/makercds.sh @@ -3,7 +3,10 @@ export HERE=`pwd` # put the tar file on a bigger disk export THERE=/exp/dune/data/users/$USER/ cd .. # go up one +echo " make tar file" tar --exclude '.git' --exclude build_slf7.x86_64 -czf $THERE/$1.tar.gz $1 +ls -lrt $THERE/$1.tar.gz +echo " upload tar file to cvmfs" export INPUT_TAR_DIR_LOCAL=`justin-cvmfs-upload $THERE/$1.tar.gz` echo $INPUT_TAR_DIR_LOCAL echo $INPUT_TAR_DIR_LOCAL > $1/cvmfs.location diff --git a/_includes/setup_before_submit.sh b/_includes/setup_before_submit.sh index 24412cc..b2b2f92 100644 --- a/_includes/setup_before_submit.sh +++ b/_includes/setup_before_submit.sh @@ -5,6 +5,7 @@ export DUNE_QUALIFIER=e26:prof export FCL_FILE=run_analyseEvents.fcl export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 2 ordered " export PROCESS_TYPE=analyze +export DESCRIPTION="$PROCESS_TYPE using $FCL_FILE" export USERF=$USER export NUM_EVENTS=1 export FNALURL='https://fndcadoor.fnal.gov:2880/dune/scratch/users' diff --git a/_includes/submit_local_code.jobscript.sh b/_includes/submit_local_code.jobscript.sh index db979ac..c3c3642 100644 --- a/_includes/submit_local_code.jobscript.sh +++ b/_includes/submit_local_code.jobscript.sh @@ -29,10 +29,21 @@ EOF # fcl file and DUNE software version/qualifier to be used FCL_FILE=${FCL_FILE:-${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/my_code/fcls/my_reco.fcl} PROCESS_TYPE=${PROCESS_TYPE:-reco2} -DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} -DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} +#DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} +#DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} + +echo "Check environment" +echo "DIRECTORY=$DIRECTORY" +echo "DUNE_VERSION=$DUNE_VERSION" +echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" +echo "FCL_FILE=$FCL_FILE" +echo "MQL=$MQL" +echo "PROCESS_TYPE=$PROCESS_TYPE" +echo "USERF=$USERF" +echo "NUM_EVENTS=$NUM_EVENTS" +echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" + -cd ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY} echo "Current working directory is `pwd`" @@ -70,6 +81,7 @@ echo "localProductsdir ${localProductsdir}" echo " check that there is a setup in ${localProductsdir}" ls -lrt ${localProductsdir}/setup +ls -lrt ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh export PRODUCTS="${localProductsdir}/:$PRODUCTS" @@ -93,11 +105,15 @@ campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" export LD_PRELOAD=${XROOTD_LIB}/libXrdPosixPreload.so echo "$LD_PRELOAD" +echo "now run lar" + lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o $outFile "$pfn" > ${fname}_${PROCESS_TYPE}_${now}.log 2>&1 ) + + echo '=== Start last 1000 lines of lar log file ===' -tail -1000 ${fname}_ana_${now}.log +tail -1000 ${fname}_${PROCESS_TYPE}_${now}.log echo '=== End last 1000 lines of lar log file ===' # Subshell exits with exit code of last command diff --git a/_includes/submit_workflow.sh b/_includes/submit_workflow.sh index 09a087e..677d91e 100644 --- a/_includes/submit_workflow.sh +++ b/_includes/submit_workflow.sh @@ -1,7 +1,20 @@ # actual submission export INPUT_TAR_DIR_LOCAL=`cat cvmfs.location` + +echo "DIRECTORY=$DIRECTORY" +echo "DUNE_VERSION=$DUNE_VERSION" +echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" +echo "FCL_FILE=$FCL_FILE" +echo "MQL=$MQL" +echo "PROCESS_TYPE=$PROCESS_TYPE" +echo "USERF=$USERF" +echo "NUM_EVENTS=$NUM_EVENTS" +echo "DESCRIPTION=$DESCRIPTION" +echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" + + echo "tardir $INPUT_TAR_DIR_LOCAL" justin simple-workflow \ --mql "$MQL" \ --jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${FNALURL}/${USERF}" --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} \ No newline at end of file + --output-pattern "*.root:${FNALURL}/${USERF}" --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --description "${DESCRIPTION}" \ No newline at end of file From 1d77778b4120c28541599fe58131289bfb993b14 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 31 Jan 2026 09:38:01 +0100 Subject: [PATCH 03/27] more fun changes --- _extras/short_submission.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_extras/short_submission.md b/_extras/short_submission.md index fad089f..a360840 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -1,5 +1,5 @@ --- -title: Short Submission +title: Short submission with your own code --- ## this collects the sequence of steps for a batch submission with local code @@ -22,11 +22,11 @@ copy these scripts into that top level directory {% include setup-grid %} ~~~ -- setup-local + - makercds From 9c0e19b1ba6b9e07b52fd4e8dd8d794ebfbf001a Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Mon, 9 Feb 2026 10:06:13 -0800 Subject: [PATCH 04/27] fixes for short submissions --- CITATION | 2 +- _episodes/07-grid-job-submission.md | 44 +++++++++++++---------------- _extras/short_submission.md | 32 ++++++++++++--------- 3 files changed, 40 insertions(+), 38 deletions(-) diff --git a/CITATION b/CITATION index 95e5df9..3c2fc55 100644 --- a/CITATION +++ b/CITATION @@ -1,3 +1,3 @@ Please cite as: -Dune Collaboration: "DUNE Computing Tutorial" Version 2024.01 +Dune Collaboration: "DUNE Computing Tutorial" Version 2025.01 diff --git a/_episodes/07-grid-job-submission.md b/_episodes/07-grid-job-submission.md index 51e4f96..8140f12 100644 --- a/_episodes/07-grid-job-submission.md +++ b/_episodes/07-grid-job-submission.md @@ -68,8 +68,8 @@ The past few months have seen significant changes in how DUNE (as well as other First, log in to a `dunegpvm` machine . Then you will need to set up the job submission tools (`jobsub`). If you set up `dunesw` it will be included, but if not, you need to do ~~~ -mkdir -p /pnfs/dune/scratch/users/${USER}/DUNE_tutorial_sep2025 # if you have not done this before -mkdir -p /pnfs/dune/scratch/users/${USER}/sep2025tutorial +mkdir -p /pnfs/dune/scratch/users/${USER}/DUNE_tutorial_jan2026 # if you have not done this before +mkdir -p /pnfs/dune/scratch/users/${USER}/jan2026tutorial ~~~ {: ..language-bash} @@ -190,8 +190,8 @@ You will have to change the last line with your own submit file instead of the p First, we should make a tarball. Here is what we can do (assuming you are starting from /exp/dune/app/users/username/): ```bash -cp /exp/dune/app/users/kherner/setupsep2025tutorial-grid.sh /exp/dune/app/users/${USER}/ -cp /exp/dune/app/users/kherner/sep2025tutorial/localProducts_larsoft_v09_72_01_e20_prof/setup-grid /exp/dune/app/users/${USER}/sep2025tutorial/localProducts_larsoft_v09_72_01_e20_prof/setup-grid +cp /exp/dune/app/users/kherner/setupjan2026tutorial-grid.sh /exp/dune/app/users/${USER}/ +cp /exp/dune/app/users/kherner/jan2026tutorial/localProducts_larsoft_v09_72_01_e20_prof/setup-grid /exp/dune/app/users/${USER}/jan2026tutorial/localProducts_larsoft_v09_72_01_e20_prof/setup-grid ``` Before we continue, let's examine these files a bit. We will source the first one in our job script, and it will set up the environment for us. @@ -199,7 +199,7 @@ Before we continue, let's examine these files a bit. We will source the first on ~~~ #!/bin/bash -DIRECTORY=sep2025tutorial +DIRECTORY=jan2026tutorial # we cannot rely on "whoami" in a grid job. We have no idea what the local username will be. # Use the GRID_USER environment variable instead (set automatically by jobsub). USERNAME=${GRID_USER} @@ -217,40 +217,38 @@ mrbslp Now let's look at the difference between the setup-grid script and the plain setup script. -Assuming you are currently in the /exp/dune/app/users/username directory: +Assuming you are currently in the `/exp/dune/app/users/$USER` directory: ```bash -diff sep2025tutorial/localProducts_larsoft_v09_72_01_e20_prof/setup sep2025tutorial/localProducts_larsoft_v09_72_01_e20_prof/setup-grid +diff jan2026tutorial/localProducts_larsoft_v09_72_01_e20_prof/setup jan2026tutorial/localProducts_larsoft_v09_72_01_e20_prof/setup-grid ``` ~~~ -< setenv MRB_TOP "/exp/dune/app/users//sep2025tutorial" -< setenv MRB_TOP_BUILD "/exp/dune/app/users//sep2025tutorial" -< setenv MRB_SOURCE "/exp/dune/app/users//sep2025tutorial/srcs" -< setenv MRB_INSTALL "/exp/dune/app/users//sep2025tutorial/localProducts_larsoft_v09_72_01_e20_prof" +< setenv MRB_TOP "/exp/dune/app/users//jan2026tutorial" +< setenv MRB_TOP_BUILD "/exp/dune/app/users//jan2026tutorial" +< setenv MRB_SOURCE "/exp/dune/app/users//jan2026tutorial/srcs" +< setenv MRB_INSTALL "/exp/dune/app/users//jan2026tutorial/localProducts_larsoft_v09_72_01_e20_prof" --- -> setenv MRB_TOP "${INPUT_TAR_DIR_LOCAL}/sep2025tutorial" -> setenv MRB_TOP_BUILD "${INPUT_TAR_DIR_LOCAL}/sep2025tutorial" -> setenv MRB_SOURCE "${INPUT_TAR_DIR_LOCAL}/sep2025tutorial/srcs" -> setenv MRB_INSTALL "${INPUT_TAR_DIR_LOCAL}/sep2025tutorial/localProducts_larsoft_v09_72_01_e20_prof" +> setenv MRB_TOP "${INPUT_TAR_DIR_LOCAL}/jan2026tutorial" +> setenv MRB_TOP_BUILD "${INPUT_TAR_DIR_LOCAL}/jan2026tutorial" +> setenv MRB_SOURCE "${INPUT_TAR_DIR_LOCAL}/jan2026tutorial/srcs" +> setenv MRB_INSTALL "${INPUT_TAR_DIR_LOCAL}/jan2026tutorial/localProducts_larsoft_v09_72_01_e20_prof" ~~~ As you can see, we have switched from the hard-coded directories to directories defined by environment variables; the `INPUT_TAR_DIR_LOCAL` variable will be set for us (see below). -Now, let's actually create our tar file. Again assuming you are in `/exp/dune/app/users/kherner/sep2025tutorial/`: +Now, let's actually create our tar file. Again assuming you are in `/exp/dune/app/users/kherner/jan2026tutorial/`: ```bash -tar --exclude '.git' -czf sep2025tutorial.tar.gz sep2025tutorial/localProducts_larsoft_v09_72_01_e20_prof sep2025tutorial/work setupsep2025tutorial-grid.sh +tar --exclude '.git' -czf jan2026tutorial.tar.gz jan2026tutorial/localProducts_larsoft_${DUNESW_VERSION}_${DUNESW_QUALIFIER} jan2026tutorial/work setupjan2026tutorial-grid.sh ``` Note how we have excluded the contents of ".git" directories in the various packages, since we don't need any of that in our jobs. It turns out that the .git directory can sometimes account for a substantial fraction of a package's size on disk! Then submit another job (in the following we keep the same submit file as above): -```bash -jobsub_submit -G dune --mail_always -N 1 --memory=2500MB --disk=2GB --expected-lifetime=3h --cpu=1 --tar_file_name=dropbox:///exp/dune/app/users//sep2025tutorial.tar.gz --singularity-image /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest --append_condor_requirements='(TARGET.HAS_Singularity==true&&TARGET.HAS_CVMFS_dune_opensciencegrid_org==true&&TARGET.HAS_CVMFS_larsoft_opensciencegrid_org==true&&TARGET.CVMFS_dune_opensciencegrid_org_REVISION>=1105&&TARGET.HAS_CVMFS_fifeuser1_opensciencegrid_org==true&&TARGET.HAS_CVMFS_fifeuser2_opensciencegrid_org==true&&TARGET.HAS_CVMFS_fifeuser3_opensciencegrid_org==true&&TARGET.HAS_CVMFS_fifeuser4_opensciencegrid_org==true)' -e GFAL_PLUGIN_DIR=/usr/lib64/gfal2-plugins -e GFAL_CONFIG_DIR=/etc/gfal2.d file:///exp/dune/app/users/kherner/run_sep2025tutorial.sh -``` + You'll see this is very similar to the previous case, but there are some new options: -* `--tar_file_name=dropbox://` automatically **copies and untars** the given tarball into a directory on the worker node, accessed via the INPUT_TAR_DIR_LOCAL environment variable in the job. The value of INPUT_TAR_DIR_LOCAL is by default $CONDOR_DIR_INPUT/name_of_tar_file_without_extension, so if you have a tar file named e.g. sep2025tutorial.tar.gz, it would be $CONDOR_DIR_INPUT/sep2025tutorial. +* `--tar_file_name=dropbox://` automatically **copies and untars** the given tarball into a directory on the worker node, accessed via the INPUT_TAR_DIR_LOCAL environment variable in the job. The value of INPUT_TAR_DIR_LOCAL is by default $CONDOR_DIR_INPUT/name_of_tar_file_without_extension, so if you have a tar file named e.g. jan2026tutorial.tar.gz, it would be $CONDOR_DIR_INPUT/jan2026tutorial. * Notice that the `--append_condor_requirements` line is longer now, because we also check for the fifeuser[1-4]. opensciencegrid.org CVMFS repositories. The submission output will look something like this: @@ -265,7 +263,7 @@ Could not locate uploaded file on RCDS. Will retry in 30 seconds. Could not locate uploaded file on RCDS. Will retry in 30 seconds. Found uploaded file on RCDS. Transferring files to web sandbox... -Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/run_sep2025tutorial.sh [DONE] after 0s +Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/run_jan2026tutorial.sh [DONE] after 0s Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/simple.cmd [DONE] after 0s Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/simple.sh [DONE] after 0s Submitting job(s). @@ -566,8 +564,6 @@ Some more background material on these topics (including some examples of why ce [Wiki page listing differences between jobsub_lite and legacy jobsub](https://fifewiki.fnal.gov/wiki/Differences_between_jobsub_lite_and_legacy_jobsub_client/server) -[DUNE Computing Tutorial:Advanced topics and best practices](DUNE_computing_tutorial_advanced_topics_20210129) - [2021 Intensity Frontier Summer School](https://indico.fnal.gov/event/49414) [The Glidein-based Workflow Management System]( https://glideinwms.fnal.gov/doc.prd/index.html ) diff --git a/_extras/short_submission.md b/_extras/short_submission.md index a360840..18c72c5 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -16,7 +16,8 @@ export DIRECTORY=myworkarea copy these scripts into that top level directory -- setup-grid +> ## setup-grid +{: .callout} ~~~ {% include setup-grid %} @@ -28,38 +29,43 @@ copy these scripts into that top level directory {% include setup-local %} ~~~ --> -- makercds +> ## makercds +{: .callout} ~~~ {% include makercds.sh %} ~~~ -- submit_workflow.sh +> ## submit_workflow.sh +{: .callout} ~~~ {% include submit_workflow.sh %} ~~~ -- submit_local_code.jobscript.sh - +> ## submit_local_code.jobscript.sh +{: .callout} ~~~ {% include submit_local_code.jobscript.sh %} ~~~ -### modify this script (should not need to change the others) +> ## setup_before_submit.sh +{: .callout} +~~~ +{% include setup_before_submit.sh %} +~~~ +### modify one script (should not need to change the others) +edit `setup_before_submit.sh` to reflect the parameters you need. -- choose your code version and fcl file -- make certain the fcl file is either in the fcl path or in `$DIRECTORY` +- choose your code version and fcl file (code version has to match your build) +- *make certain the fcl file is either in the fcl path or in `$DIRECTORY`* - add a string `PROCESS_TYPE` that will go in your filename +- add a description in `DESCRIPTION` -#### setup_before_submit.sh -~~~ -{% include setup_before_submit.sh %} -~~~ Then run it to set things up @@ -89,7 +95,7 @@ go to [justin](https://dunejustin.fnal.gov/dashboard/?method=list-workflows) to track your job. - +[internal link](/files/setup-grid) From d94661a195267f457fe28ce7844f03179561e6d2 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Mon, 9 Feb 2026 13:28:49 -0800 Subject: [PATCH 05/27] add in -T option and more verbose setup script --- _episodes/02-submit-jobs-w-justin.md | 2 +- _episodes/07-grid-job-submission.md | 2 +- _episodes/08-justin-job-submission.md | 9 ++++++--- _includes/setup_local_test.sh | 23 ----------------------- 4 files changed, 8 insertions(+), 28 deletions(-) delete mode 100644 _includes/setup_local_test.sh diff --git a/_episodes/02-submit-jobs-w-justin.md b/_episodes/02-submit-jobs-w-justin.md index a31fe63..17bc29a 100644 --- a/_episodes/02-submit-jobs-w-justin.md +++ b/_episodes/02-submit-jobs-w-justin.md @@ -1,5 +1,5 @@ --- -title: Submit grid jobs with JustIn +title: New Justin Job Submission System teaching: 20 exercises: 0 questions: diff --git a/_episodes/07-grid-job-submission.md b/_episodes/07-grid-job-submission.md index 8140f12..ea93a3b 100644 --- a/_episodes/07-grid-job-submission.md +++ b/_episodes/07-grid-job-submission.md @@ -1,5 +1,5 @@ --- -title: Jobsub Grid Job Submission and Common Errors - still 2024 version +title: Jobsub Grid Job Submission and Common Errors (SPECIAL PURPOSE) teaching: 65 exercises: 0 questions: diff --git a/_episodes/08-justin-job-submission.md b/_episodes/08-justin-job-submission.md index 86249b0..c51bd4e 100644 --- a/_episodes/08-justin-job-submission.md +++ b/_episodes/08-justin-job-submission.md @@ -1,5 +1,5 @@ --- -title: Jobsub Grid Job Submission and Common Errors - justIn Version +title: justIn Grid Job Submission (UNDER CONSTRUCTION) teaching: 65 exercises: 0 questions: @@ -18,12 +18,12 @@ keypoints: - + -The video from the two day version of this training in May 2022 is provided [here](https://www.youtube.com/embed/QuDxkhq64Og) as a reference. +The video from the two day version of this training in May 2022 is provided [here](https://www.youtube.com/embed/QuDxkhq64Og) as a reference. --> +For now, please look at the short version of this sequence at +[Short Submission Runthrough]({{ site.baseurl }}/short_submission) + ## Submit a job Go to [The Justin Tutorial](https://dunejustin.fnal.gov/docs/tutorials.dune.md) diff --git a/_includes/setup_local_test.sh b/_includes/setup_local_test.sh deleted file mode 100644 index b2b648f..0000000 --- a/_includes/setup_local_test.sh +++ /dev/null @@ -1,23 +0,0 @@ -# setup_local_code.txt for example -export DUNE_VERSION=v09_91_02d01 -export DUNE_QUALIFIER=e26:prof -export FCL_FILE=run_analyseEvents.fcl -export INPUT_TAR_DIR_LOCAL=$PWD -export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 1 ordered " -export PROCESS_TYPE=analyze -export USERF=$USER - -# try doing a local setup for testing -echo "I am here now: `pwd`" -export localProductsdir=`ls -c1d $PWD/localProducts*` -echo "Local products directory is ${localProductsdir}" -source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh -export PRODUCTS="${localProductsdir}/:$PRODUCTS" -mv ${localProductsdir}/setup ${localProductsdir}/setup.bak -cp setup-local ${localProductsdir}/setup -# Then we can set up our local products -setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" -setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" -mrbslp -source ${localProductsdir}/setup - From 261bacd895a91ff4f14a3233c23a85d2e8d0a53a Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Tue, 10 Feb 2026 09:10:50 -0800 Subject: [PATCH 06/27] split tar from rcds --- _extras/short_submission.md | 22 ++++++++++++++++------ _includes/makercds.sh | 17 ++++++++++------- _includes/maketar.sh | 15 +++++++++++++++ _includes/setup-grid.md | 1 + _includes/setup_before_submit.sh | 12 +++++++++--- _includes/submit_local_code.jobscript.sh | 10 +++++++--- gitadd.sh | 15 ++------------- 7 files changed, 60 insertions(+), 32 deletions(-) create mode 100644 _includes/maketar.sh create mode 100644 _includes/setup-grid.md diff --git a/_extras/short_submission.md b/_extras/short_submission.md index 18c72c5..d6c2cb1 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -29,6 +29,19 @@ copy these scripts into that top level directory {% include setup-local %} ~~~ --> +> ## setup_before_submit.sh +{: .callout} +~~~ +{% include setup_before_submit.sh %} +~~~ + +> ## maketar +{: .callout} + +~~~ +{% include maketar.sh %} +~~~ + > ## makercds {: .callout} @@ -50,11 +63,7 @@ copy these scripts into that top level directory {% include submit_local_code.jobscript.sh %} ~~~ -> ## setup_before_submit.sh -{: .callout} -~~~ -{% include setup_before_submit.sh %} -~~~ + ### modify one script (should not need to change the others) edit `setup_before_submit.sh` to reflect the parameters you need. @@ -78,10 +87,11 @@ source setup_before_submit.sh If you have changed any scripts or code, you must redo this. ~~~ +./maketar.sh $DIRECTORY ./makercds.sh $DIRECTORY ~~~ -will take a while, produce a tarball on /exp/dune/data and put the cvmfs location in cvmfs.location +will take a while, produce a tarball on /exp/dune/data and put the cvmfs location in cvmfs.location in `$DIRECTORY` ### Submit the job diff --git a/_includes/makercds.sh b/_includes/makercds.sh index 9d0ef09..778b20e 100644 --- a/_includes/makercds.sh +++ b/_includes/makercds.sh @@ -1,13 +1,16 @@ # give me the directory name as argument +echo "----------------------------------------------------------------" +echo "makercds.sh" +justin get-token export HERE=`pwd` # put the tar file on a bigger disk export THERE=/exp/dune/data/users/$USER/ -cd .. # go up one -echo " make tar file" -tar --exclude '.git' --exclude build_slf7.x86_64 -czf $THERE/$1.tar.gz $1 +date ls -lrt $THERE/$1.tar.gz -echo " upload tar file to cvmfs" +echo " upload tar file to cvmfs and store location in cvmfs.location file" export INPUT_TAR_DIR_LOCAL=`justin-cvmfs-upload $THERE/$1.tar.gz` -echo $INPUT_TAR_DIR_LOCAL -echo $INPUT_TAR_DIR_LOCAL > $1/cvmfs.location -cd $HERE \ No newline at end of file +echo "file uploaded to $INPUT_TAR_DIR_LOCAL" +echo $INPUT_TAR_DIR_LOCAL > $HERE/cvmfs.location +echo "return to previous directory" +cd $HERE +echo "----------------------------------------------------------------" \ No newline at end of file diff --git a/_includes/maketar.sh b/_includes/maketar.sh new file mode 100644 index 0000000..ef7541d --- /dev/null +++ b/_includes/maketar.sh @@ -0,0 +1,15 @@ +# give me the directory name as argument +echo "----------------------------------------------------------------" +echo "maketar.sh" +export HERE=`pwd` +# put the tar file on a bigger disk +export THERE=/exp/dune/data/users/$USER/ +cd .. # go up one from current directory +date +echo " make tar file" +tar --exclude '.git' --exclude build_slf7.x86_64 -cf $THERE/$1.tar $1 +date +gzip -f $THERE/$1.tar +date +echo " tar file is at $THERE/$1.tar.gz" +echo "----------------------------------------------------------------" diff --git a/_includes/setup-grid.md b/_includes/setup-grid.md new file mode 100644 index 0000000..6b54c66 --- /dev/null +++ b/_includes/setup-grid.md @@ -0,0 +1 @@ +{% include setup-grid %} \ No newline at end of file diff --git a/_includes/setup_before_submit.sh b/_includes/setup_before_submit.sh index b2b2f92..8a29cdd 100644 --- a/_includes/setup_before_submit.sh +++ b/_includes/setup_before_submit.sh @@ -1,10 +1,11 @@ -# setup_local_test.sh for example +echo "----------------------------------------------------------------" +echo "setup_before_submit.sh" export DIRECTORY="$(basename "${PWD}")" export DUNE_VERSION=v09_91_02d01 export DUNE_QUALIFIER=e26:prof export FCL_FILE=run_analyseEvents.fcl -export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 2 ordered " -export PROCESS_TYPE=analyze +export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 5 ordered " +export PROCESS_TYPE=ana export DESCRIPTION="$PROCESS_TYPE using $FCL_FILE" export USERF=$USER export NUM_EVENTS=1 @@ -15,12 +16,17 @@ source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" setup justin +echo " code set up" justin time justin get-token +echo " got a token from justin" # make certain the setup script in localProducts is what you want export localProductsdir=`ls -c1d $PWD/localProducts*` cp setup-grid $localProductsdir/setup +echo "Now you should ./makercds.sh $DIRECTORY" +echo "and then submit with ./submit_workflow.sh" +echo "----------------------------------------------------------------" diff --git a/_includes/submit_local_code.jobscript.sh b/_includes/submit_local_code.jobscript.sh index c3c3642..eb93f90 100644 --- a/_includes/submit_local_code.jobscript.sh +++ b/_includes/submit_local_code.jobscript.sh @@ -92,10 +92,14 @@ source ${localProductsdir}/setup mrbslp # Construct outFile from input $pfn -now=$(date -u +"%Y-%m-%dT_%H%M%SZ") +now=$(date -u +"%Y%m%d%H%M%SZ") Ffname=`echo $pfn | awk -F/ '{print $NF}'` fname=`echo $Ffname | awk -F. '{print $1}'` -outFile=${fname}_${PROCESS_TYPE}_${now}.root +# outFile1 is artroot format +# outFile2 is root format for analysis +outFile1=${fname}_${PROCESS_TYPE}_${now}.root +outFile2=${fname}_${PROCESS_TYPE}_tuple_${now}.root + campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" @@ -107,7 +111,7 @@ echo "$LD_PRELOAD" echo "now run lar" -lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o $outFile "$pfn" > ${fname}_${PROCESS_TYPE}_${now}.log 2>&1 +lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o $outFile1 -T ${outFile2} "$pfn" > ${fname}_${PROCESS_TYPE}_${now}.log 2>&1 ) diff --git a/gitadd.sh b/gitadd.sh index 5f235b1..f65fe7f 100644 --- a/gitadd.sh +++ b/gitadd.sh @@ -1,19 +1,8 @@ git add *.md git add _episodes/*.md -#git add _episodes/01-introduction.md -#git add _episodes/02-storage-spaces.md -#git add _episodes/03-data-management.md -#git add _episodes/03.2-UPS.md -#git add _episodes/03.3-cvmfs.md -#git add _episodes/04-intro-art-larsoft.md -#git add _episodes/05.5-mrb.md -#git add _episodes/06-larsoft-modify-module.md -#git add _episodes/07-grid-job-submission_al9.md -#git add _episodes/07-grid-job-submission-al9.md -#git add _episodes/08-submit-jobs-w-justin.md -#git add _episodes/09-grid-batch-debug.md -#git add _episodes/10-closing-remarks.md git add _includes/*.html +git add _includes/*.sh +git add _includes/setup* git add *.yml git add _extras/*.md git add AUTHORS CITATION From 08e795727225dc3e50de72b232004000192666fb Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Tue, 10 Feb 2026 11:50:20 -0800 Subject: [PATCH 07/27] add justin commands to makercds --- _includes/makercds.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/_includes/makercds.sh b/_includes/makercds.sh index 778b20e..101d1fc 100644 --- a/_includes/makercds.sh +++ b/_includes/makercds.sh @@ -1,6 +1,8 @@ # give me the directory name as argument echo "----------------------------------------------------------------" echo "makercds.sh" +echo "first ensure you have a justin token" +justin time justin get-token export HERE=`pwd` # put the tar file on a bigger disk From d228ea1cdaa74736d2ca15b25b118981ebcf2992 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Wed, 11 Feb 2026 11:56:50 -0800 Subject: [PATCH 08/27] clean up the setup-grid situation --- _extras/short_submission.md | 50 ++------- _includes/setup-grid | 11 +- _includes/setup-grid.sav | 129 +++++++++++++++++++++++ _includes/setup_before_submit.sh | 26 +++-- _includes/setup_own_build.sh | 26 +++++ _includes/submission_instructions.md | 6 ++ _includes/submit_local_code.jobscript.sh | 4 +- 7 files changed, 194 insertions(+), 58 deletions(-) create mode 100644 _includes/setup-grid.sav create mode 100644 _includes/setup_own_build.sh create mode 100644 _includes/submission_instructions.md diff --git a/_extras/short_submission.md b/_extras/short_submission.md index d6c2cb1..63af0bd 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -14,54 +14,22 @@ need to have a name for it as you will be making a tarball export DIRECTORY=myworkarea ~~~ -copy these scripts into that top level directory +### copy these scripts into that top level directory -> ## setup-grid -{: .callout} -~~~ -{% include setup-grid %} -~~~ +[setup-grid](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup-grid) (should not need to modify) - - -> ## setup_before_submit.sh -{: .callout} -~~~ -{% include setup_before_submit.sh %} -~~~ +[makerdcs.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/makerdcs.sh) (should not need to modify) -> ## maketar -{: .callout} +[setup_before_submit.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup_before_submit.sh) (customize for your code) -~~~ -{% include maketar.sh %} -~~~ +[submit_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_workflow.sh) (modify running time and memory) -> ## makercds -{: .callout} +[submit_local_code.jobscript.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_local_code.jobscript.sh) (may need to modify if expert) -~~~ -{% include makercds.sh %} -~~~ - -> ## submit_workflow.sh -{: .callout} - -~~~ -{% include submit_workflow.sh %} -~~~ - - -> ## submit_local_code.jobscript.sh -{: .callout} -~~~ -{% include submit_local_code.jobscript.sh %} -~~~ ### modify one script (should not need to change the others) @@ -95,6 +63,8 @@ will take a while, produce a tarball on /exp/dune/data and put the cvmfs locatio ### Submit the job +[submit_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_workflow.sh) + ~~~ ./submit_workflow.sh ~~~ diff --git a/_includes/setup-grid b/_includes/setup-grid index 264d7ea..d8c2f85 100644 --- a/_includes/setup-grid +++ b/_includes/setup-grid @@ -87,15 +87,15 @@ tnotnull MRB_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg tnotnull MRB_DIR || return 1 setenv MRB_PROJECT "larsoft" -setenv MRB_PROJECT_VERSION "v09_91_02d01" -setenv MRB_QUALS "e26:prof" -setenv MRB_QUALSTRING = "e26_prof" +setenv MRB_PROJECT_VERSION ${DUNE_VERSION} +setenv MRB_QUALS ${DUNE_QUALIFIER} +setenv MRB_QUALSTRING `sed s/:/_/g <<< ${MRB_QUALS}` setenv MRB_TOP "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}" setenv MRB_TOP_BUILD "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}" setenv MRB_SOURCE "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/srcs" -setenv MRB_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_v09_91_02d01_e26_prof" +setenv MRB_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_${MRB_PROJECT_VERSION}_${MRB_QUALSTRING}" setenv PRODUCTS "${MRB_INSTALL}:${PRODUCTS}" -setenv CETPKG_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_v09_91_02d01_e26_prof" +setenv CETPKG_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_${MRB_PROJECT_VERSION}_${MRB_QUALSTRING}" #--- begin middle boilerplate @@ -114,6 +114,7 @@ echo echo MRB_PROJECT=$MRB_PROJECT echo MRB_PROJECT_VERSION=$MRB_PROJECT_VERSION echo MRB_QUALS=$MRB_QUALS +echo MRB_QUALSTRING=$DUNESW_QUALIFIER_STRING echo MRB_TOP=$MRB_TOP echo MRB_SOURCE=$MRB_SOURCE echo MRB_BUILDDIR=$MRB_BUILDDIR diff --git a/_includes/setup-grid.sav b/_includes/setup-grid.sav new file mode 100644 index 0000000..264d7ea --- /dev/null +++ b/_includes/setup-grid.sav @@ -0,0 +1,129 @@ +# No magic #!, this script must be sourced! +# this script is part of mrb and gets renamed to setup when copied to the localProducts area + +# NOTICE: this script is not relocatable + +# +# Begin boilerplate. +# + +# Note: All the following special tricks for $_ must continue +# relaying the value to the next rule. Be careful! +# Special trick to nail the value of $_ down in a variety of shells. +echo $_ >& /dev/null +# Special trick for tcsh which is one-off on the command history stack. +: $_ +# Special trick to capture the value of $_ in zsh and bash +test $?shell$_ != 1$_ >& /dev/null && \ + dollar_underscore="$_" && \ + dollar_underscore=`expr "${dollar_underscore}" : ".\(.*\)"` +# Special trick to capture the value of $_ in tcsh +test $?shell = 1 && set dollar_underscore=`echo $_` + +# need to be able to check for mrb +test $?shell = 1 && set ss="csh" || ss="sh" +test "$ss" = "csh" && alias return exit + +test "$ss" = "csh" && \ + alias tnotnull "eval '"'test $?'"\!* -eq 1' && eval '"'test -n "$'"\!*"'"'"'" +test "$ss" = "sh" && \ + eval 'tnotnull() { eval "test -n \"\${$1-}\"" ;}' + +# check for mrb +tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS!" ; echo "ERROR:" ) +tnotnull UPS_DIR || unset ss +tnotnull UPS_DIR || return 1 +tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first setup mrb!"; echo "ERROR:" ) +tnotnull MRB_DIR || unset ss +tnotnull MRB_DIR || return 1 +test -f "$MRB_DIR/libexec/shell_independence" || \ + ( echo "ERROR:" ; echo "ERROR: this mrb area expects mrb >= v5_00_00 (found $MRB_VERSION)!" ; echo "ERROR:" ) +test -f "$MRB_DIR/libexec/shell_independence" || unset ss +test -f "$MRB_DIR/libexec/shell_independence" || return 1 + +# Get the shell independence aliases and functions. +source "$MRB_DIR/libexec/shell_independence" + +# Capture the value of $0 +set_ dollar_zed=`echo "${0}" | sed -e 's/^-//'` + +# Special tricks to figure out if this script has been sourced. +# Works for bash, tcsh, and in some cases for zsh. +set_ is_sourced=false +ifcsh_ + # Note: It is unfortunate that we must hard-code the name + # of this script here, but there is no other way + # that works, tcsh is brain-dead. + set base=`basename "${dollar_zed}"` + test "${base}" != "setup" && \ + set is_sourced=true +else + # Special trick for zsh. + test "${ZSH_NAME}" && test "${dollar_underscore}" = "${dollar_zed}" && \ + is_sourced=true + # If there were arguments then there is no safe way to find out + # whether or not the script was sourced in zsh. Pretend it was. + test "${ZSH_NAME}" && test "${#argv}" != "0" && \ + is_sourced=true + # Special trick for bash. + test "${BASH}" && test "${BASH_SOURCE}" != "${dollar_zed}" && \ + is_sourced=true +# Warning, this must be here because the tcsh parser is brain-dead. +endif +endifcsh_ + +# +# End of boilerplate. Begin of real work. +# + +tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS" ; echo "ERROR:" ) +tnotnull UPS_DIR || source "$MRB_DIR/libexec/unset_shell_independence" +tnotnull UPS_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav +tnotnull UPS_DIR || return 1 + + +tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first set up mrb!"; echo "ERROR:" ) +tnotnull MRB_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav +tnotnull MRB_DIR || return 1 + +setenv MRB_PROJECT "larsoft" +setenv MRB_PROJECT_VERSION "v09_91_02d01" +setenv MRB_QUALS "e26:prof" +setenv MRB_QUALSTRING = "e26_prof" +setenv MRB_TOP "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}" +setenv MRB_TOP_BUILD "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}" +setenv MRB_SOURCE "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/srcs" +setenv MRB_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_v09_91_02d01_e26_prof" +setenv PRODUCTS "${MRB_INSTALL}:${PRODUCTS}" +setenv CETPKG_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_v09_91_02d01_e26_prof" + +#--- begin middle boilerplate + +set_ flav=`get-directory-name subdir` +set_ buildDirName="build_${flav}" + +test "$ss" = sh && test -n "${MRB_BUILDDIR}" && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" +test "$ss" = csh && tnotnull MRB_BUILDDIR && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" +setenv MRB_BUILDDIR ${MRB_TOP_BUILD}/${buildDirName} + +unset me dollar_underscore dollar_zed is_sourced base msg1 flav + +#--- end middle boilerplate +# report the environment +echo +echo MRB_PROJECT=$MRB_PROJECT +echo MRB_PROJECT_VERSION=$MRB_PROJECT_VERSION +echo MRB_QUALS=$MRB_QUALS +echo MRB_TOP=$MRB_TOP +echo MRB_SOURCE=$MRB_SOURCE +echo MRB_BUILDDIR=$MRB_BUILDDIR +echo MRB_INSTALL=$MRB_INSTALL +echo MRB_DIR=$MRB_DIR +echo +echo PRODUCTS=$PRODUCTS +echo CETPKG_INSTALL=$CETPKG_INSTALL +echo + +source "$MRB_DIR/libexec/unset_shell_independence" +unset db buildDirName + diff --git a/_includes/setup_before_submit.sh b/_includes/setup_before_submit.sh index 8a29cdd..ace14f3 100644 --- a/_includes/setup_before_submit.sh +++ b/_includes/setup_before_submit.sh @@ -1,14 +1,16 @@ echo "----------------------------------------------------------------" echo "setup_before_submit.sh" + export DIRECTORY="$(basename "${PWD}")" export DUNE_VERSION=v09_91_02d01 export DUNE_QUALIFIER=e26:prof -export FCL_FILE=run_analyseEvents.fcl +export DUNE_QUALIFIER_STRING=`echo ${DUNE_QUALIFIER} | tr : _` +export FCL_FILE="run_analyseEvents.fcl" export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 5 ordered " -export PROCESS_TYPE=ana +export PROCESS_TYPE="ana" export DESCRIPTION="$PROCESS_TYPE using $FCL_FILE" -export USERF=$USER -export NUM_EVENTS=1 +export USERF=${USER} +export NUM_EVENTS=-1 # process them all export FNALURL='https://fndcadoor.fnal.gov:2880/dune/scratch/users' ## set up locally just as a check @@ -18,15 +20,17 @@ setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" setup justin echo " code set up" justin time +echo " you may need to authorize this computer to run the justin command - check to see if there is a URL above and go there to authenticate" justin get-token -echo " got a token from justin" -# make certain the setup script in localProducts is what you want -export localProductsdir=`ls -c1d $PWD/localProducts*` -cp setup-grid $localProductsdir/setup -echo "Now you should ./makercds.sh $DIRECTORY" -echo "and then submit with ./submit_workflow.sh" -echo "----------------------------------------------------------------" +export localProductsdir="${PWD}/localProducts_larsoft_${DUNE_VERSION}_${DUNE_QUALIFIER_STRING}" +echo " localProductsdir ${localProductsdir}" +cp setup-grid $localProductsdir/setup-grid +echo "Now you should:" +echo "./maketar.sh $DIRECTORY" +echo "./makercds.sh $DIRECTORY" +echo "./submit_workflow.sh" +echo "----------------------------------------------------------------" \ No newline at end of file diff --git a/_includes/setup_own_build.sh b/_includes/setup_own_build.sh new file mode 100644 index 0000000..12c0360 --- /dev/null +++ b/_includes/setup_own_build.sh @@ -0,0 +1,26 @@ +#!/bin/bash + +DIRECTORY=jan2023tutorial +# we cannot rely on "whoami" in a grid job. We have no idea what the local username will be. +# Use the GRID_USER environment variable instead (set automatically by jobsub). +USERNAME=${GRID_USER} +source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh +export WORKDIR=${_CONDOR_JOB_IWD} # if we use the RCDS the our tarball will be placed in $INPUT_TAR_DIR_LOCAL. +if [ ! -d "$WORKDIR" ]; then + export WORKDIR=`echo .` +fi + +# if you are using an older release you MAY get an error of this form: + +# ERROR: +# ERROR: this mrb area expects mrb < v5_00_00 (found some version > v5)! +# ERROR: + +# If you do, the workaround is the following (comment these lines out if not needed) +#unsetup mrb +#setup mrb -o + + +source ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts*/setup-grid + +mrbslp diff --git a/_includes/submission_instructions.md b/_includes/submission_instructions.md new file mode 100644 index 0000000..35498e6 --- /dev/null +++ b/_includes/submission_instructions.md @@ -0,0 +1,6 @@ +## submission instructions + +There are several files you need. + + +`setup-grid` and `setup-local` are mrb setup files, let's hope you don't need to change them. diff --git a/_includes/submit_local_code.jobscript.sh b/_includes/submit_local_code.jobscript.sh index eb93f90..e6e3565 100644 --- a/_includes/submit_local_code.jobscript.sh +++ b/_includes/submit_local_code.jobscript.sh @@ -80,7 +80,7 @@ echo "localProductsdir ${localProductsdir}" # seems to require the right name for the setup script echo " check that there is a setup in ${localProductsdir}" -ls -lrt ${localProductsdir}/setup +ls -lrt ${localProductsdir}/setup-grid ls -lrt ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh export PRODUCTS="${localProductsdir}/:$PRODUCTS" @@ -88,7 +88,7 @@ export PRODUCTS="${localProductsdir}/:$PRODUCTS" # Then we can set up our local products setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" -source ${localProductsdir}/setup +source ${localProductsdir}/setup-grid mrbslp # Construct outFile from input $pfn From c38efbc8e3a14cbd50888144cd71a17bc58f1221 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Wed, 11 Feb 2026 11:57:45 -0800 Subject: [PATCH 09/27] clean up the setup-grid situation --- _includes/setup-grid.sav | 129 --------------------------- _includes/setup_own_build.sh | 26 ------ _includes/submission_instructions.md | 6 -- 3 files changed, 161 deletions(-) delete mode 100644 _includes/setup-grid.sav delete mode 100644 _includes/setup_own_build.sh delete mode 100644 _includes/submission_instructions.md diff --git a/_includes/setup-grid.sav b/_includes/setup-grid.sav deleted file mode 100644 index 264d7ea..0000000 --- a/_includes/setup-grid.sav +++ /dev/null @@ -1,129 +0,0 @@ -# No magic #!, this script must be sourced! -# this script is part of mrb and gets renamed to setup when copied to the localProducts area - -# NOTICE: this script is not relocatable - -# -# Begin boilerplate. -# - -# Note: All the following special tricks for $_ must continue -# relaying the value to the next rule. Be careful! -# Special trick to nail the value of $_ down in a variety of shells. -echo $_ >& /dev/null -# Special trick for tcsh which is one-off on the command history stack. -: $_ -# Special trick to capture the value of $_ in zsh and bash -test $?shell$_ != 1$_ >& /dev/null && \ - dollar_underscore="$_" && \ - dollar_underscore=`expr "${dollar_underscore}" : ".\(.*\)"` -# Special trick to capture the value of $_ in tcsh -test $?shell = 1 && set dollar_underscore=`echo $_` - -# need to be able to check for mrb -test $?shell = 1 && set ss="csh" || ss="sh" -test "$ss" = "csh" && alias return exit - -test "$ss" = "csh" && \ - alias tnotnull "eval '"'test $?'"\!* -eq 1' && eval '"'test -n "$'"\!*"'"'"'" -test "$ss" = "sh" && \ - eval 'tnotnull() { eval "test -n \"\${$1-}\"" ;}' - -# check for mrb -tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS!" ; echo "ERROR:" ) -tnotnull UPS_DIR || unset ss -tnotnull UPS_DIR || return 1 -tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first setup mrb!"; echo "ERROR:" ) -tnotnull MRB_DIR || unset ss -tnotnull MRB_DIR || return 1 -test -f "$MRB_DIR/libexec/shell_independence" || \ - ( echo "ERROR:" ; echo "ERROR: this mrb area expects mrb >= v5_00_00 (found $MRB_VERSION)!" ; echo "ERROR:" ) -test -f "$MRB_DIR/libexec/shell_independence" || unset ss -test -f "$MRB_DIR/libexec/shell_independence" || return 1 - -# Get the shell independence aliases and functions. -source "$MRB_DIR/libexec/shell_independence" - -# Capture the value of $0 -set_ dollar_zed=`echo "${0}" | sed -e 's/^-//'` - -# Special tricks to figure out if this script has been sourced. -# Works for bash, tcsh, and in some cases for zsh. -set_ is_sourced=false -ifcsh_ - # Note: It is unfortunate that we must hard-code the name - # of this script here, but there is no other way - # that works, tcsh is brain-dead. - set base=`basename "${dollar_zed}"` - test "${base}" != "setup" && \ - set is_sourced=true -else - # Special trick for zsh. - test "${ZSH_NAME}" && test "${dollar_underscore}" = "${dollar_zed}" && \ - is_sourced=true - # If there were arguments then there is no safe way to find out - # whether or not the script was sourced in zsh. Pretend it was. - test "${ZSH_NAME}" && test "${#argv}" != "0" && \ - is_sourced=true - # Special trick for bash. - test "${BASH}" && test "${BASH_SOURCE}" != "${dollar_zed}" && \ - is_sourced=true -# Warning, this must be here because the tcsh parser is brain-dead. -endif -endifcsh_ - -# -# End of boilerplate. Begin of real work. -# - -tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS" ; echo "ERROR:" ) -tnotnull UPS_DIR || source "$MRB_DIR/libexec/unset_shell_independence" -tnotnull UPS_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav -tnotnull UPS_DIR || return 1 - - -tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first set up mrb!"; echo "ERROR:" ) -tnotnull MRB_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav -tnotnull MRB_DIR || return 1 - -setenv MRB_PROJECT "larsoft" -setenv MRB_PROJECT_VERSION "v09_91_02d01" -setenv MRB_QUALS "e26:prof" -setenv MRB_QUALSTRING = "e26_prof" -setenv MRB_TOP "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}" -setenv MRB_TOP_BUILD "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}" -setenv MRB_SOURCE "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/srcs" -setenv MRB_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_v09_91_02d01_e26_prof" -setenv PRODUCTS "${MRB_INSTALL}:${PRODUCTS}" -setenv CETPKG_INSTALL "${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts_larsoft_v09_91_02d01_e26_prof" - -#--- begin middle boilerplate - -set_ flav=`get-directory-name subdir` -set_ buildDirName="build_${flav}" - -test "$ss" = sh && test -n "${MRB_BUILDDIR}" && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" -test "$ss" = csh && tnotnull MRB_BUILDDIR && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" -setenv MRB_BUILDDIR ${MRB_TOP_BUILD}/${buildDirName} - -unset me dollar_underscore dollar_zed is_sourced base msg1 flav - -#--- end middle boilerplate -# report the environment -echo -echo MRB_PROJECT=$MRB_PROJECT -echo MRB_PROJECT_VERSION=$MRB_PROJECT_VERSION -echo MRB_QUALS=$MRB_QUALS -echo MRB_TOP=$MRB_TOP -echo MRB_SOURCE=$MRB_SOURCE -echo MRB_BUILDDIR=$MRB_BUILDDIR -echo MRB_INSTALL=$MRB_INSTALL -echo MRB_DIR=$MRB_DIR -echo -echo PRODUCTS=$PRODUCTS -echo CETPKG_INSTALL=$CETPKG_INSTALL -echo - -source "$MRB_DIR/libexec/unset_shell_independence" -unset db buildDirName - diff --git a/_includes/setup_own_build.sh b/_includes/setup_own_build.sh deleted file mode 100644 index 12c0360..0000000 --- a/_includes/setup_own_build.sh +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/bash - -DIRECTORY=jan2023tutorial -# we cannot rely on "whoami" in a grid job. We have no idea what the local username will be. -# Use the GRID_USER environment variable instead (set automatically by jobsub). -USERNAME=${GRID_USER} -source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh -export WORKDIR=${_CONDOR_JOB_IWD} # if we use the RCDS the our tarball will be placed in $INPUT_TAR_DIR_LOCAL. -if [ ! -d "$WORKDIR" ]; then - export WORKDIR=`echo .` -fi - -# if you are using an older release you MAY get an error of this form: - -# ERROR: -# ERROR: this mrb area expects mrb < v5_00_00 (found some version > v5)! -# ERROR: - -# If you do, the workaround is the following (comment these lines out if not needed) -#unsetup mrb -#setup mrb -o - - -source ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts*/setup-grid - -mrbslp diff --git a/_includes/submission_instructions.md b/_includes/submission_instructions.md deleted file mode 100644 index 35498e6..0000000 --- a/_includes/submission_instructions.md +++ /dev/null @@ -1,6 +0,0 @@ -## submission instructions - -There are several files you need. - - -`setup-grid` and `setup-local` are mrb setup files, let's hope you don't need to change them. From c34e8b6b262b095d6f4a6a91107c9212b0424fc1 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 14 Feb 2026 13:59:29 -0800 Subject: [PATCH 10/27] make metadata --- _extras/short_submission.md | 22 +- _includes/DUNEmdSpec.json | 123 +++++ _includes/MDValidator.py | 158 ++++++ _includes/extractor_new.py | 490 ++++++++++++++++++ _includes/job_config.sh | 10 + _includes/makercds.sh | 0 _includes/maketar.sh | 1 + _includes/sam2metacat.py | 195 +++++++ _includes/setup_before_submit.sh | 11 +- _includes/submit_local_code.jobscript.sh | 77 ++- .../submit_local_code_rucio.jobscript.sh | 179 +++++++ _includes/submit_workflow.sh | 10 +- _includes/submit_workflow_rucio.sh | 22 + _includes/test_workflow.sh | 26 + 14 files changed, 1290 insertions(+), 34 deletions(-) create mode 100644 _includes/DUNEmdSpec.json create mode 100644 _includes/MDValidator.py create mode 100755 _includes/extractor_new.py create mode 100644 _includes/job_config.sh mode change 100644 => 100755 _includes/makercds.sh mode change 100644 => 100755 _includes/maketar.sh create mode 100644 _includes/sam2metacat.py mode change 100644 => 100755 _includes/setup_before_submit.sh mode change 100644 => 100755 _includes/submit_local_code.jobscript.sh create mode 100755 _includes/submit_local_code_rucio.jobscript.sh mode change 100644 => 100755 _includes/submit_workflow.sh create mode 100755 _includes/submit_workflow_rucio.sh create mode 100755 _includes/test_workflow.sh diff --git a/_extras/short_submission.md b/_extras/short_submission.md index 63af0bd..37ef1a0 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -2,7 +2,7 @@ title: Short submission with your own code --- -## this collects the sequence of steps for a batch submission with local code +## this collects the sequence of steps for a batch submission with local code which produces both an artroot and root file ### in your top level mrb directory @@ -24,26 +24,30 @@ export DIRECTORY=myworkarea [makerdcs.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/makerdcs.sh) (should not need to modify) -[setup_before_submit.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup_before_submit.sh) (customize for your code) +[job_config.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/job_config.sh) (you modify this to choose things like MQL query, number of events..) + +[setup_before_submit.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup_before_submit.sh) (customize versions for your code) [submit_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_workflow.sh) (modify running time and memory) +[test_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/test_workflow.sh) (script to do interactive tests of your jobscript) + + +[extractor_new.py](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/extractor_new.py) (this makes metadata for your files) + [submit_local_code.jobscript.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_local_code.jobscript.sh) (may need to modify if expert) ### modify one script (should not need to change the others) -edit `setup_before_submit.sh` to reflect the parameters you need. +edit `setup_before_submit.sh` and `job_config.sh` to reflect the parameters you need. -- choose your code version and fcl file (code version has to match your build) +- choose your code version (code version has to match your build) - *make certain the fcl file is either in the fcl path or in `$DIRECTORY`* -- add a string `PROCESS_TYPE` that will go in your filename +- add a string `APP_NAME` that will go in your filename - add a description in `DESCRIPTION` - - - Then run it to set things up ~~~ @@ -61,6 +65,8 @@ If you have changed any scripts or code, you must redo this. will take a while, produce a tarball on /exp/dune/data and put the cvmfs location in cvmfs.location in `$DIRECTORY` +Then edit `job_config.sh` to reflect the # of events you want and other run-time parameters. + ### Submit the job [submit_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_workflow.sh) diff --git a/_includes/DUNEmdSpec.json b/_includes/DUNEmdSpec.json new file mode 100644 index 0000000..bb6ed4f --- /dev/null +++ b/_includes/DUNEmdSpec.json @@ -0,0 +1,123 @@ + +{ "known_fields": { + "core.run_type":[ + "fardet", + "neardet", + "protodune", + "protodune-sp", + "protodune-dp", + "35ton", + "311", + "311_dp_light", + "iceberg", + "fardet-sp", + "fardet-dp", + "fardet-moo", + "np04_vst", + "vd-coldbox-bottom", + "vd-coldbox-top", + "protodune-hd", + "hd-coldbox", + "vd-protodune-arapucas", + "protodune-vst", + "vd-protodune-pds", + "fardet-hd", + "fardet-vd", + "dc4-vd-coldbox-bottom", + "dc4-vd-coldbox-top", + "dc4-hd-protodune", + "hd-protodune", + "neardet-lar", + "neardet-2x2-minerva", + "neardet-2x2-lar-charge", + "neardet-2x2-lar-light", + "neardet-2x2", + "neardet-2x2-lar", + "vd-protodune", + "vd-coldbox" + ], + "core.file_type":[ + "detector", + "mc", + "importedDetector" + ], + "core.data_tier":[ + "simulated", + "raw", + "hit-reconstructed", + "full-reconstructed", + "generated", + "detector-simulated", + "reconstructed-2d", + "reconstructed-3d", + "sliced", + "dc1input", + "dc1output", + "root-tuple", + "root-hist", + "dqm", + "decoded-raw", + "sam-user", + "pandora_info", + "reco-recalibrated", + "storage-testing", + "root-tuple-virtual", + "binary-raw", + "trigprim", + "pandora-info" + ], + "core.data_stream":[ + "out1", + "noise", + "test", + "cosmics", + "calibration", + "physics", + "commissioning", + "out2", + "pedestal", + "study", + "trigprim", + "pdstl", + "linjc", + "numib", + "numip", + "numil" + ] + }, + "basetypes":{ + "name": "STRING", + "namespace": "STRING", + "size":"INT", + "metadata": { + "core.application.family": "STRING", + "core.application.name": "STRING", + "core.application.version": "STRING", + "core.data_stream":"STRING", + "core.data_tier": "STRING", + "core.end_time": "FLOAT", + "core.event_count": "INT", + "core.events": "LIST", + "core.file_content_status": "STRING", + "core.file_format": "STRING", + "core.file_type": "STRING", + "core.first_event_number": "INT", + "core.last_event_number": "INT", + "core.run_type": "STRING", + "core.runs": "LIST", + "core.runs_subruns": "LIST", + "core.start_time": "FLOAT", + "dune.daq_test": "STRING", + "dune.config_file": "STRING", + "dune_mc.gen_fcl_filename": "STRING", + "dune_mc.geometry_version":"STRING", + "retention.status": "STRING", + "retention.class": "STRING" + } + }, + "fixDefaults":{ + "core.file_content_status": "good", + "retention.status": "active", + "retention.class": "unknown" + } +} \ No newline at end of file diff --git a/_includes/MDValidator.py b/_includes/MDValidator.py new file mode 100644 index 0000000..96125e8 --- /dev/null +++ b/_includes/MDValidator.py @@ -0,0 +1,158 @@ +"""Check metadata against a template""" +import os,sys,json + +DEBUG=False + +def TypeChecker(filemd=None, errfile=None, verbose=False): + " check for type and missing required fields in metadata" + + # define types + valuetypes = { + "STRING" : type(""), + "FLOAT" : type(1.0), + "INT" : type(1), + "LIST" : type([]), + "DICT" : type({}), + } + + # read in the defaults + + f = open("DUNEmdSpec.json",'r') + config = json.load(f) + f.close() + + # list defaults for metadata fields + + + + # set default values for fields that are often missing but needed + fixDefaults = { + "core.file_content_status":"good", + "retention.status":"active", + "retention.class":"unknown" + } + + # place to put optional fields: all is optional for all, otherwise you need to tell it data_tier + + optional = { + "all":["core.events","dune.daq_test"], + "root-tuple":["core.event_count","core.first_event_number","core.last_event_number"], + "raw":["dune.config_file", "dune_mc.gen_fcl_filename","dune_mc.geometry_version","core.application.family","core.application.name","core.application.version"], + "binary-raw":["dune.config_file", "dune_mc.gen_fcl_filename","dune_mc.geometry_version","core.application.family","core.application.name","core.application.version"], + "trigprim":["dune.config_file", "dune_mc.gen_fcl_filename","dune_mc.geometry_version","core.application.family","core.application.name","core.application.version"], + "root-tuple-virtual":["core.event_count","core.first_event_number","core.last_event_number"] + } + + + did = filemd["namespace"]+":"+filemd["name"] + + # do this as file may not have an fid yet, but fid makes shorter error messages. + if "fid" in filemd: + fid = filemd["fid"] + else: + fid = did + + # start out with valid and no fixes needed + valid = True + fixes = {} + + # loop over default md keys + + for x, xtype in config["basetypes"].items(): + if DEBUG: print (x,xtype) + if x in optional["all"]: continue + # check required + if x not in filemd.keys(): + error = x+" is missing from "+ fid + "\n" + print (error) + if errfile is not None: errfile.write(error) + valid *= False + print (filemd.keys()) + + # check type + if x != "metadata" and valuetypes[xtype] != type(filemd[x]) : + #print (x,xtype) + if xtype == valuetypes["FLOAT"] and type(filemd[x]) == valuetypes["INT"]: continue + error = "top level item %s has wrong type in %s \n"%(x,fid) + print (error) + if errfile is not None: errfile.write(error) + valid *= False + + # now do the metadata + if DEBUG: + print ("keys",filemd.keys()) + + if "metadata" not in filemd.keys(): + print ("strange - no metadata for this file") + + md = filemd["metadata"] + + for x, xtype in config["basetypes"]["metadata"].items(): + if DEBUG: print ("checking", x,xtype) + if x in optional["all"]: continue # skip optional items + if "core.run_type" in md and md["core.run_type"] != "mc" and "mc" in x: + if verbose: print ("skipping mc only",x) + continue + + # check required keys + if x not in md.keys(): + if "core.data_tier" in md and md["core.data_tier"] in optional and x in optional[md["core.data_tier"]]: # skip optional items by data_tier + + if verbose: print ("skipping optional missing field for data_tier",md["core.data_tier"],x) + continue + error = x+ " is missing from " + fid + "\n" + print (error) + if errfile is not None: errfile.write(error) + valid *= False + if x in fixDefaults: + fixes[x]=fixDefaults[x] + continue + # check for type + if DEBUG: print ("xtype",xtype) + if valuetypes[xtype] != type(md[x]): + if xtype == "FLOAT" and type(md[x]) == valuetypes["INT"]: continue + error = "%s has wrong type in %s\n "%(x,fid) + print (error) + if errfile is not None: errfile.write(error+"\n") + valid *= False + for x,core in config["known_fields"].items(): + if x not in md: + print ("required field",x,"not present") + valid *=False + continue + if md[x] not in core: + print ("unknown required metadata field",x,"=",md[x]) + valid *= False + if not valid: + print (did, " fails basic metadata tests") + if len(fixes) !=0: + print ("you could fix this by applying this fix") + print (json.dumps(fixes,indent=4)) + + # look for upper case in keys + + for x,v in md.items(): + if x != x.lower(): + print ("OOPS upper case",x) + + + + + + return valid, fixes + + +if __name__ == '__main__': + + if len(sys.argv) < 2: + print ("please provide a json file to check") + sys.exit(1) + jsonname = sys.argv[1] + if not os.path.exists(jsonname): + print ("input file does not exist",jsonname) + sys.exit(1) + jsonfile = open(jsonname,'r') + filemd = json.load(jsonfile) + errfile = open(jsonname+".err",'w') + status,fixes = TypeChecker(filemd=filemd,errfile=errfile,verbose=True) + errfile.close() \ No newline at end of file diff --git a/_includes/extractor_new.py b/_includes/extractor_new.py new file mode 100755 index 0000000..257c5dd --- /dev/null +++ b/_includes/extractor_new.py @@ -0,0 +1,490 @@ +#!/usr/bin/env python +import sys, getopt +import os +from subprocess import Popen, PIPE +import threading +import queue +import json +import abc +import datetime + +DEBUG=True + +import argparse + +from metacat.webapi import MetaCatClient + +mc_client = MetaCatClient(os.getenv("METACAT_SERVER_URL")) + + +# Function to wait for a subprocess to finish and fetch return code, +# standard output, and standard error. +# Call this function like this: +# +# q = Queue.Queue() +# jobinfo = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) +# wait_for_subprocess(jobinfo, q) +# rc = q.get() # Return code. +# jobout = q.get() # Standard output +# joberr = q.get() # Standard error + +"""extractor_new.py +Purpose: To extract metadata from output file on worker node, generate JSON file +""" + + +class MetaData(object): + """Base class to hold / interpret general metadata""" + __metaclass__ = abc.ABCMeta + + @abc.abstractmethod + def __init__(self, inputfile): + self.inputfile = inputfile + + def extract_metadata_to_pipe(self): + """Extract metadata from inputfile into a pipe for further processing.""" + local = self.inputfile + if len(local) > 0: + proc = Popen(["sam_metadata_dumper", local], stdout=PIPE, + stderr=PIPE) + else: + url = self.inputfile + proc = Popen(["sam_metadata_dumper", url], stdout=PIPE, + stderr=PIPE) + if len(local) > 0 and local != self.inputfile: + os.remove(local) + return proc + + def get_job(self, proc): + """Run the proc in a 60-sec timeout queue, return stdout, stderr""" + q = queue.Queue() + thread = threading.Thread(target=self.wait_for_subprocess, args=[proc, q]) + thread.start() + thread.join(timeout=7200) + if thread.is_alive(): + print('Terminating subprocess because of timeout.') + proc.terminate() + thread.join() + rc = q.get() + jobout = q.get() + joberr = q.get() + if rc != 0: + raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc)) + return jobout, joberr + + @staticmethod + def wait_for_subprocess(jobinfo, q): + """Run jobinfo, put the return code, stdout, and stderr into a queue""" + jobout, joberr = jobinfo.communicate() + rc = jobinfo.poll() + for item in (rc, jobout, joberr): + q.put(item) + return + + @staticmethod + + def mdart_gen(jobtuple): + """Take Jobout and Joberr (in jobtuple) and return mdart object from that""" +### mdtext = ''.join(line.replace(", ,", ",") for line in jobtuple[0].split('\n') if line[-3:-1] != ' ,') + mdtext = ''.join(line.replace(", ,", ",") for line in jobtuple[0].decode().split('\n') if line[-3:-1] != ' ,') + mdtop = json.JSONDecoder().decode(mdtext) + if len(list(mdtop.keys())) == 0: + print('No top-level key in extracted metadata.') + sys.exit(1) + file_name = list(mdtop.keys())[0] + return mdtop[file_name] + + @staticmethod + def md_handle_application(md): + """If there's no application key in md dict, create the key with a blank dictionary. + Then return md['application'], along with mdval""" + if 'application' not in md: + md['application'] = {} + return md['application'] + + + +class MetaDataKey: + + def __init__(self): + self.expname = '' + + def metadataList(self): + return [self.expname + elt for elt in ('lbneMCGenerators','lbneMCName','lbneMCDetectorType','StageName')] + + def translateKey(self, key): + if key == 'lbneMCDetectorType': + return 'lbne_MC.detector_type' + elif key == 'StageName': + return 'lbne_MC.miscellaneous' + else: + prefix = key[:4] + stem = key[4:] + projNoun = stem.split("MC") + return prefix + "_MC." + projNoun[1] + + + +class expMetaData(MetaData): + """Class to hold/interpret experiment-specific metadata""" + def __init__(self, expname, inputfile): + MetaData.__init__(self, inputfile) + self.expname = expname + #self.exp_md_keyfile = expname + '_metadata_key' +# try: +# #translateMetaData = __import__("experiment_utilities", "MetaDataKey") +# from experiment_utilities import MetaDataKey +# except ImportError: +# print("You have not defined an experiment-specific metadata and key-translating module in experiment_utilities. Exiting") +# raise +# + metaDataModule = MetaDataKey() + self.metadataList, self.translateKeyf = metaDataModule.metadataList(), metaDataModule.translateKey + + def translateKey(self, key): + """Returns the output of the imported translateKey function (as translateKeyf) called on key""" + return self.translateKeyf(key) + + def md_gen(self, mdart, md0={}): + """Loop through art metdata, generate metadata dictionary""" + # define an empty python dictionary which will hold sam metadata. + # Some fields can be copied directly from art metadata to sam metadata. + # Other fields require conversion. + md = {} + topmd = {} + + # Loop over art metadata. + if DEBUG: print ("EXTRACTOR: art",mdart.keys()) + for mdkey in list(mdart.keys()): + mdval = mdart[mdkey] + # Skip some art-specific fields. + if mdkey == 'file_format_version': + pass + elif mdkey == 'file_format_era': + pass + + # Ignore primary run_type field (if any). + # Instead, get run_type from runs field. + + # #HMS elif mdkey == 'run_type': + # # pass + # elif mdkey == 'application.version': + # pass + # elif mdkey == 'application.family': + # pass + # elif mdkey == 'application.name': + # pass + + # do not Ignore data_stream any longer. + + elif mdkey == 'data_stream': + if 'dunemeta.data_stream' not in list(mdart.keys()): # only use this data_stream value if dunemeta.data_stream is not present + md['core.data_stream'] = mdval + + # Ignore process_name as of 2018-09-22 because it is not in SAM yet + elif mdkey == 'art.process_name': +# md['core.application.name'] = mdval + pass + # Application family/name/version. + + elif mdkey == 'applicationFamily': + md['core. application.family'] = mdval + elif mdkey == 'StageName' or mdkey == 'applicationName': + md['core.application.name'] = mdval + elif mdkey == 'applicationVersion': + md['core.application.version'] = mdval + + # patch time format + + elif mdkey in ("start_time", "end_time"): + + newk = "core."+mdkey + t = mdval + if t is not None: + t = datetime.datetime.fromisoformat(t).replace( + tzinfo=datetime.timezone.utc).timestamp() + md[newk] = t + print ("EXTRACTOR: fix time for",mdval,newk,t,md[newk]) + + # Parents. + + elif mdkey == 'parents': + mdparents = [] + if not args.strip_parents: + for parent in mdval: + parent_dict = {'name': parent,'namespace':'unknown'} + mdparents.append(parent_dict) + topmd['parents'] = mdparents + + # Other fields where the key or value requires minor conversion. + elif mdkey == 'runs': + runsSubruns = set() + runs = set() + for run, subrun in mdart.pop("runs", []): + runs.add(run) + runsSubruns.add(100000 * run + subrun) + md['core.runs'] = runs + md['core.runs_subruns'] = runsSubruns + + elif mdkey == 'art.first_event': + md[mdkey] = mdval[2] + elif mdkey == 'art.last_event': + md[mdkey] = mdval[2] + elif mdkey == 'first_event': + md['core.'+mdkey+ "_number"] = mdval + elif mdkey == 'last_event': + md['core.'+mdkey+ "_number"] = mdval + elif mdkey == 'detector.hv_status': + md[mdkey] = mdval + elif mdkey == 'detector.hv_value': + md[mdkey] = mdval + elif mdkey == 'detector.tpc_status': + md[mdkey] = mdval + elif mdkey == 'detector.tpc_apa_status': + md[mdkey] = mdval + elif mdkey == 'detector.tpc_apas': + md[mdkey] = mdval + elif mdkey == 'detector.tpc_apa_1': + md[mdkey] = mdval + elif mdkey == 'detector.tpc_apa_2': + md[mdkey] = mdval + elif mdkey == 'detector.tpc_apa_3': + md[mdkey] = mdval + elif mdkey == 'detector.tpc_apa_4': + md[mdkey] = mdval + elif mdkey == 'detector.tpc_apa_5': + md[mdkey] = mdval + elif mdkey == 'detector.tpc_apa_6': + md[mdkey] = mdval + elif mdkey == 'detector.pd_status': + md[mdkey] = mdval + elif mdkey == 'detector.crt_status': + md[mdkey] = mdval + elif mdkey == 'daq.readout': + md[mdkey] = mdval + elif mdkey == 'daq.felix_status': + md[mdkey] = mdval + elif mdkey == 'beam.polarity': + md[mdkey] = mdval + elif mdkey == 'beam.momentum': + md[mdkey] = mdval + elif mdkey == 'dunemeta.data_stream': + md['core.data_stream'] = mdval + elif mdkey == 'file_type': + md['core.'+mdkey] = mdval + elif mdkey == 'data_quality.level': + md[mdkey] = mdval + elif mdkey == 'data_quality.is_junk': + md[mdkey] = mdval + elif mdkey == 'data_quality.do_not_process': + md[mdkey] = mdval + elif mdkey == 'data_quality.online_good_run_list': + md[mdkey] = mdval + elif mdkey == 'dunemeta.dune_data.accouple': + md['dune_data.accouple'] = int(mdval) + elif mdkey == 'dunemeta.dune_data.calibpulsemode': + md['dune_data.calibpulsemode'] = int(mdval) + elif mdkey == 'dunemeta.dune_data.daqconfigname': + md['dune_data.DAQConfigName'] = mdval + elif mdkey == 'dunemeta.dune_data.detector_config': + md['dune_data.detector_config'] = mdval + elif mdkey == 'dunemeta.dune_data.febaselinehigh': + md['dune_data.febaselinehigh'] = int(mdval) + elif mdkey == 'dunemeta.dune_data.fegain': + md['dune_data.fegain'] = int(mdval) + elif mdkey == 'dunemeta.dune_data.feleak10x': + md['dune_data.feleak10x'] = int(mdval) + elif mdkey == 'dunemeta.dune_data.feleakhigh': + md['dune_data.feleakhigh'] = int(mdval) + elif mdkey == 'dunemeta.dune_data.feshapingtime': + md['dune_data.feshapingtime'] = int(mdval) + elif mdkey == 'dunemeta.dune_data.inconsistent_hw_config': + md['dune_data.inconsistent_hw_config'] = int(mdval) + elif mdkey == 'dunemeta.dune_data.is_fake_data': + md['dune_data.is_fake_data'] = int(mdval) + elif mdkey == 'dunemeta.dune_data.readout_window': + md['dune_data.readout_window'] = float(mdval) + + # For all other keys, copy art metadata directly to sam metadata. + # This works for run-tuple (run, subrun, runtype) and time stamps. + + else: + if 'art' not in mdkey: + md['core.'+mdkey] = mdart[mdkey] + + # Make the other meta data field parameters + topmd['created_by'] = os.environ['USERF'] + topmd['name'] = self.inputfile.split("/")[-1] + if 'file_size' in md0: + topmd['size'] = md0['file_size'] + else: + topmd['size'] = os.path.getsize(self.inputfile) + # if 'crc' in md0 and not args.no_crc: + # topmd['crc'] = md0['crc'] + # elif not args.no_crc: + # topmdmd['crc'] = root_metadata.fileEnstoreChecksum(self.inputfile) + + # In case we ever want to check out what md is for any instance of MetaData by calling instance.md + topmd['metadata'] = md + self.topmd = topmd + return self.topmd + + def getmetadata(self, md0={}): + """ Get metadata from input file and return as python dictionary. + Calls other methods in class and returns metadata dictionary""" + proc = self.extract_metadata_to_pipe() + jobt = self.get_job(proc) + mdart = self.mdart_gen(jobt) + return self.md_gen(mdart, md0) + +def main(): + + argparser = argparse.ArgumentParser('Parse arguments') + argparser.add_argument('--infile',help='path to input file',required=True,type=str) + argparser.add_argument('--declare',help='validate and declare the metadata for the file specified in --infile to SAM',action='store_true') + argparser.add_argument('--appname',help='application name for metadata',type=str) + argparser.add_argument('--appversion',help='application version for metadata',type=str) + argparser.add_argument('--appfamily',help='application family for metadata',type=str) + argparser.add_argument('--file_type',help='file_type (mc or detector)',type=str) + argparser.add_argument('--file_format',help='file_format (root, artroot ..)',type=str,required=True) + argparser.add_argument('--run_type',help='run_type - (fardet-hd, iceberg ...)',type=str) + argparser.add_argument('--campaign',help='Value for dune.campaign for metadata',type=str) + argparser.add_argument('--data_stream',help='Value for data_stream for metadata',type=str) + argparser.add_argument('--data_tier',help='Value for data_tier for metadata',type=str,required=True) + argparser.add_argument('--fcl_file',type=str,help="fcl file name",required=True) + argparser.add_argument('--requestid',help='Value for dune.requestid for metadata',type=str) + #argparser.add_argument('--set_processed',help='Set for parent file as processed in metadata',action="store_true") + argparser.add_argument('--strip_parents',help='Do not include the file\'s parents in metadata for declaration',action="store_true") + argparser.add_argument('--no_crc',help='Leave the crc out of the generated json',action="store_true") + argparser.add_argument('--skip_dumper',help='Skip running sam_metadata_dumper on the input file',action="store_true") + argparser.add_argument('--input_json',help='Input json file containing metadata to be added to output (can contain ANY valid metacat metadata parameters)',type=str) + argparser.add_argument('--inputDidsFile', type=str, default=None, + help='Optional path to a file containing all input DIDs, one per line') + argparser.add_argument('--no_extract',help='use this if not artroot',action="store_true") + argparser.add_argument('--namespace',type=str,help="namespace for output file",required=True) + global args + args = argparser.parse_args() + + try: +# expSpecificMetadata = expMetaData(os.environ['SAM_EXPERIMENT'], str(sys.argv[1])) + expSpecificMetadata = expMetaData('dune', args.infile) + if not args.no_extract: + mddict = expSpecificMetadata.getmetadata() + else: + mddict = {} + mddict['name']=os.path.basename(args.infile) + + mddict['metadata']={} + print ("EXTRACTOR: building metadata from parent and args as no artroot dump available") + # If --input_json is supplied, open that dict now and add it to the output json + if args.input_json != None: + if os.path.exists(args.input_json): + try: + arbjson = json.load(open(args.input_json,'r')) + #print ("EXTRACTOR: arbjson",arbjson) + arbjson.pop('name') + arbjson.pop('namespace') + for key in list(arbjson.keys()): + mddict[key] = arbjson[key] + except: + print('Error loading input json file.',args.input_json) + + else: + print('warning, could not open the input json file', args.input_json) + + + if args.appname != None: + mddict['metadata']['core.application.name'] = args.appname + if args.appversion != None: + mddict['metadata']['core.application.version'] = args.appversion + if args.appfamily != None: + mddict['metadata']['core.application.family'] = args.appfamily + if args.campaign != None: + mddict['metadata']['dune.campaign'] = args.campaign + if args.data_stream != None: + mddict['metadata']['core.data_stream'] = args.data_stream + if args.data_tier != None: + mddict['metadata']['core.data_tier'] = args.data_tier + if args.file_type != None: + mddict['metadata']['core.file_type'] = args.file_type + if args.file_format != None: + mddict['metadata']['core.file_format'] = args.file_format + if args.run_type != None: + mddict['metadata']['core.run_type'] = args.run_type + if args.requestid != None: + mddict['metadata']['dune.requestid'] = args.requestid + if args.inputDidsFile is not None: + parentDids = [] + for line in open(args.inputDidsFile, "r").read().splitlines(): + ns = line.split(':')[0] + name = line.split(':')[1] + if DEBUG: print ("EXTRACTOR: found a parent",line) + parentDids.append({ "name" : name, "namespace": ns }) + print ("EXTRACTOR: overriding parents with dids from " + args.inputDidsFile, file=sys.stdout) + mddict["parents"] = parentDids + + if args.fcl_file is not None: + mddict["metadata"]["dune.config_file"]=os.path.basename(args.fcl_file) + + mddict["metadata"]["retention.class"]="user" + mddict["metadata"]["retention.status"]="active" + mddict["metadata"]["core.file_content_status"]="good" + # get info from the parent file if possible + # force some items to be like parents + # and replace those that are missing from parents + + force = ['core.data_stream', 'core.run_type', 'core.file_type'] + inheritable = ['core.runs','core.runs_subruns'] + if 'parents' in mddict and len(mddict['parents']) > 0: + theparent = mddict['parents'][0] + thename = theparent['name'] + thenamespace = theparent['namespace'] + thedid = "%s:%s"%(thenamespace,thename) + try: + parentmd = mc_client.get_file(name=thename,namespace=thenamespace,with_metadata=True) + except: + print('Error retrieving parent metadata for did %s:%s' % (thenamespace,thename)) + parentmd = None + if parentmd is not None: + for key in force: # these need to be the same as parents + if key in parentmd['metadata'] : + mddict['metadata'][key] = parentmd['metadata'][key] + print ("EXTRACTOR: forcing " + key + " from parent file " + thedid ) + for key in inheritable: + if key in parentmd['metadata'] and key not in mddict['metadata']: + mddict['metadata'][key] = parentmd['metadata'][key] + print ("EXTRACTOR: inheriting " + key + " from parent file " + thedid) + print ("EXTRACTOR: setting namespace for output",args.namespace) + mddict['namespace']=args.namespace + except TypeError: + print('You have not implemented a defineMetaData function by providing an experiment.') + print('No metadata keys will be saved') + raise +# mdtext = json.dumps(expSpecificMetadata.getmetadata(), indent=2, sort_keys=True) + mdtext = json.dumps(mddict, indent=2, sort_keys=True) + + # if args.declare: + # ih.declareFile(mdtext) + + # if args.set_processed: + # swc = mc_client() + # moddict = {"DUNE.production_status" : "processed" } + # for parent in moddict['parents']: + # fname = moddict['parents'][parent]['file_name'] + # try: + # swc.modifyFileMetadata(fname, moddict) + # except: + # print('Error modidying metadata for %s' % fname) + # raise + of = open(mddict["name"]+".json",'w') + json.dump(mddict,of,indent=4, sort_keys=True) + of.close() + #print(mdtext) + sys.exit(0) + + + +if __name__ == "__main__": + main() + + diff --git a/_includes/job_config.sh b/_includes/job_config.sh new file mode 100644 index 0000000..db42fb9 --- /dev/null +++ b/_includes/job_config.sh @@ -0,0 +1,10 @@ +export FCL_FILE="run_analyseEvents.fcl" # fcl file +export OUTPUT_DATA_TIER1="full-reconstructed" # tier for artroot output +export OUTPUT_DATA_TIER2="root-tuple" # tier for root output +export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 2 ordered " # metacat query for files +export APP_NAME="ana" # application name +export DESCRIPTION="$APP_NAME using $FCL_FILE" # appears as jobname in justin +export USERF=${USER} # make certain the grid knows who your are +export NUM_EVENTS=-1 # process them all +export FNALURL='https://fndcadoor.fnal.gov:2880/dune/scratch/users' # sends output to scratch +export NAMESPACE="usertests" # don't change this unless doing production diff --git a/_includes/makercds.sh b/_includes/makercds.sh old mode 100644 new mode 100755 diff --git a/_includes/maketar.sh b/_includes/maketar.sh old mode 100644 new mode 100755 index ef7541d..d56bd03 --- a/_includes/maketar.sh +++ b/_includes/maketar.sh @@ -9,6 +9,7 @@ date echo " make tar file" tar --exclude '.git' --exclude build_slf7.x86_64 -cf $THERE/$1.tar $1 date +echo " gzip step " gzip -f $THERE/$1.tar date echo " tar file is at $THERE/$1.tar.gz" diff --git a/_includes/sam2metacat.py b/_includes/sam2metacat.py new file mode 100644 index 0000000..ddeced0 --- /dev/null +++ b/_includes/sam2metacat.py @@ -0,0 +1,195 @@ + #!/usr/bin/env python3 + # + # Convert protoDUNE metadata from extractor_prod.py into JSON suitable to be + # the value of the "metadata" key in the JSON sent to MetaCat + # + # THIS IS NOT REALLY PART OF justIN AND IT SHOULD BE INCORPORATED INTO THE + # SCRIPTS ASSOCIATED WITH APPLICATIONS, LIKE extractor_prod.py ! + # + # Adapted from + # https://github.com/ivmfnal/protodune/blob/main/tools/declare_meta.py and + # https://github.com/ivmfnal/protodune/blob/main/declad/mover.py + # to write out modified JSON rather than uploading it. This allows the script + # to be run inside jobscripts supplied by users. + # + # A version can be found in the jobutils area of the justIN UPS product + # in cvmfs + # + +import sys, json +import datetime +import argparse + +def getCoreAttribute(): + coreAttributes = { + "event_count": "core.event_count", + "file_type" : "core.file_type", + "file_format": "core.file_format", + "data_tier" : "core.data_tier", + "data_stream": "core.data_stream", + "events" : "core.events", + "first_event": "core.first_event_number", + "last_event" : "core.last_event_number", + "event_count": "core.event_count", + "user" : None, + "group" : None + } + return coreAttributes + +def doit(inputMetadataFile, allInputDidsFile=None, namespace="usertests", outputFormat=None, outputTier=None ): + coreAttributes = getCoreAttribute() + + try: + inputMetadata = json.load(open(inputMetadataFile, "r")) + except Exception as e: + print("Error reading metadata from file: " + str(e), file=sys.stderr) + sys.exit(1) + + allInputDids = [] + if allInputDidsFile is not None: + try: + for line in open(allInputDidsFile, "r").read().splitlines(): + allInputDids.append(line) + except Exception as e: + print("Error read all input DIDs file: " + str(e), file=sys.stderr) + sys.exit(2) + + + inputMetadata.pop("file_size", None) + inputMetadata.pop("checksum", None) + inputMetadata.pop("file_name", None) + + + + # Most of the metadata goes in "metadata" within the outer dictionary + outputMetadata = { "metadata": {}} + + runsSubruns = set() + runType = None + runs = set() + for run, subrun, rtype in inputMetadata.pop("runs", []): + runType = rtype + runs.add(run) + runsSubruns.add(100000 * run + subrun) + + outputMetadata["metadata"]["core.runs_subruns"] = sorted(list(runsSubruns)) + outputMetadata["metadata"]["core.runs"] = sorted(list(runs)) + outputMetadata["metadata"]["core.run_type"] = runType + + app = inputMetadata.pop("application", None) + if app: + if "name" in app: + outputMetadata["metadata"]["core.application.name"] = app["name"] + if "version" in app: + outputMetadata["metadata"]["core.application.version"] = app["version"] + if "family" in app: + outputMetadata["metadata"]["core.application.family"] = app["family"] + if "family" in app and "name" in app: + outputMetadata["metadata"]["core.application"] = \ + app["family"] + "." + app["name"] + + for k in ("start_time", "end_time"): + t = inputMetadata.pop(k, None) + if t is not None: + t = datetime.datetime.fromisoformat(t).replace( + tzinfo=datetime.timezone.utc).timestamp() + outputMetadata["metadata"]["core." + k] = t + + for name, value in inputMetadata.items(): + + + if name == 'parents': + parentDids = [] + for parent in value: + matchingDid = None + for did in allInputDids: + if did.endswith(parent["file_name"]): + matchingDid = did + break + name = did.split(':')[1] + namespace = did.split(':')[0] + if name != 'noparents.root': + outputMetadata["parents"] = [ + {"name": name, + "namespace": namespace + } + ] + + # if matchingDid: + # parentDids.append({ "did" : matchingDid }) + # else: + # print("No matching input DID for file %s with parent file_name %s- exiting" + # % (str(parent), str(parent["file_name"])), + # file=sys.stderr) + # sys.exit(3) + + # Add the list of { "did": "..." } dictionaries to top level + + + + # outputMetadata["parents"] = parentDids + + + + else: + if '.' in name: + outputMetadata["metadata"][name] = value + else: + if name in coreAttributes: + if coreAttributes[name]: + outputMetadata["metadata"][coreAttributes[name]] = value + else: + outputMetadata["metadata"]['x.' + name] = value + print ("setting " + name + " to " + str(value)) + +# patches added by HMS to make this more general + if namespace != None: + outputMetadata['namespace'] = namespace + print ("overriding namespace to " + namespace, file=sys.stdout) + +# add the parentage if not inherited + if allInputDidsFile is not None and outputMetadata.get("parents", None) is None: + parentDids = [] + for line in open(allInputDidsFile, "r").read().splitlines(): + ns = line.split(':')[0] + name = line.split(':')[1] + parentDids.append({ "name" : name, "namespace": ns }) + outputMetadata["parents"] = parentDids + + if outputFormat != None: + outputMetadata["metadata"]["core.file_format"] = outputFormat + print ("overriding output format to " + outputFormat, file=sys.stdout) + + if outputTier != None: + outputMetadata["metadata"]["core.data_tier"] = outputTier + print ("overriding output data_tier to " + outputTier, file=sys.stdout) + + + outputMetadata["metadata"].setdefault("core.event_count", + len(outputMetadata["metadata"].get("core.events", []))) + + + json.dump(outputMetadata, sys.stdout, indent=4, sort_keys=True) + + outname = inputMetadataFile.replace(".ext.json",".json") + with open(outname, "w") as f: + json.dump(outputMetadata, f, indent=4, sort_keys=True) + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description='Command line interface for merge_utils') + + parser.add_argument('--inputMetadataFile', type=str, help='Path to the input metadata JSON file') + + parser.add_argument('--namespace', type=str, help='Namespace to use for the metadata', default="usertests") + parser.add_argument('--outputFormat', type=str, default=None, + help='Output format (default: inherits from input)') + parser.add_argument('--outputTier', default=None, help='Output data tier (default: inherits)') + parser.add_argument('--InputDidsFile', type=str, default=None, + help='Optional path to a file containing all input DIDs, one per line') + args = parser.parse_args() + doit(inputMetadataFile=args.inputMetadataFile, + allInputDidsFile=args.allInputDidsFile, + namespace=args.namespace, + outputFormat=args.outputFormat, + outputTier=args.outputTier) \ No newline at end of file diff --git a/_includes/setup_before_submit.sh b/_includes/setup_before_submit.sh old mode 100644 new mode 100755 index ace14f3..f0efb3c --- a/_includes/setup_before_submit.sh +++ b/_includes/setup_before_submit.sh @@ -5,18 +5,14 @@ export DIRECTORY="$(basename "${PWD}")" export DUNE_VERSION=v09_91_02d01 export DUNE_QUALIFIER=e26:prof export DUNE_QUALIFIER_STRING=`echo ${DUNE_QUALIFIER} | tr : _` -export FCL_FILE="run_analyseEvents.fcl" -export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 5 ordered " -export PROCESS_TYPE="ana" -export DESCRIPTION="$PROCESS_TYPE using $FCL_FILE" -export USERF=${USER} -export NUM_EVENTS=-1 # process them all -export FNALURL='https://fndcadoor.fnal.gov:2880/dune/scratch/users' ## set up locally just as a check source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +setup metacat +export METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app +export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune setup justin echo " code set up" justin time @@ -32,5 +28,6 @@ cp setup-grid $localProductsdir/setup-grid echo "Now you should:" echo "./maketar.sh $DIRECTORY" echo "./makercds.sh $DIRECTORY" +echo "# edit job_config.sh" echo "./submit_workflow.sh" echo "----------------------------------------------------------------" \ No newline at end of file diff --git a/_includes/submit_local_code.jobscript.sh b/_includes/submit_local_code.jobscript.sh old mode 100644 new mode 100755 index e6e3565..467ed82 --- a/_includes/submit_local_code.jobscript.sh +++ b/_includes/submit_local_code.jobscript.sh @@ -13,35 +13,37 @@ export INPUT_TAR_DIR_LOCAL= export MQL= export DIRECTORY= +(see jobs_config.sh for the full list) + Use this command to create the workflow: justin simple-workflow \ --mql "$MQL" \ --jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${FNALURL}/${USERF}" --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 - + --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_NAME=${APP_NAME} --env DIRECTORY=${DIRECTORY} --scope $NAMESPACE --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" -The following optional environment variables can be set when creating the -workflow/stage: FCL_FILE, PROCESS_TYPE, NUM_EVENTS, DUNE_VERSION, DUNE_QUALIFIER +see job_config.sh for explanations EOF # fcl file and DUNE software version/qualifier to be used FCL_FILE=${FCL_FILE:-${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/my_code/fcls/my_reco.fcl} -PROCESS_TYPE=${PROCESS_TYPE:-reco2} +APP_NAME=${APP_NAME:-unknown} #DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} #DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} +echo "------ set things up -------" echo "Check environment" echo "DIRECTORY=$DIRECTORY" echo "DUNE_VERSION=$DUNE_VERSION" echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" echo "FCL_FILE=$FCL_FILE" echo "MQL=$MQL" -echo "PROCESS_TYPE=$PROCESS_TYPE" +echo "APP_NAME=$APP_NAME" echo "USERF=$USERF" echo "NUM_EVENTS=$NUM_EVENTS" echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" +echo "NAMESPACE=$NAMESPACE" @@ -56,6 +58,7 @@ fi # First get an unprocessed file from this stage did_pfn_rse=`$JUSTIN_PATH/justin-get-file` + if [ "$did_pfn_rse" = "" ] ; then echo "Nothing to process - exit jobscript" exit 0 @@ -66,6 +69,8 @@ echo "$did_pfn_rse" | cut -f1 -d' ' >>all-input-dids.txt # pfn is also needed when creating justin-processed-pfns.txt pfn=`echo $did_pfn_rse | cut -f2 -d' '` +did=`echo $did_pfn_rse | cut -f1 -d' '` + echo "Input PFN = $pfn" echo "TARDIR ${INPUT_TAR_DIR_LOCAL}" @@ -88,42 +93,82 @@ export PRODUCTS="${localProductsdir}/:$PRODUCTS" # Then we can set up our local products setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" + +setup metacat +export METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app +export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune + source ${localProductsdir}/setup-grid mrbslp +#echo "----- code is set up -----" + # Construct outFile from input $pfn now=$(date -u +"%Y%m%d%H%M%SZ") Ffname=`echo $pfn | awk -F/ '{print $NF}'` fname=`echo $Ffname | awk -F. '{print $1}'` # outFile1 is artroot format # outFile2 is root format for analysis -outFile1=${fname}_${PROCESS_TYPE}_${now}.root -outFile2=${fname}_${PROCESS_TYPE}_tuple_${now}.root - +export outFile1=${fname}_${APP_NAME}_${now}.root +export outFile2=${fname}_${APP_NAME}_tuple_${now}.root +# echo "make $outFile1" campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" # Here is where the LArSoft command is call it ( # Do the scary preload stuff in a subshell! export LD_PRELOAD=${XROOTD_LIB}/libXrdPosixPreload.so -echo "$LD_PRELOAD" +# echo "$LD_PRELOAD" -echo "now run lar" -lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o $outFile1 -T ${outFile2} "$pfn" > ${fname}_${PROCESS_TYPE}_${now}.log 2>&1 + +#sam_metadata_dumper $pfn + +echo "----- now run lar ------" + +echo "lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_NAME}_${now}.log 2>&1" + +lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_NAME}_${now}.log 2>&1 ) +larExit=$? +# Subshell exits with exit code of last command +echo "lar exit code $larExit" + echo '=== Start last 1000 lines of lar log file ===' -tail -1000 ${fname}_${PROCESS_TYPE}_${now}.log +tail -1000 ${fname}_${APP_NAME}_${now}.log echo '=== End last 1000 lines of lar log file ===' -# Subshell exits with exit code of last command -larExit=$? -echo "lar exit code $larExit" +echo "$did" > justin-input-dids.txt + +echo "--------make metadata---------" + +#sam_metadata_dumper ${outFile1} + +echo "python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=${outFile1} --appversion=$DUNE_VERSION --appname=${APP_NAME} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} # > $outFile1.json" + +python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile1 --appversion=$DUNE_VERSION --appname=${APP_NAME} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} #> $outFile1.json + +file1Exit=$? + +#cat ${outFile1}.json + +echo "------------ non-artroot metadata -----------" +# here for non-artroot files, salvage what you can from outFile1 + +oldjson=${outFile1}.json + +echo " python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_NAME} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json" + +python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_NAME} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json + +file2Exit=$? + +echo "------- finish up ------" if [ $larExit -eq 0 ] ; then # Success ! echo "$pfn" > justin-processed-pfns.txt diff --git a/_includes/submit_local_code_rucio.jobscript.sh b/_includes/submit_local_code_rucio.jobscript.sh new file mode 100755 index 0000000..9ab23ca --- /dev/null +++ b/_includes/submit_local_code_rucio.jobscript.sh @@ -0,0 +1,179 @@ +#!/bin/bash +:<<'EOF' + +To use this jobscript to process 5 files from the dataset fardet-hd__fd_mc_2023a_reco2__full-reconstructed__v09_81_00d02__standard_reco2_dune10kt_nu_1x2x6__prodgenie_nu_dune10kt_1x2x6__out1__validation +data and put the output logs in the `usertests` namespace and saves the output in /scratch + +Use these commands to set up ahead of time: + +export DUNE_VERSION= +export DUNE_QUALIFIER= +export FCL_FILE= +export INPUT_TAR_DIR_LOCAL= +export MQL= +export DIRECTORY= + +Use this command to create the workflow: + +justin simple-workflow \ +--mql "$MQL" \ +--jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ +--output-pattern "*.root:${USER}-output" --env APP_NAME=${APP_NAME} --env DIRECTORY=${DIRECTORY} --scope $NAMESPACE --lifetime 30 + + +The following optional environment variables can be set when creating the +workflow/stage: FCL_FILE, APP_NAME, NUM_EVENTS, DUNE_VERSION, DUNE_QUALIFIER + +EOF + +# fcl file and DUNE software version/qualifier to be used +FCL_FILE=${FCL_FILE:-${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/my_code/fcls/my_reco.fcl} +APP_NAME=${APP_NAME:-reco2} +#DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} +#DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} + +echo "Check environment" +echo "DIRECTORY=$DIRECTORY" +echo "DUNE_VERSION=$DUNE_VERSION" +echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" +echo "FCL_FILE=$FCL_FILE" +echo "MQL=$MQL" +echo "APP_NAME=$APP_NAME" +echo "USERF=$USERF" +echo "NUM_EVENTS=$NUM_EVENTS" +echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" + + + +echo "Current working directory is `pwd`" + + +# number of events to process from the input file +if [ "$NUM_EVENTS" != "" ] ; then + events_option="-n $NUM_EVENTS" +fi + +# First get an unprocessed file from this stage +did_pfn_rse=`$JUSTIN_PATH/justin-get-file` + +if [ "$did_pfn_rse" = "" ] ; then + echo "Nothing to process - exit jobscript" + exit 0 +fi + +# Keep a record of all input DIDs, for pdjson2meta file -> DID mapping +echo "$did_pfn_rse" | cut -f1 -d' ' >>all-input-dids.txt + +# pfn is also needed when creating justin-processed-pfns.txt +pfn=`echo $did_pfn_rse | cut -f2 -d' '` +echo "Input PFN = $pfn" + +echo "TARDIR ${INPUT_TAR_DIR_LOCAL}" +echo "CODE DIR ${DIRECTORY}" + +# Setup DUNE environment +localProductsdir=`ls -c1d ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts*` + +echo "localProductsdir ${localProductsdir}" + + +# seems to require the right name for the setup script + +echo " check that there is a setup in ${localProductsdir}" +ls -lrt ${localProductsdir}/setup-grid +ls -lrt ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE +source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh +export PRODUCTS="${localProductsdir}/:$PRODUCTS" + +# Then we can set up our local products +setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +source ${localProductsdir}/setup-grid +setup metacat +export METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app +export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune + +mrbslp + +# Construct outFile from input $pfn +now=$(date -u +"%Y%m%d%H%M%SZ") +Ffname=`echo $pfn | awk -F/ '{print $NF}'` +fname=`echo $Ffname | awk -F. '{print $1}'` +# outFile1 is artroot format +# outFile2 is root format for analysis +outFile1=${fname}_${APP_NAME}_${now}.root +outFile2=${fname}_${APP_NAME}_tuple_${now}.root + + +campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" + +# Here is where the LArSoft command is call it +( +# Do the scary preload stuff in a subshell! +export LD_PRELOAD=${XROOTD_LIB}/libXrdPosixPreload.so +echo "$LD_PRELOAD" + +echo "now run lar" + +lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_NAME}_${now}.log 2>&1 +) +# Subshell exits with exit code of last command +larExit=$? + +echo "lar exit code $larExit" + + +echo '=== Start last 1000 lines of lar log file ===' +tail -1000 ${fname}_${APP_NAME}_${now}.log +echo '=== End last 1000 lines of lar log file ===' + + +if [ $larExit -eq 0 ] ; then + # Success ! + echo "$pfn" > justin-processed-pfns.txt + jobscriptExit=0 +else + # Oh :( + jobscriptExit=1 +fi + +# make metadata for both files + +${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_prod.py --infile ${outFile1} --appfamily duneana --appname ${APP_NAME} --appversion ${DUNE_VERSION} --no_crc > ${outFile1}.ext.json + +file1Exit=$? + +${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_prod.py --infile ${outFile2} --appfamily larsoft --appname ${APP_NAME} --appversion ${DUNE_VERSION} --no_crc --input_json ${DIRECTORY}/pdvd_input.json > ${outFile1}.ext.json + +file2Exit=$? + +if [ $? -ne 0 ] ;then + ex_code=$((file1Exit+2*file2Exit)) + echo "ERROR: metadata extraction (reco) exit code: $ex_code" + files=`ls *_${now}_*` + for f in $files + do + size=`stat -c %s $f` + echo "written output file: $f $size" + done + fi +fi + +if [ $? -ne 0 ] +then + echo "Exiting with error" + exit 1 +else + files=`ls *_${now}*` + for f in $files + do + size=`stat -c %s $f` + echo "written output file: $f $size" + done + + echo "$pfn" > justin-processed-pfns.txt +fi + +# Create compressed tar file with all log files +tar zcf `echo "$JUSTIN_JOBSUB_ID.logs.tgz" | sed 's/@/_/g'` *.log +exit $jobscriptExit diff --git a/_includes/submit_workflow.sh b/_includes/submit_workflow.sh old mode 100644 new mode 100755 index 677d91e..139364a --- a/_includes/submit_workflow.sh +++ b/_includes/submit_workflow.sh @@ -1,20 +1,24 @@ # actual submission export INPUT_TAR_DIR_LOCAL=`cat cvmfs.location` +source job_config.sh # pick up the configuration + +echo "---- check the configuration ----" echo "DIRECTORY=$DIRECTORY" echo "DUNE_VERSION=$DUNE_VERSION" echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" echo "FCL_FILE=$FCL_FILE" echo "MQL=$MQL" -echo "PROCESS_TYPE=$PROCESS_TYPE" +echo "APP_NAME=$APP_NAME" echo "USERF=$USERF" echo "NUM_EVENTS=$NUM_EVENTS" echo "DESCRIPTION=$DESCRIPTION" echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" +echo "NAMESPACE=${NAMESPACE}" -echo "tardir $INPUT_TAR_DIR_LOCAL" +echo "---- do the submission ----" justin simple-workflow \ --mql "$MQL" \ --jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${FNALURL}/${USERF}" --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --description "${DESCRIPTION}" \ No newline at end of file + --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_NAME=${APP_NAME} --env DIRECTORY=${DIRECTORY} --scope ${NAMESPACE} --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" \ No newline at end of file diff --git a/_includes/submit_workflow_rucio.sh b/_includes/submit_workflow_rucio.sh new file mode 100755 index 0000000..e465d24 --- /dev/null +++ b/_includes/submit_workflow_rucio.sh @@ -0,0 +1,22 @@ +# actual submission - this one uses rucio as output location + +export INPUT_TAR_DIR_LOCAL=`cat cvmfs.location` + +echo "DIRECTORY=$DIRECTORY" +echo "DUNE_VERSION=$DUNE_VERSION" +echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" +echo "FCL_FILE=$FCL_FILE" +echo "MQL=$MQL" +echo "APP_NAME=$APP_NAME" +echo "USERF=$USERF" +echo "NUM_EVENTS=$NUM_EVENTS" +echo "DESCRIPTION=$DESCRIPTION" +echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" +echo "Output NAMESPACE=$NAMESPACE" + + +echo "tardir $INPUT_TAR_DIR_LOCAL" +justin simple-workflow \ +--mql "$MQL" \ +--jobscript submit_local_code_rucio.jobscript.sh --rss-mb 4000 \ + --output-pattern "*.root:${USER}-output" --env APP_NAME=${APP_NAME} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" \ No newline at end of file diff --git a/_includes/test_workflow.sh b/_includes/test_workflow.sh new file mode 100755 index 0000000..c9f1182 --- /dev/null +++ b/_includes/test_workflow.sh @@ -0,0 +1,26 @@ +# actual submission +#export INPUT_TAR_DIR_LOCAL=`cat cvmfs.location` +export INPUT_TAR_DIR_LOCAL=$DUNEDATA +source $DIRECTORY/job_config.sh + +export NUM_EVENTS=2 +echo "DIRECTORY=$DIRECTORY" +echo "DUNE_VERSION=$DUNE_VERSION" +echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" +echo "FCL_FILE=$FCL_FILE" +echo "MQL=$MQL" +echo "APP_NAME=$APP_NAME" +echo "USERF=$USERF" +echo "NUM_EVENTS=$NUM_EVENTS" +echo "DESCRIPTION=$DESCRIPTION" +echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" + + +echo "tardir $INPUT_TAR_DIR_LOCAL" +export HERE=$PWD +echo " go up one directory " +cd .. +justin-test-jobscript \ +--mql "$MQL" \ +--jobscript $DIRECTORY/submit_local_code.jobscript.sh --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USER} --env APP_NAME=${APP_NAME} --env NAMESPACE=${NAMESPACE} +cd $HERE From 764ee28e088c482d80e0dc9b5545ad163185d472 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 14 Feb 2026 14:00:49 -0800 Subject: [PATCH 11/27] make metadata --- _includes/job_config.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 _includes/job_config.sh diff --git a/_includes/job_config.sh b/_includes/job_config.sh old mode 100644 new mode 100755 From ec64ceb017915b0ee89de62022fbbee8a8611281 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 14 Feb 2026 14:40:55 -0800 Subject: [PATCH 12/27] fixed setup-grid --- _includes/setup-grid | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_includes/setup-grid b/_includes/setup-grid index d8c2f85..7b3fef1 100644 --- a/_includes/setup-grid +++ b/_includes/setup-grid @@ -114,7 +114,7 @@ echo echo MRB_PROJECT=$MRB_PROJECT echo MRB_PROJECT_VERSION=$MRB_PROJECT_VERSION echo MRB_QUALS=$MRB_QUALS -echo MRB_QUALSTRING=$DUNESW_QUALIFIER_STRING +echo MRB_QUALSTRING=$DUNE_QUALIFIER_STRING echo MRB_TOP=$MRB_TOP echo MRB_SOURCE=$MRB_SOURCE echo MRB_BUILDDIR=$MRB_BUILDDIR From 18c8a2dedd082c48f8bc95bf0f45170745a92404 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 14 Feb 2026 14:47:38 -0800 Subject: [PATCH 13/27] make a tarball --- _includes/gitadd.sh | 13 +++++++++++++ files/usefulcode.tar | Bin 0 -> 71680 bytes 2 files changed, 13 insertions(+) create mode 100644 _includes/gitadd.sh create mode 100644 files/usefulcode.tar diff --git a/_includes/gitadd.sh b/_includes/gitadd.sh new file mode 100644 index 0000000..b401f6d --- /dev/null +++ b/_includes/gitadd.sh @@ -0,0 +1,13 @@ +git add *.sh *.py setup-grid DUNE*json +tar cBf - \ +extractor_new.py \ +gitadd.sh \ +job_config.sh \ +makercds.sh \ +maketar.sh \ +setup_before_submit.sh \ +submit_local_code.jobscript.sh \ +setup-grid \ +submit_workflow.sh \ +> ../files/usefulcode.tar +git add ../files/usefulcode.tar diff --git a/files/usefulcode.tar b/files/usefulcode.tar new file mode 100644 index 0000000000000000000000000000000000000000..25e1b8073d3e923010a8233920abee7684d40200 GIT binary patch literal 71680 zcmeHwOKfAwdS1_kukE3OWN-rqh>^t^NTM@`n_ud2WP`s;sHRl#;&1YM^Zw7Yi8JuP(3?%evPSS&8g%~^N& z@BD)NTP)A2f00(4nVGXnbEVnY()_~AY{@E?=H`mUJ67@5dy)dM=Lb#~UGuBQZllq3 zQi@0tiq-2EWg?veDHhG3tV>FMyRyFjw|?o)_wL-;bgI_&p7n*LKTNyxHvTK)zu(1w z{QVdB?;4-w{rz2Cj5Yoc|K07YMrcg`l{Irk_76{1|Ln)_Jz>_J@t^+U zJOAhJ|K9Kag8Ul&xeNn4&WnexQ**o1!{Y;37niR8+0sJa`d=u|%-ykGT-Nd1%KJHA z|1(9a8F)>%Qd(Fjm5TGl(n0}TxHMlX-D}n(8kOZo z>&u^bZgKt7;-?FXN154sgRKr!inEg2qN<-6IKx}@5ehC}|8u2Mx!AY<=gRY?(f%J{ zaCqOw-kt9G-RVQGHSM;ZTb;AuxZTQl%}%=;SpJzmWgWReyAznCwx6kY+fB>w9b&hx zy1sAe!aMDb+nTa=)_2xS;o!K7y~S%CnS^IOx96H4U;9x@&SBLg*POs5N6KVY*X}?5 zxU%0x6_e?7k2+4*cQcly`q6X)r|JZSQ}@v6c&0C#NL_J)m4@fGf=s4awX5>gs#uAv ziMC(B_%QN`?B?43^2+kQy|=de+1jrCcy}W^Iho01#;gauRyFY2Eh}hSr;Zm`^>){C z;wV5y-D`RNvE{UCR^1J%$5z)3dfk>)ZP(l>0D=72syW@7)$RqIUN9vk!$h~+ZFdVO zw&FAzR&eb3R^7DP@J?JwTFQ)}#4`-y5%E}%zY~*4`?`$@thbR9%}YThWTM)vO`-7` znyth$F-?+=miyBQk>j*6PCKeIp<9`xPtO5AsR*HUFL>&NXz z&6>8nR^WC!m~joZ5q5+yS<|-JHK}LY7%K?AuzY`o&19s&Z!pC<^hFs%+2eLi(ra!V zl|fvAZBO`aqdtY!KtLRn?iJxPrx0eKf$+NmPcRxu(i0-d4%&9d>$n0o1`B;&Q%Yet z4291SCI{Tc5@2mv4fS3(I0kMtPr#Bdn31!0jds;(pi{|S%J`V=)vboxnvhbH)<;%R zVVha}5GX1udOpbdPBR)pt=H_h-RzWV;z>yAWJ6P70}4*;FBSv|pQR?lQf>p(6#ioA zUiXq-v%s~w1}Hb{*Z_Wnc_^g5+mF@c2*BXFZWC&i#%?$~&7L>_Ns~XM5-eyN3s=(` z^OK~p+XEv!cGb|m7N}spnD^bPMW)T=E=+#P@IS?R46BI~1sR!?4H+3R%8%MNkR5Iy zg?-sdCZMaMe-)vl6cz=p+CB0s2kbT2)RSm#6{G?fZWsJ2#DY%#y6v?lbpI<0R+|Q3xvJBIv4qyIKJM zklKJAc8;0x4n{d~-WwM0=eF^T1cMDTFg8E}1@$esuH&J{m}vHRja$=1E@jo}DRygM zsjZi8muwijBXAt_eCySDAy>c-(sY7}ZWUW8hr^GiJk{MPZW8F77`5=eAStFj&I{m|^eWCez!2|>oGG}%qiq=rmAsM%|I5I*8_ zCNpD-blQzZY(6PAc!AqQTNY3vL&t(KDCi@rX9aY9g5~>Br#FlQo&%^o-?lQ57@m8=OU!i&fLzI7n3gz;YW(P2s;7+eEH$*m5^lK2}n1yX)ts zbkY_mB>b@oWr)6j{j&}e_X6k0l>(ER8aVe5QCofk(}jU)F#&TH>k}}{Nd%4SHWFZy zSl0cQg7)SLmXhj$9k2@_w%tYyXwB5Qb8kon$vxve7am2t7#M%m-GMBh6YLP>2!$2PNRw z&<%)S)2JWji^eYKct!LRt$R#Ing?pRUF_GbAn$kFs#o`_A-gc-&)G}R-xPR@P+YoJ zVOlTrRSFe>XqXB5s~E81{EeA>yIHe=WzosPV!%W$lIfA1gI=gVK^Lnh{KdqvkN{2p zfPeH*j26(6peh>P9u)7ge_tSnJZV z2ugijRZKnmWxHn`JI`HVHJAkKZZ&MO#L|UxCYvx&kvF5IVA*WfdQfI%PcT)ZF&3;f z>;Wj(KROZUGGlKOd*PHcHH~d5n94MwhZvJuWa3zs`wlq}WO*u^rd#9!bxc4KTp?a# zVsffKJ6=0tbn(xg9DvFL0E-YMsI@@wO9^Ar0B(eij`OAF?I;R8*PXVkmcLGBkXH1tpPp6 zXUq;^5Ek3+;yoH;dlZ~VHqB0Orfqex;@0W02gB2;2NRMQCLptiEFjuCVS{NyD5!gG zqvitzE$ApPQ*}IGqULqM{~KpY($gJP78P)ui{Nc@lQ#gn?z0}C0AlO8+x4+=M@?h9 zPwb(PbWGCQnYYI*K@*f!nAa4Jl&}+$EcO5~3N9(7%bLipeX+m0yt2Q&yTmHlDVYN! zW`*@Ac{gjAI<6co3!%6vm)u1}U?4$rIuE4Ulg!)b-QJ1Uv3%kV84;6Jh>QkF+CWED zd)xy>qEecWtx=LoHpwa-n4Q87GeNezbSSV~T?9dTx_;DZV+z21nvj~TZm%WVshT3l z`gKsz*wn3iMpEC|Ovu+F-b2GxQcn$j@@ z7oV1N_MrbmF}VtAOZ4W6tRKM6B7fXxz;=O#i99l|XLeR}(vF~Pr0IEBe{Mm_Bxn9v5-{c@z?Sc;aWS#>&OZwb8g z_b#gZAdLc;9qel*l8CL4)sgkYhAU$HAlrFe%=9%JYv41*?Q=^>L?)1v z7;UU>xZ2Z4n}9KfRZ9m8jZ?HIz`%6jf}-k^g`or{5f-Nq=(cL=du)eRJx;X3Bm_jA zfRM97-q{a?7Y`Jo*+aIBfgCKVkd(LDtv=hbY7-Vd)W?DppBLAV`a*fs^V@286TxI? zOc`j!z%l;@FKPIygpfGymuS7v3#yX^{sKOlotT7YYVu!@14qqiNG>{4NF>_i#!J=7 zOa^ygEc+<}nF{bjfrH4zm9*n@!AAVc86X^dER@3hnl+t=-W?=0F?J95CbOam7>3@i z*Rgf+L8@nQ3I(cw(*Duy^nxs`3feSQpz&8ZwH4^(Df#JlVC>2*<$A4?R{OM-i%bOs z5dVbZVB8bvaUupi(x~D!L{K`&>G9;$D$&0bY7;WpDUB{76I15FI2D)IqM2{O)QrS5 z35|{REpbW#DU-qEz22e72=qb3&FZG>mu3?GC;)Xrpr}a@X5Q{hWLXu<_29|CGF8Bt zUvO%*34BS%8B6F6O2HBX^orK|Y!nKoYIU$`$PigyFwHLm%PvujUeZKy-EN{*;zu8j zIqbUE^#dDrzgCcfZnURFQIMZ3v4e7Q1*AZPrQr4eUfDe*1>_W)q>ZE1Yr?ji4Ox0j z_e>$Pt2%o_@W!rGIDQU^j2?>XAo;R6i$j$4b!z~FPW86?NRUp4!sx9IU4y+nT$+BC zZWWmvdHL2M&fGf0*;|J=ck2-6Zw+FncH87wg`9Uw6z)8O$mK9>=Ema%s5-dwi(Bg2 zA+*T?T#HWE3;MNM^3{Qz;xZ;Cz?!gWw1Qsq#v@HrOb2DEaO;fep<1K?AHA%%Vo4Fl zz$H-i4is)XmjRR>JnO-)ADk7$=XPTpP~|re5i;;K#KBvIS!>&na&4{g-YRHV31DHg zkJ{~;4ROY%E#bxugfxLKHt}6>s?|0f)9IkaywO*XbIBkJ5qy#&Tc6kySuw)qVC3}< zJ3R#VziJ-`W?is5ld{<@|8&L zFaM_P))9dTJ2(jF*gHDDve~}k*G6;P{1r}d-92)=)-~FcV%KgfW@(Ex)o`7YQt`z# zS~DeYp}}k7lr2ip8#d`5Bbv5#M6LcBMxaEaVN15O`WarPOW%wSqL2b2zrcNo zo}7_(ZYae$W7ZbkcW~1;OjIm%iH#qXjqiqq_@Resli^Qv7AgeLj85wt0*2}4pgjkc zT?FI#Z{rj4Yi)i}K`J%-@Jz%N#8x75-0QYm2f4=x8-I|CG;&=-_&#SvJc*&(+yn|t zXD1I3;uN;SQHzY@doSIbvWWD>=4rrEfPmp`Jc!kaU=2pI_E$y!VM#q^2sgor7c+_> zj<(q`nT=|h4m5;wgz9^ms(BK;Mc+G%bbo{9IEPI;4GXsVEH(wZK=>8Hj)*c0Vg*4|Gk8)hY3 z&nyvnn=w%aVCZrl^fg-_!;xpo#R9iM;s_mJ%%5%o9vF4rBj!Vn+?Xq16L8T8p8zz( zqvNOt&?n(15fm5`b!+3(Y%6tbF_pf5zP0(a`&i5-Dro0P7nnv|1w)reI1l& zig#&ev`y|rE}w@f9!Sqk;cRiI0`o1R+CXi>2V^>OQ`#n5lXEUpWSAA$uGK&;hKqnh z=zk$#s7)TYn3`RkCtNO^8`3h_p3s1f4}!lR6KXoij=|fOo5EpmIPEJrSyXm_<2P3h z6J~7Tbb6dI$BxBxSxF(CVzgUGr40MvnuA(h!exhUR&YRy>-kPZ61d|)& zj}b25Rb5zjkI+)QsjGlj;d*kQ51Ao+ChliaCpEW%$$nGJi$1R$?kO$Ayzy+&ERO5X zfq+LeS@#0q%#yATf_}X@s8<^{XE*Aanv5(m)Eefx%Dx^@vKl!5yjn`{iTp7Ca_o;? z%$DE241zZeuM0q?sL6#6 z7hq>)H_STV=7K0QY>$|4eAj_{6O2C7#jWM7FX^ZBRFpd+%A?wtFfEqHxY|{$MmoCc z)%*vtkSw#ePBD_s={151o$^yrKqTe1=%cpm?G86X99}bfH#G%_QL!;nol-bmU0;Pd z1)vsUGZAOjx_ajX#b}|(U^9%;3EkPk(cg$l$Rw0oI^1(L9|?1W?)7wVWT`ymsL?(I z4zL21Ba~laVp11+1mq)1J~-W=#HnQ1E&^TmbWrl95f>%;m%moQmw6I5RHh(?*lSIc|-J3s3@?A|1|kh-BnNa}g;sEYqxM zw|`1oRGtix$3iLwm9G#;>w{6Ga1JgFF+jJ7n+(L|!xp^^}}yCR4m1@im#;S*Q!Q%5 zqztxTw4Un^MaV4FhGK0PPy>x5WK9#kJjfi>HcOclY_%_fk<8khjP2WYj1Y_H)ozxd zpxBx8*;Up6FJDUXM{+}i?(^Iy^g4aEb}2U*KlI5m{E!!=l#E%WRXN#}FFt)y$1ykv z4_dd%PO;^|T`59<34MHJW!ZzQwIn^k5cyMfwJef+>wGdfwv;WFCq}xSh&8GKxF>jk zv_?dWe8aPL(KZv|lk{4lV5zPtNOoAW@~noL<@$^UFt{%mrTyahHxSL{kbQEHRXk1n zzM}GHFA@R`cr9RaR?^_X@(F__kKKr=7T~f_DZ1(!lB+AJ@EO*kZus22yk5jxs1VU> z7u9DNK~#H8&X+JW*_Kw8qE{9RlCz?UXD}Y1pjfuO-pw1ACY7*;n@L0o?rP2(uf&8EnLS7p4 zZ+I<4;d%i@X0GNCK^csn+J=yF@EJgNQMXFK5L^n2CXs?go(^!&7W^rVr=}EKpTuCpanLB8}rI9V&-5WHjlK zy#}TtM5SOIHeI>XCR|uCixHJt8cxiP9RV_946|UJNR}1S(+8;c;r7zUr->kb^ukM= zoW=p)22Q?VqoJgs&ckjRhirIX8a%VGEk9TCK;n>f{MHN&AJ=UdYUOlNaDX`vi^YfS zwidf|L?5AM*TVZps7DrP9PuZvG!5~9eGOf&lwr-n<$62>@5dNX8c>DA%W4V|#Xm2| z)viB9ddd&)9rJb)Itw2_*1_(f^07 zp2Dm3#CvGO07!M&(9oZ?lzB?k=2}#2LDr9g72Brby&%@>BqjD`pyYu9^F+r%u$<=7pQ_`AIM{l4qYUoC!Ux>YmeZBIy$%~X?z4l5-?p=An)rj3-^}oCsvJQc1_lcD z5k%Blt>7OI1H8b*sjK-}b>I6H_rK$sXj~LrEX@{Y%A@;Vhc+Q`z~9OH-y84z%isDJ z|M{=S$2t0*ih=0<_u=t@tJZ&X|GQcLU;_)I``=S1?3NSz?ti~mo|!3>7Vpi?&MeH7 zZ+icG`Seln<@U?+quDpR|6MQR(f#kYJn1oT!&~)HBUi5f@{GR!-K_ta`O?g2|Bo;@ zyl8aGmK%-%tmfiY(L=wAD;-RM zsFPo1R85;F5^(t8RDt*ifr-;J%lf$B-mUi<0tQX9vohq1@`s1)Zz1FY1FWO@8cGkb z{x5z1C3^p9VHUd%<^TBw?Ej(d?r9NwxAn~--OFM0p5G*c=Y`QPmSvvc#K{XfFs@V>!s#h0EQtZdj1);HEF zym!lX5b%EHyKB6P$}b@9Bnx-4Ie7y4U#e_B-rsq=Z?7)zFWdX;YrCaN7E*R2j|*gR zh8xGms&Lb>Vt6RVgcGZX-;3as(FdGhQb&pt_2pnwvhwDm4KzzB9|t#i=b1#)7){_H zXTI4u^zx)K1M`N&QAqej7-d%(>^B0Zpo~~$oYlhX5O6oLBXr~NSOQMb;8Ioka-?N5 zkma2nduw?UI0XJ!J{IvQu}!An>e}AQ?)uLD`u0{OJ8r7sdQSR^#?4e{2reS)>9EH_ z6$co6&0piqLa$|@$;so&_$&T<&R+6_bJgwg2!^=)MU~@~bi`5HI&1fMy{>EOZ#_n^ z{bz`?->c+HsIT31tcpuZ8x2$L!PfEy9>uEUj)S1%FHKL^TeT`K$icf+aAq|MNA2fJ z<;BI~G^b!1${H_er0#{g<^dMr>q0Id}5%6V{$OSfu_}oSm6pyrKG6-1Z-N&z~(EJvx0O_0O#0k@`18N*La%k2-zD z`Y)B|l>Qg3|Jl;qDE=ct;1ItdEFVGsP;Eq2P|ts~q1}kC0ks2Rg>uzN#;rnk$mVbO z$!a(;vNK6?zp&-D{2tUeC@Jc}97k(Ai2FZrTiJ}#edM}v^Br;EdC4`p#KX1SwaU}Z zY3(T<0>HsO85%uURM+^}3mz8pj_?oz4v6?CmZ`Qc^`@cTDiwzMq0(Qb$EjRRXOQA% za4Al{0hM`N%1@7%3iM|kz0@PKaI+y!I$B|S&~>WMn{}WP5yNt+fxK196B-PBl)M5x zU6R5s0wYSrUf-gIy}!I`udeUf8`~?(8J*Nye}<*DwB3|N&>QQ+e#AoT>K zW#KK>abjgU*-H&w-?4)M;(dtcUK@`-h6I$&RBNiPE__46$!CYS2!4)1EB%iYJiGwF zEBJqLW@c8~|4K767QnPka_2~e|WO`XFq=L3A65u|IdH@&j0;K|7}KojQ(7Tfu#O796oT>`iHdF zxBllAW?}z-acR$QDHFC-rwbQ_k1GHhjiz%ESQM+*FUmwZ28oXq{(DK$Z&Oyn|5sd` znO#^Yl#2ICbBha$3z>x(Yh(TX^6tvR_0QG{FSz>n?s0NCR#uMJKV4gIFF3{h&%W?} zR9jY`lZEQ?qkEsUI-Ah{iq;;0Y`kjdgKZ{7Vd z8JrUs`Trse4)5FlNBvKcU8w8~O>2W9QYgTcbkv4!Cbx6R8SrYa%Xg+Cl|`v{O3zhZ zH1tq(#;n9Y=JPoCC3nf?3Wy=h;o>8cAh#;nextrnc(FKd&(B&`zFJSJh}L*1(ZbP7 zz9t#C9V?rW@AbqD!@7?C_q(W0jqNWk1{2ST>3^(2J@0NjAK$O5|NYt<>3=`@ z_wW3tKmW)7^_Tk^9sL}{K&1Z-j}Kh6{-gNMl3M@eqWJ$u`rqJu-F8Z!{#U*?vrt$p zEzI6qD9+!zsRMAc{iw9{tUh~m^kxn~y^Kd1;BC)(1aEk&K5FBN^Dl(){_gtL$Calt*zs2a zJ*<3g{U&Y)!9^wZ(@bWJfr+`Fg z3%fOAEeq}L(X?V+Wzv&^1cmkCTiOZ=QPorx?~i~d2e=hMtU#TX7O70+BD4c{h*c~??1d2 zyhTLyO*_5dSb*vgnNU`7F5|*hT-2`cpY-%jrUU1+zSD}(+Fd+dqas6LnS{3& zZe4gJ5ymWTU+phOR+IkMyngA00jRD#dygWMrzzr%UggnMqZdqtEn z_}2KV1Y|Uc!kH=z3(*0|@%|#yQ6_pWsNcfX%*3vL%sX?JOoN5#$UbO|hpRW4Z4P%1 z=HrE*8ltEZXAGjcgU2>a-i@soLu8oWG0@QeFFk8}8^gU~|1U1g8vE}&z9Rm&TrAJd zjqJa-f%_|ATtokV>z@M(H1@at@=9n@AkzO?11stNGcvS~-7ju~f{%?K!tNFN=4Y>LYW-3l_)q@BJOB2N ze(TTwM!a(=-(&j!(D=X=>%YW1pGr7yh&W)q{>#N8;y*_Ef69d2a#BM7FBRwUzry0e zy@h)-<(b(V>;L|v({69Ie6;vx`oAXOZ&3fo&Xex(PG?XNv#EU9y6-z*Y!x?GSB{rI zTYR+h@hARad41k_xLd3~+??MyyVr8Q*lpK7U+;Zcz87qiAEE36??+!hdbzQG|9Gpp z8Emb7cCzUe=eAeZPB-?e#jUR&&u*62<~BcnRNh)X`cymCmzN){g_x{%X7js`cy8lJz@blBxz|;NB3~Ub9#ZzAo zB3!-3_V8`GG^7Q4LI{F%2<<4MfYVP3!fwo<**|;|r9CgOA zc__Gyq=5s%woUWSBFsC*S`j4>X4R_UkwIO9CX-U}B(PgOyY!;`V%`=*1RhZHT&9Kj zIuqwhO8}{!!$gENedvy;^Ck{lpFnN2@yd}tAp0~tar#u^-qj7~60VI)fN|w~^hyzG zn4_(k%ws_jFWJ`S1-YGG*)J^T7_T(&4p+W>j3-8QNO#=UQ~8kHxAX%oA0>)}HUM4d zSr22v8^ztQm^mk0`4I2NS`9pBi_ zhjLa}g2?5}bC?-g0?<1TbBm|?^pOF)h3o0p_DsmaN0|&>t96BtqML@~{4_C3n!s^u zpO+FVjDqMiLP>J!(PfzgbDA6I)HWDju7u8;3|u#r&o`enU?M~Rv+^^d-^+O&!a2IW zpKl&ovk+KO6$y%1@_73VW~G*ueZRmn=u6|T^kL}pX?5=U!8xpO7nGLEdM~PiC(kT% z032ty&DV3rmLMy_GfYLscFIg2Z+5%Zcz9|WNgxkbUfG67r)sBB!1$|_)Udg@*|pC& zWqOn}ybc;uRnsAH0#*8I0n4!j!1BuT7JeoJjt9GMc)fuz3$9Y!(p?YmmQ=Vvr^7W!vcwhPU>miJ z86J9OzAoj{u?%8~CULrXCO4Z+W6k>fK^Doo7S1z<3StDYl<&28guiv38B4H`HtZ=dr%c9Rhf zuQF&1Om;i|E3Dl|TT&+MJmyo`Dh-+|(NzgLhdfIye4P+s(tI;*X)=?4LdX#{R>q_6 zVRp*Osid8nF>QpD3TWnf16XGbrdMxuP4ve8yX1G3xlxMd}qZ-JGB*Fa=5b&u^n z@Van)z{-s!fm;{!){nhHHU!-=x{59P$j#ucbGuV-*?0%C@^t)@$9wzhTlUWK{=;d- zs`E6LvnISOri-yUS=@D==%Qr|(?W&uG3W8-oU$sbwa+(ZB&Dc3{;7+gItP0juKAEH zxc`Mtxq80Fiw(;(IjI5VKLWd5g@2*b!E9&(8c%fbH?1m8ljZ9rD__f5IqRd302yz5 zL$=7HpkD}H1jHQhARdK=k2h%Hs{qb$I|46OLe)JLxE|zad(R;cO5i@ z?Cmyuj#~WO+1*}!yn?GCsAYpIrZ$4EVM^NJsHO%BnU zsayH>x--ojrJ+?wM{ZQjG`oj>ql1;BXifH2<@O{aRi0-_lnGcd7b)@HoCiF-DqL6b zdM;xt1DT*QJ^?MqwemgdgY4Ze?>6t&?moP`d3Wzec&YV)JTzfg1*`m~<(!^a`3KWh z?v*_I0N(%LJohw{D5>g&g$q&P63dR69+iaA>33qWuJB}Flv!q`+}fmaMCJ#T;BqO? zH|mZaS_#sQ^nh+qvs?2!MtCk_cVC5adIj(!ypR~LhW;pQ)0b(DoAz)*Rlh}F$&w$h zZxv3*U+InO^X=VFA8c%YZm+MN`(f(d{_@9bOp@7osCk3Bc#8aFW4XHrp)u56F#4+O zJ>V`A=mwr77q(M%pf2IetQ-{fgI>K(Rn)=DxraVaA-!vcy0L2S?5=HWFRx;-U+ivg z@2}c0nVnBJyu&ZLwVk%_z1UH33Vu5r=y+5QJq5L;y#`MDF+5MvA>J}N6~{UJ(^}+1 zb%i@gs$PqIhkVN#R|1QcGpu|Y^$q$Vz3)dq%jb|-xG(X<@g;NswcRRz^j-=5`j+)5 z1N9YMT7zg(8UJ-m#h#u_kxlf2gd(O$lmn!>h6C|RE4rAZ2+vWeaShQ-L#~3iSN5P- z2dZ|7nyNejN%4>vA%a9cL&+SDEqDzpFQu-)s%l=~uOUuC`gRp`(`O1voH#Aj^vgpr zRtGwhMQ9v-LU;Uw0j3jaL@%bb&KZI}xDBUCEJ;CP;HK(4gf(IZt0_i@d5C=42m`$( zrUPRUAd6QaphC!#RH}OBwgQ|}PedAC53hEs_#)$9)$h(%QjvmEG)&pN8n2QYw2JX; z3Hxa)zFEoXtMeqi+K^}O&?Tb@o#VJkp4}A76*59~gBQYKMVY~>hl4Vi+rgoMipF0P zgQ~{OPzV^ei^Gdi*pI;s6pe;kGHX3E##;enBqeAz^cvDQzJSBF^5z;uvj{Md@RJ4O z5Sf!AYP>KT^zoLH(|8VT3+e<~WWWw(GZ6~RWMne!hNw1i8c%e{y4Bf-Mso4&tGr<> zpycdF7e)@O9A;g(Y-H7HE&+(La^)R+Adex5U?&=zS1vKbP|D{+S!9=^`2swvCL ztAeP=MlQKnX}2OLiFBlx;RE#mFlKEZTZ9xEp(Veh4HYZ22Z;9o!k`apVsvYaykAV& z(QN3xC5I4S;#i@IW6c^P%O~6S_a5JeSc+R@@EBuo^fC*%ozQar^lwbt(?_|dc-jG( z5G!Fg8BcWP+vbo=@llPEhKB)t{!g^iVYLDw^;+-0;mIL{{>lBH<(bj_&%>IIB;0Gn z|K0gnJOB5;e({~(|KI=SufCrg>F8Gq24eAlVZ?Y!u^S~_wf@cdzi9o>EiB;t?&v71zy*&ICpRH#_vBIefh|L{_y?>M~~kq{?9DqZ!rGvS~&YI6^d-r z!yDD*>%WBC-qiX3X#LM3XlNAwHM|*#Adgvm0o^(Bs@B-MQ!r1{51|}lO`d$?OXzaq zt@bWV6Y*>m#Jk%84Uh0@s3TJ@ywkW;1(qnNGCX2M@1SyhwmY6mqy;?Tdg5aQ4xNcx z+xzP)YfJsDwAulHvP$3_!oQsvLs{#-djy;LVHDyUneJOK7QjS~ zyXx2jSYNt)T8^mMi^FH}7HzK)0H6u&)eyp=Z2ixjuIC14M4#CF{em*VqR>b0S<^N5 z`E(1BQ3H@^7z(OTYEO^7>MqFrd=P{Z*ZUsc)cxQ|CI5a=Me=mlc>aI{q4 zqe|sGb3s9cS5-W~f*@&FSr@cv<+3?Fz+CpLci$JAMk&WJCbALZksX&3S=7M)`dVUt z!wFP1O+yot)++@pD=h)kSJ|(|UyYaY=U*Xc)686E3Kj_Kv&ojqP(OJq!Fi^|&W z?)L5yDkgr5J#!O*BgPSmn(y`(%36~d(=RIqy4T{2A~H2KS7W#oWp3`?zo1tabxPe& ztXLXm|J`h|V4`C&=qVkGql?dd;Z~<*#n~R#5{?P_Bz805d*^d z7&A-uB$xpAj4+`V5JGgxrs};`Rbu)I#uEf-@8?but5x@o$Y5c&v=EvV-$IYE_zztKS?DbmTbT7ZqQTVmM7E0^ z8N6aj#s`k;c0?EuCV-_d2MlbJB^hI0EJkoK zoZvMoymzE*vb4%3w-9crg35RUKiYdJuQ{O+utn4Cz?Ok*bRpcgq^A)dCBXh}Fm23Q zCpsBGaLPaCmUYkzV{TLqumqI)P924C=pEY+%-M)%>jfIS5Ud2>;wsWhM2T*vAS_J} zlr}gx%R-<)01jfo1dM!zb~CQ@(T<^C&o!2deHcz3)*xa@#mTnrFYi4}VwEZO-uC0& zl{MYk>mU?s#_6^oD=`biLY@o;+t|H`^iGUgi5 zT~7#^M({*3SZc#!@TN$fX`9j&+?oEnzWwgH+nt)+l-)u&uAQ=HL|fm$u8Fl)J2Y%A z6531(XgWs3vP!L{e^j#S4d;0(*ob*Y*|ANp4=u@kG?D_s4VV6#*WPh&XHcgdZf7n#6A91kuyC+9Xr8RfiZ3WOoggGJ;b6YD9RJO1<% zeLL2NAAX36?2G_8e;S9ozr925bY-YSqTi@w-+#QmvHJQ|)gr!Dy=lK&^%L!`Zy~^H zW8-?Y6DnlvMvd647{oJ*EX0ps5CL2o`6l3zjk1jH!#NIYn5?Mq%G&ngpJ+WYt&Y~?iO~}N><%j zim)KmWk8$W3Y3NdSu5Q3L~{`22vIJHCMpVqDG>M)rK;7eGSv4d z&rGk%HNT>Ky^vJ4{t@MZh*S=~NUF**ORf?IBCGV|!mErYe=7hN3;h@GiHa7m`Ne4@ z^Yi2q5AT05N9cy|4hZbOmz@7C#_oTeFXQD$-v3(U``@GUzmc7GXy3kz{@}39?G~e@n%gxzfEt zX=ZT&4*1fI?|<1|d(`;!YybFk{f+#;W*LwCzr&*heI|Ygt3K-U73;rToL?~af9Ul; zH$U?KM)(`%H@s3`4RE&vHN~dq`@AzlpA}3o8dRP#(mnYYAgtcn8ilyh*hzaFgN3J+ zwj1g~@Ue;=Tu$A`L*v8amVIRqyUb%DDg+l=g1$HQyLhfVc54C7pRDbz?5^+Z}jV1$7Y21XbdVPJ%T5e7yW7-3+9fe{8q7#Lw-gn}j WV1$7Y21XbdVPJ%T5e8lp1OE?E6a8TT literal 0 HcmV?d00001 From c5bea2be206779e838e5bcfd968e9ec7909ca7ec Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 14 Feb 2026 14:48:30 -0800 Subject: [PATCH 14/27] document user script --- _extras/short_submission.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/_extras/short_submission.md b/_extras/short_submission.md index 37ef1a0..f7a0b16 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -16,6 +16,7 @@ export DIRECTORY=myworkarea ### copy these scripts into that top level directory +(You can access a tarball with them all [here](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/files/usefulcode.tar) ) [setup-grid](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup-grid) (should not need to modify) @@ -39,9 +40,9 @@ export DIRECTORY=myworkarea -### modify one script (should not need to change the others) +### modify two script (should not need to change the others) -edit `setup_before_submit.sh` and `job_config.sh` to reflect the parameters you need. +edit `setup_before_submit.sh` if you change code versions and `job_config.sh` if you change more temporary things like fcl files . - choose your code version (code version has to match your build) - *make certain the fcl file is either in the fcl path or in `$DIRECTORY`* @@ -51,10 +52,10 @@ edit `setup_before_submit.sh` and `job_config.sh` to reflect the parameters you Then run it to set things up ~~~ -source setup_before_submit.sh +source setup_before_submit.sh # sets up larsoft ~~~ -### from DIRECTORY make a tarball and put in rcds +### from $DIRECTORY make a tarball and put in rcds If you have changed any scripts or code, you must redo this. @@ -63,7 +64,7 @@ If you have changed any scripts or code, you must redo this. ./makercds.sh $DIRECTORY ~~~ -will take a while, produce a tarball on /exp/dune/data and put the cvmfs location in cvmfs.location in `$DIRECTORY` +will take a while, produce a tarball on `/exp/dune/data/users` and put the cvmfs location in cvmfs.location in `$DIRECTORY` Then edit `job_config.sh` to reflect the # of events you want and other run-time parameters. From e0c99f27fc1b83cd0fc8249a7ca8ef2f20291574 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 14 Feb 2026 14:53:18 -0800 Subject: [PATCH 15/27] update doc --- _extras/short_submission.md | 6 +++++- _includes/gitadd.sh | 1 + files/usefulcode.tar | Bin 71680 -> 71680 bytes 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/_extras/short_submission.md b/_extras/short_submission.md index f7a0b16..86bec11 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -16,7 +16,11 @@ export DIRECTORY=myworkarea ### copy these scripts into that top level directory -(You can access a tarball with them all [here](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/files/usefulcode.tar) ) +You can access a tarball with them all [here](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/files/usefulcode.tar). + +Download that tarball into the top level directory for your build and `tar xBf usefulcode.tar` to get the code. + +Here are links to each of the scripts. [setup-grid](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup-grid) (should not need to modify) diff --git a/_includes/gitadd.sh b/_includes/gitadd.sh index b401f6d..5e5142c 100644 --- a/_includes/gitadd.sh +++ b/_includes/gitadd.sh @@ -11,3 +11,4 @@ setup-grid \ submit_workflow.sh \ > ../files/usefulcode.tar git add ../files/usefulcode.tar +git add ../_extras/short_submission.md diff --git a/files/usefulcode.tar b/files/usefulcode.tar index 25e1b8073d3e923010a8233920abee7684d40200..d91920b3d8155999fdb69c103598c4af19a49fa3 100644 GIT binary patch delta 156 zcmZqJz|ydRWkY`)pP7-Fv4Vl2iGi^ZgMz{2hfFD(8RPqyVdAEyX66hE6%g@cenwtX zQ&T+)3j+gVV*_Ke&9W&OOcLfs3WlbJCMJgFP{WMP41t=UhNTNJsikL@C?uw&DCp_w n$EQ}56eSkx7iZ)bmBbg9Cgo-p7iZ?@>E)(OW;}I#QPw&D(0nb_ delta 117 zcmZqJz|ydRWkY`)pQ)*tp@M;-iGi^ZgMz{2hfFD(8RPqyVd6k(a|VS9h2_s_#LsLT&6GL;TVa8^L#&E;Zg_tHYoqD?{YaIY7xF4DT From 41a85a2769a83621b1744138d644c6230a0038eb Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sun, 15 Feb 2026 14:39:26 -0800 Subject: [PATCH 16/27] change APP_NAME to APP_TAG --- _extras/short_submission.md | 4 +-- _includes/job_config.sh | 4 +-- _includes/submit_local_code.jobscript.sh | 24 +++++++++--------- .../submit_local_code_rucio.jobscript.sh | 20 +++++++-------- _includes/submit_workflow.sh | 4 +-- _includes/submit_workflow_rucio.sh | 4 +-- _includes/test_workflow.sh | 4 +-- files/usefulcode.tar | Bin 71680 -> 71680 bytes 8 files changed, 32 insertions(+), 32 deletions(-) diff --git a/_extras/short_submission.md b/_extras/short_submission.md index 86bec11..295a86b 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -44,13 +44,13 @@ Here are links to each of the scripts. -### modify two script (should not need to change the others) +### modify two scripts (should not need to change the others) edit `setup_before_submit.sh` if you change code versions and `job_config.sh` if you change more temporary things like fcl files . - choose your code version (code version has to match your build) - *make certain the fcl file is either in the fcl path or in `$DIRECTORY`* -- add a string `APP_NAME` that will go in your filename +- add a string `APP_TAG` that will go in your filename - add a description in `DESCRIPTION` Then run it to set things up diff --git a/_includes/job_config.sh b/_includes/job_config.sh index db42fb9..b62cfbf 100755 --- a/_includes/job_config.sh +++ b/_includes/job_config.sh @@ -2,8 +2,8 @@ export FCL_FILE="run_analyseEvents.fcl" # fcl file export OUTPUT_DATA_TIER1="full-reconstructed" # tier for artroot output export OUTPUT_DATA_TIER2="root-tuple" # tier for root output export MQL="files where dune.workflow['workflow_id']=3923 and core.data_tier=full-reconstructed limit 2 ordered " # metacat query for files -export APP_NAME="ana" # application name -export DESCRIPTION="$APP_NAME using $FCL_FILE" # appears as jobname in justin +export APP_TAG="ana" # application name +export DESCRIPTION="$APP_TAG using $FCL_FILE" # appears as jobname in justin export USERF=${USER} # make certain the grid knows who your are export NUM_EVENTS=-1 # process them all export FNALURL='https://fndcadoor.fnal.gov:2880/dune/scratch/users' # sends output to scratch diff --git a/_includes/submit_local_code.jobscript.sh b/_includes/submit_local_code.jobscript.sh index 467ed82..d48d803 100755 --- a/_includes/submit_local_code.jobscript.sh +++ b/_includes/submit_local_code.jobscript.sh @@ -20,7 +20,7 @@ Use this command to create the workflow: justin simple-workflow \ --mql "$MQL" \ --jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_NAME=${APP_NAME} --env DIRECTORY=${DIRECTORY} --scope $NAMESPACE --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" + --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope $NAMESPACE --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" see job_config.sh for explanations @@ -28,7 +28,7 @@ EOF # fcl file and DUNE software version/qualifier to be used FCL_FILE=${FCL_FILE:-${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/my_code/fcls/my_reco.fcl} -APP_NAME=${APP_NAME:-unknown} +APP_TAG=${APP_TAG:-unknown} #DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} #DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} @@ -39,7 +39,7 @@ echo "DUNE_VERSION=$DUNE_VERSION" echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" echo "FCL_FILE=$FCL_FILE" echo "MQL=$MQL" -echo "APP_NAME=$APP_NAME" +echo "APP_TAG=$APP_TAG" echo "USERF=$USERF" echo "NUM_EVENTS=$NUM_EVENTS" echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" @@ -109,8 +109,8 @@ Ffname=`echo $pfn | awk -F/ '{print $NF}'` fname=`echo $Ffname | awk -F. '{print $1}'` # outFile1 is artroot format # outFile2 is root format for analysis -export outFile1=${fname}_${APP_NAME}_${now}.root -export outFile2=${fname}_${APP_NAME}_tuple_${now}.root +export outFile1=${fname}_${APP_TAG}_${now}.root +export outFile2=${fname}_${APP_TAG}_tuple_${now}.root # echo "make $outFile1" campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" @@ -127,9 +127,9 @@ export LD_PRELOAD=${XROOTD_LIB}/libXrdPosixPreload.so echo "----- now run lar ------" -echo "lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_NAME}_${now}.log 2>&1" +echo "lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_TAG}_${now}.log 2>&1" -lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_NAME}_${now}.log 2>&1 +lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_TAG}_${now}.log 2>&1 ) larExit=$? @@ -139,7 +139,7 @@ larExit=$? echo "lar exit code $larExit" echo '=== Start last 1000 lines of lar log file ===' -tail -1000 ${fname}_${APP_NAME}_${now}.log +tail -1000 ${fname}_${APP_TAG}_${now}.log echo '=== End last 1000 lines of lar log file ===' @@ -149,9 +149,9 @@ echo "--------make metadata---------" #sam_metadata_dumper ${outFile1} -echo "python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=${outFile1} --appversion=$DUNE_VERSION --appname=${APP_NAME} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} # > $outFile1.json" +echo "python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=${outFile1} --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} # > $outFile1.json" -python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile1 --appversion=$DUNE_VERSION --appname=${APP_NAME} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} #> $outFile1.json +python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile1 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} #> $outFile1.json file1Exit=$? @@ -162,9 +162,9 @@ echo "------------ non-artroot metadata -----------" oldjson=${outFile1}.json -echo " python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_NAME} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json" +echo " python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json" -python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_NAME} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json +python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json file2Exit=$? diff --git a/_includes/submit_local_code_rucio.jobscript.sh b/_includes/submit_local_code_rucio.jobscript.sh index 9ab23ca..e8cadf8 100755 --- a/_includes/submit_local_code_rucio.jobscript.sh +++ b/_includes/submit_local_code_rucio.jobscript.sh @@ -18,17 +18,17 @@ Use this command to create the workflow: justin simple-workflow \ --mql "$MQL" \ --jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ ---output-pattern "*.root:${USER}-output" --env APP_NAME=${APP_NAME} --env DIRECTORY=${DIRECTORY} --scope $NAMESPACE --lifetime 30 +--output-pattern "*.root:${USER}-output" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope $NAMESPACE --lifetime 30 The following optional environment variables can be set when creating the -workflow/stage: FCL_FILE, APP_NAME, NUM_EVENTS, DUNE_VERSION, DUNE_QUALIFIER +workflow/stage: FCL_FILE, APP_TAG, NUM_EVENTS, DUNE_VERSION, DUNE_QUALIFIER EOF # fcl file and DUNE software version/qualifier to be used FCL_FILE=${FCL_FILE:-${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/my_code/fcls/my_reco.fcl} -APP_NAME=${APP_NAME:-reco2} +APP_TAG=${APP_TAG:-reco2} #DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} #DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} @@ -38,7 +38,7 @@ echo "DUNE_VERSION=$DUNE_VERSION" echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" echo "FCL_FILE=$FCL_FILE" echo "MQL=$MQL" -echo "APP_NAME=$APP_NAME" +echo "APP_TAG=$APP_TAG" echo "USERF=$USERF" echo "NUM_EVENTS=$NUM_EVENTS" echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" @@ -101,8 +101,8 @@ Ffname=`echo $pfn | awk -F/ '{print $NF}'` fname=`echo $Ffname | awk -F. '{print $1}'` # outFile1 is artroot format # outFile2 is root format for analysis -outFile1=${fname}_${APP_NAME}_${now}.root -outFile2=${fname}_${APP_NAME}_tuple_${now}.root +outFile1=${fname}_${APP_TAG}_${now}.root +outFile2=${fname}_${APP_TAG}_tuple_${now}.root campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" @@ -115,7 +115,7 @@ echo "$LD_PRELOAD" echo "now run lar" -lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_NAME}_${now}.log 2>&1 +lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_TAG}_${now}.log 2>&1 ) # Subshell exits with exit code of last command larExit=$? @@ -124,7 +124,7 @@ echo "lar exit code $larExit" echo '=== Start last 1000 lines of lar log file ===' -tail -1000 ${fname}_${APP_NAME}_${now}.log +tail -1000 ${fname}_${APP_TAG}_${now}.log echo '=== End last 1000 lines of lar log file ===' @@ -139,11 +139,11 @@ fi # make metadata for both files -${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_prod.py --infile ${outFile1} --appfamily duneana --appname ${APP_NAME} --appversion ${DUNE_VERSION} --no_crc > ${outFile1}.ext.json +${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_prod.py --infile ${outFile1} --appfamily duneana --appname ${APP_TAG} --appversion ${DUNE_VERSION} --no_crc > ${outFile1}.ext.json file1Exit=$? -${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_prod.py --infile ${outFile2} --appfamily larsoft --appname ${APP_NAME} --appversion ${DUNE_VERSION} --no_crc --input_json ${DIRECTORY}/pdvd_input.json > ${outFile1}.ext.json +${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_prod.py --infile ${outFile2} --appfamily larsoft --appname ${APP_TAG} --appversion ${DUNE_VERSION} --no_crc --input_json ${DIRECTORY}/pdvd_input.json > ${outFile1}.ext.json file2Exit=$? diff --git a/_includes/submit_workflow.sh b/_includes/submit_workflow.sh index 139364a..cfe92ef 100755 --- a/_includes/submit_workflow.sh +++ b/_includes/submit_workflow.sh @@ -9,7 +9,7 @@ echo "DUNE_VERSION=$DUNE_VERSION" echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" echo "FCL_FILE=$FCL_FILE" echo "MQL=$MQL" -echo "APP_NAME=$APP_NAME" +echo "APP_TAG=$APP_TAG" echo "USERF=$USERF" echo "NUM_EVENTS=$NUM_EVENTS" echo "DESCRIPTION=$DESCRIPTION" @@ -21,4 +21,4 @@ echo "---- do the submission ----" justin simple-workflow \ --mql "$MQL" \ --jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_NAME=${APP_NAME} --env DIRECTORY=${DIRECTORY} --scope ${NAMESPACE} --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" \ No newline at end of file + --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope ${NAMESPACE} --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" \ No newline at end of file diff --git a/_includes/submit_workflow_rucio.sh b/_includes/submit_workflow_rucio.sh index e465d24..5964560 100755 --- a/_includes/submit_workflow_rucio.sh +++ b/_includes/submit_workflow_rucio.sh @@ -7,7 +7,7 @@ echo "DUNE_VERSION=$DUNE_VERSION" echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" echo "FCL_FILE=$FCL_FILE" echo "MQL=$MQL" -echo "APP_NAME=$APP_NAME" +echo "APP_TAG=$APP_TAG" echo "USERF=$USERF" echo "NUM_EVENTS=$NUM_EVENTS" echo "DESCRIPTION=$DESCRIPTION" @@ -19,4 +19,4 @@ echo "tardir $INPUT_TAR_DIR_LOCAL" justin simple-workflow \ --mql "$MQL" \ --jobscript submit_local_code_rucio.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${USER}-output" --env APP_NAME=${APP_NAME} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" \ No newline at end of file + --output-pattern "*.root:${USER}-output" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" \ No newline at end of file diff --git a/_includes/test_workflow.sh b/_includes/test_workflow.sh index c9f1182..c93e945 100755 --- a/_includes/test_workflow.sh +++ b/_includes/test_workflow.sh @@ -9,7 +9,7 @@ echo "DUNE_VERSION=$DUNE_VERSION" echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" echo "FCL_FILE=$FCL_FILE" echo "MQL=$MQL" -echo "APP_NAME=$APP_NAME" +echo "APP_TAG=$APP_TAG" echo "USERF=$USERF" echo "NUM_EVENTS=$NUM_EVENTS" echo "DESCRIPTION=$DESCRIPTION" @@ -22,5 +22,5 @@ echo " go up one directory " cd .. justin-test-jobscript \ --mql "$MQL" \ ---jobscript $DIRECTORY/submit_local_code.jobscript.sh --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USER} --env APP_NAME=${APP_NAME} --env NAMESPACE=${NAMESPACE} +--jobscript $DIRECTORY/submit_local_code.jobscript.sh --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USER} --env APP_TAG=${APP_TAG} --env NAMESPACE=${NAMESPACE} cd $HERE diff --git a/files/usefulcode.tar b/files/usefulcode.tar index d91920b3d8155999fdb69c103598c4af19a49fa3..6bf1140b020f587cd39756b87a9facb12d404c08 100644 GIT binary patch delta 555 zcmZqJz|ydRWkYY4fQgBzp`p2gfuV_+xhaE!!Q_WbDVrIydzcY2W+ui)3{SF{%@Q`b-Sq zMy_vT;RRZ4t5WS45D*V?%=>Q|oA+(L#fU1rdHGgrMjo(AsKU)3wyk0ZJAd9jU9j`d z@4L(h7Fc$m4@_GeJOGyXbI^#91*o@n^4;SulgkhP1k2t&9zliZIiW*uLY~W zecXg;)0tj5crX~685kSG-TWFU7!1u#&4Iqbj0OWEQv+iIQ=`qYAAr$-76yiNjtRmz6H_y|am9X2EPjr@u9JgGjlqn~9i?SV zn<{#B1Psm0O$<#G42%qnOibY#mV)eu$ru?J7$NKzVB|3~(6cbNG`BQ1GB(*PyDE)E z$`q&s>HwH7pa{^_Q2(rNW8numz*eQ&F(4ow?3UMGG&b+qe2WoXV)Npy){J}*)iA}I zUu|2(4)MbDeYy}Y9N%}D5h67AKp%vmcklp2?&CotMplp!wUf^scbS}h_$Nfcnd2dw zcOO}fklQv{=J;BOo-@Zym^SZgableOfoc7uDeRlsPqoPbL&3#&F|4S~8(|c=D@l QHk+G1$6$ue>o0 Date: Mon, 16 Feb 2026 08:15:35 -0800 Subject: [PATCH 17/27] make the tar file al9 compatible --- _includes/gitadd.sh | 6 +++--- files/usefulcode.tar | Bin 71680 -> 54784 bytes 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/_includes/gitadd.sh b/_includes/gitadd.sh index 5e5142c..3c06329 100644 --- a/_includes/gitadd.sh +++ b/_includes/gitadd.sh @@ -1,5 +1,5 @@ git add *.sh *.py setup-grid DUNE*json -tar cBf - \ +tar --no-xattrs -cf ../files/usefulcode.tar \ extractor_new.py \ gitadd.sh \ job_config.sh \ @@ -8,7 +8,7 @@ maketar.sh \ setup_before_submit.sh \ submit_local_code.jobscript.sh \ setup-grid \ -submit_workflow.sh \ -> ../files/usefulcode.tar +submit_workflow.sh +#> ../files/usefulcode.tar git add ../files/usefulcode.tar git add ../_extras/short_submission.md diff --git a/files/usefulcode.tar b/files/usefulcode.tar index 6bf1140b020f587cd39756b87a9facb12d404c08..8fcb0383c1bb8d3816bbb3f5f808300778532929 100644 GIT binary patch delta 152 zcmZqJz|ydWc>&8Nj@~c3|Ez|#i*{Uo0qR!kyuhvRIH$zoTi|sr=OOYlUl4_TAZ3znvuI-^5&Zvtc;7YSQZs=Eo$Ie)W@}G4$q=>3X3jDE&9L%0BSKY2mk;8 delta 3436 zcmchZOKclO7{}va7qp7#!T~9W!>UvPfmr6bvnyl?J2u$yD;v@zHW0e;W8*w*$97{k zmlP!uQYD(U=E4c&3mm8vB_IyH04EeK5rPA`l^fzwm2xVFG7dKJ#<7KLIeS=*S2N#? zfAjzT-*@G+N3Lw!Keh|wM%kA%63P6_H|?<-32B%>CP4}bfeOO}N&qs1>3~$O7`8;k zW_!4ffwWvqFDIRlF$jUeXUKdmR3Orbz=#y|yIm8bKL3N^kHe!KgLLdQ%wTNkP2&%A)7T z*^##gYR~z*XT73g*ZNz-T}MHDtTCHy;i*W1fvw60v5^ zjYX6ytOI9_QYxz|02l+T^vW5P^kOu!hIG;;=OyurjoqIfKW)x8XE%olLv@KZ&|w8~ zbameEykK+0-6`p%%EZM_2Hz5P|Gwz3-2msRa~8R$ta2xWLN0v(6;ivqc5>HTV)dZ<{iCOS4b_y;tm#~xrE+*LhyltFsj-BJG_yq7 zk#zt@k){F}a}Oa{QF}xQA_YCYy24Do)DC5C{m6h?on!x3@#=%y?+*13r6Jb*=9qK* zS231cz@8l);ynZWSL?)N`kIPTK5u}m(+QG!-C ztOe2Tfh!ErT3<1nQ2a-s`OjaQy8Uu>ft13fT z9fns@#5CKfvU5G#-xGg0_~_@y`p5F*dRE;WYEJO$ylADQVajXrVRO)MON0P`ZDGQO zTOtUUN_s6?AXLyx`AZ&Ts$IhTPW7G~+H9Ghq<& zjLy5JRDJyQvBDhkD~4|Z#C;(pcv;IBGZWdwwEt2RX~iJY`P|piu?2lCIO(1bFNca@ z&(vZl4QRyUEd?jzAiQvagpikprgao{nSpv^<#OrX|AsEJHhlKz_u3iBm@cP#hsh#!3V V+k1nj&T?;PcYJ@_(fWGT@ec&8HLw5x From 045049805b4846d0d5251df2dac4bd48a24d2e23 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Tue, 17 Feb 2026 11:58:07 -0800 Subject: [PATCH 18/27] work on getting json input to work --- _includes/extractor_new.py | 65 +++--- _includes/job_config.sh | 1 + .../submit_local_code_patchrun.jobscript.sh | 189 ++++++++++++++++++ files/usefulcode.tar | Bin 54784 -> 55296 bytes 4 files changed, 229 insertions(+), 26 deletions(-) create mode 100644 _includes/submit_local_code_patchrun.jobscript.sh diff --git a/_includes/extractor_new.py b/_includes/extractor_new.py index 257c5dd..6dfad32 100755 --- a/_includes/extractor_new.py +++ b/_includes/extractor_new.py @@ -8,7 +8,7 @@ import abc import datetime -DEBUG=True +DEBUG=False import argparse @@ -218,11 +218,12 @@ def md_gen(self, mdart, md0={}): # Other fields where the key or value requires minor conversion. elif mdkey == 'runs': - runsSubruns = set() - runs = set() - for run, subrun in mdart.pop("runs", []): - runs.add(run) - runsSubruns.add(100000 * run + subrun) + runsSubruns = [] + runs = [] + print (mdart['runs']) + for run, subrun, runtype in mdart.pop("runs", []): + if run not in runs: runs.append(run) + if subrun not in runsSubruns: runsSubruns.append(100000 * run + subrun) md['core.runs'] = runs md['core.runs_subruns'] = runsSubruns @@ -346,12 +347,12 @@ def main(): argparser.add_argument('--appversion',help='application version for metadata',type=str) argparser.add_argument('--appfamily',help='application family for metadata',type=str) argparser.add_argument('--file_type',help='file_type (mc or detector)',type=str) - argparser.add_argument('--file_format',help='file_format (root, artroot ..)',type=str,required=True) + argparser.add_argument('--file_format',help='file_format (root, artroot ..)',type=str) argparser.add_argument('--run_type',help='run_type - (fardet-hd, iceberg ...)',type=str) argparser.add_argument('--campaign',help='Value for dune.campaign for metadata',type=str) argparser.add_argument('--data_stream',help='Value for data_stream for metadata',type=str) - argparser.add_argument('--data_tier',help='Value for data_tier for metadata',type=str,required=True) - argparser.add_argument('--fcl_file',type=str,help="fcl file name",required=True) + argparser.add_argument('--data_tier',help='Value for data_tier for metadata',type=str) + argparser.add_argument('--fcl_file',type=str,help="fcl file name", default="unknown") argparser.add_argument('--requestid',help='Value for dune.requestid for metadata',type=str) #argparser.add_argument('--set_processed',help='Set for parent file as processed in metadata',action="store_true") argparser.add_argument('--strip_parents',help='Do not include the file\'s parents in metadata for declaration',action="store_true") @@ -377,21 +378,7 @@ def main(): mddict['metadata']={} print ("EXTRACTOR: building metadata from parent and args as no artroot dump available") # If --input_json is supplied, open that dict now and add it to the output json - if args.input_json != None: - if os.path.exists(args.input_json): - try: - arbjson = json.load(open(args.input_json,'r')) - #print ("EXTRACTOR: arbjson",arbjson) - arbjson.pop('name') - arbjson.pop('namespace') - for key in list(arbjson.keys()): - mddict[key] = arbjson[key] - except: - print('Error loading input json file.',args.input_json) - - else: - print('warning, could not open the input json file', args.input_json) - + if args.appname != None: mddict['metadata']['core.application.name'] = args.appname @@ -456,13 +443,39 @@ def main(): print ("EXTRACTOR: inheriting " + key + " from parent file " + thedid) print ("EXTRACTOR: setting namespace for output",args.namespace) mddict['namespace']=args.namespace + + + if args.input_json != None: + if os.path.exists(args.input_json): + try: + arbjson = json.load(open(args.input_json,'r')) + print ("EXTRACTOR: arbjson",arbjson) + #arbjson.pop('name') + #arbjson.pop('namespace') + if DEBUG: "got here" + for key,val in arbjson["metadata"].items(): + + if DEBUG: print (key, val) + newval = os.path.expandvars(val) + if DEBUG: print (newval) + if key in mddict["metadata"]: + print ("EXTRACTOR: overriding ",key,mddict["metadata"][key],"with", newval, "from json file" ) + mddict["metadata"][key] = newval + except: + print('Error loading input json file.',args.input_json) + + else: + print('warning, could not open the input json file', args.input_json) + except TypeError: print('You have not implemented a defineMetaData function by providing an experiment.') print('No metadata keys will be saved') raise # mdtext = json.dumps(expSpecificMetadata.getmetadata(), indent=2, sort_keys=True) - mdtext = json.dumps(mddict, indent=2, sort_keys=True) - + + if DEBUG: + mdtext = json.dumps(mddict, indent=2, sort_keys=True) + print(mdtext) # if args.declare: # ih.declareFile(mdtext) diff --git a/_includes/job_config.sh b/_includes/job_config.sh index b62cfbf..a1fd43b 100755 --- a/_includes/job_config.sh +++ b/_includes/job_config.sh @@ -8,3 +8,4 @@ export USERF=${USER} # make certain the grid knows who your are export NUM_EVENTS=-1 # process them all export FNALURL='https://fndcadoor.fnal.gov:2880/dune/scratch/users' # sends output to scratch export NAMESPACE="usertests" # don't change this unless doing production + diff --git a/_includes/submit_local_code_patchrun.jobscript.sh b/_includes/submit_local_code_patchrun.jobscript.sh new file mode 100644 index 0000000..6169e27 --- /dev/null +++ b/_includes/submit_local_code_patchrun.jobscript.sh @@ -0,0 +1,189 @@ +#!/bin/bash +:<<'EOF' + +To use this jobscript to process 5 files from the dataset fardet-hd__fd_mc_2023a_reco2__full-reconstructed__v09_81_00d02__standard_reco2_dune10kt_nu_1x2x6__prodgenie_nu_dune10kt_1x2x6__out1__validation +data and put the output logs in the `usertests` namespace and saves the output in /scratch + +Use these commands to set up ahead of time: + +export DUNE_VERSION= +export DUNE_QUALIFIER= +export FCL_FILE= +export INPUT_TAR_DIR_LOCAL= +export MQL= +export DIRECTORY= + +(see jobs_config.sh for the full list) + +Use this command to create the workflow: + +justin simple-workflow \ +--mql "$MQL" \ +--jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ + --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope $NAMESPACE --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" + +see job_config.sh for explanations + +EOF + +# fcl file and DUNE software version/qualifier to be used +FCL_FILE=${FCL_FILE:-${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/my_code/fcls/my_reco.fcl} +APP_TAG=${APP_TAG:-unknown} + +#DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} +#DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} + +echo "------ set things up -------" +echo "Check environment" +echo "DIRECTORY=$DIRECTORY" +echo "DUNE_VERSION=$DUNE_VERSION" +echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" +echo "FCL_FILE=$FCL_FILE" +echo "MQL=$MQL" +echo "APP_TAG=$APP_TAG" +echo "USERF=$USERF" +echo "NUM_EVENTS=$NUM_EVENTS" +echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" +echo "NAMESPACE=$NAMESPACE" + + + +echo "Current working directory is `pwd`" + + +# number of events to process from the input file +if [ "$NUM_EVENTS" != "" ] ; then + events_option="-n $NUM_EVENTS" +fi + +if [ "$PATCH_RUN" == "YES" ] ; then + runid=${JUSTIN_WORKFLOW_ID:99999} + run_option="-e ${runid}:0:0" +fi + +# First get an unprocessed file from this stage +did_pfn_rse=`$JUSTIN_PATH/justin-get-file` + + +if [ "$did_pfn_rse" = "" ] ; then + echo "Nothing to process - exit jobscript" + exit 0 +fi + +# Keep a record of all input DIDs, for pdjson2meta file -> DID mapping +echo "$did_pfn_rse" | cut -f1 -d' ' >>all-input-dids.txt + +# pfn is also needed when creating justin-processed-pfns.txt +pfn=`echo $did_pfn_rse | cut -f2 -d' '` +did=`echo $did_pfn_rse | cut -f1 -d' '` + +echo "Input PFN = $pfn" + +echo "TARDIR ${INPUT_TAR_DIR_LOCAL}" +echo "CODE DIR ${DIRECTORY}" + +# Setup DUNE environment +localProductsdir=`ls -c1d ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts*` + +echo "localProductsdir ${localProductsdir}" + + +# seems to require the right name for the setup script + +echo " check that there is a setup in ${localProductsdir}" +ls -lrt ${localProductsdir}/setup-grid +ls -lrt ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE +source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh +export PRODUCTS="${localProductsdir}/:$PRODUCTS" + +# Then we can set up our local products +setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" +setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" + +setup metacat +export METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app +export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune + +source ${localProductsdir}/setup-grid +mrbslp + +#echo "----- code is set up -----" + +# Construct outFile from input $pfn +now=$(date -u +"%Y%m%d%H%M%SZ") +Ffname=`echo $pfn | awk -F/ '{print $NF}'` +fname=`echo $Ffname | awk -F. '{print $1}'` +# outFile1 is artroot format +# outFile2 is root format for analysis +export outFile1=${fname}_${APP_TAG}_${now}.root +export outFile2=${fname}_${APP_TAG}_tuple_${now}.root + +# echo "make $outFile1" +campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" + +# Here is where the LArSoft command is call it +( +# Do the scary preload stuff in a subshell! +export LD_PRELOAD=${XROOTD_LIB}/libXrdPosixPreload.so +# echo "$LD_PRELOAD" + + + +#sam_metadata_dumper $pfn + +echo "----- now run lar ------" + +echo "lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option $run_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_TAG}_${now}.log 2>&1" + +lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option $run_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_TAG}_${now}.log 2>&1 +) + +larExit=$? +# Subshell exits with exit code of last command + + +echo "lar exit code $larExit" + +echo '=== Start last 1000 lines of lar log file ===' +tail -1000 ${fname}_${APP_TAG}_${now}.log +echo '=== End last 1000 lines of lar log file ===' + + +echo "$did" > justin-input-dids.txt + +echo "--------make metadata---------" + +#sam_metadata_dumper ${outFile1} + +echo "python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=${outFile1} --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} # > $outFile1.json" + +python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile1 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} #> $outFile1.json + +file1Exit=$? + +#cat ${outFile1}.json + +echo "------------ non-artroot metadata -----------" +# here for non-artroot files, salvage what you can from outFile1 + +oldjson=${outFile1}.json + +echo " python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json" + +python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json + +file2Exit=$? + +echo "------- finish up ------" +if [ $larExit -eq 0 ] ; then + # Success ! + echo "$pfn" > justin-processed-pfns.txt + jobscriptExit=0 +else + # Oh ! + jobscriptExit=1 +fi + +# Create compressed tar file with all log files +tar zcf `echo "$JUSTIN_JOBSUB_ID.logs.tgz" | sed 's/@/_/g'` *.log +exit $jobscriptExit diff --git a/files/usefulcode.tar b/files/usefulcode.tar index 8fcb0383c1bb8d3816bbb3f5f808300778532929..b8a5790916d98d0ecd5228ac37833d0745f8e8fd 100644 GIT binary patch delta 762 zcmZXS&ubGw6vtgRrgd%FgxDV_)yX8-6xRIQO`tSTq-jCggNYF}1etUv$+pR6*_|Y< zry>XY*uKUplnW@wHt zmIMgkSx-YD1}iQYEzMjgff@>&7LgsWJ)OeCQ1pUYaYCdu8B#UpXfE~k2+?(_!C~lp z)JDu!`5!9ZyBWZPps%sMg?`V#rxj%Rmv3}iz*T$58Z{_}4}od;ksAw% zZBlpUCPsPNY-&Z^po}Pl5nPI>j5f^h;!+Qu2>1X6$L%`4@g2G+B3lSggER0`a9z}B z$ElM+O*5hsR*TSGooGs%7-smG^XA4Nx-seQvpuOdH|cLEin_)A?(6iMciwQd*`|h} zDymjRSah2H4;St0l88Ght6>rKYl{e1jAjEl4?-2SPBH4W)AMKG*XD%Bk=U~i28peG zweDV@;?4@lUiMe5X0H_1mryT+SYZxc6!5f@bCzzX8VffRMt9kw^|^pt&u%}SN~MzN zgzY>jmP`*_RqXP^L^72~4ik4C@wQnL2`TAFQt6Ro>pnj_bm}o6$s;#ck9bb7uH8&? Iuj<=>0P#WawEzGB delta 431 zcmZqJz}&Egc|xy%p_#d{iJ^jlp{c2X0fU0UN3f|K3E%-j?%nA*xL4h>)v6H`M&bC7W+rbciD Qna7zo7e-kyF3QUS0Lbft`2YX_ From 74ffd59100ba7014d992f1c3125cdc1dea0e3c56 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Tue, 17 Feb 2026 16:25:01 -0800 Subject: [PATCH 19/27] fix bug with json readin --- _includes/extractor_new.py | 43 +++++++++++------------ _includes/makercds.sh | 36 ++++++++++--------- _includes/maketar.sh | 24 ++++++++----- _includes/submit_local_code.jobscript.sh | 12 ++++--- _includes/submit_workflow.sh | 15 ++++---- _includes/test_workflow.sh | 10 +++--- files/usefulcode.tar | Bin 55296 -> 55808 bytes 7 files changed, 79 insertions(+), 61 deletions(-) diff --git a/_includes/extractor_new.py b/_includes/extractor_new.py index 6dfad32..d338446 100755 --- a/_includes/extractor_new.py +++ b/_includes/extractor_new.py @@ -374,11 +374,30 @@ def main(): else: mddict = {} mddict['name']=os.path.basename(args.infile) - + mddict['size'] = os.path.getsize(args.infile) + mddict['created_by'] = os.environ['USERF'] mddict['metadata']={} print ("EXTRACTOR: building metadata from parent and args as no artroot dump available") # If --input_json is supplied, open that dict now and add it to the output json + if args.input_json != None: + if os.path.exists(args.input_json): + try: + arbjson = json.load(open(args.input_json,'r')) + if DEBUG: print ("EXTRACTOR: arbjson",arbjson) + + for key,newval in arbjson["metadata"].items(): + if key in mddict["metadata"]: + if DEBUG: print ("EXTRACTOR: overriding ",key,mddict["metadata"][key],"with", newval, "from json file" ) + else: + if DEBUG: print ("EXTRACTOR: adding ",key, newval, "from json file" ) + + mddict["metadata"][key] = newval + except: + print('Error loading input json file.',args.input_json) + + else: + print('warning, could not open the input json file', args.input_json) if args.appname != None: mddict['metadata']['core.application.name'] = args.appname @@ -445,27 +464,7 @@ def main(): mddict['namespace']=args.namespace - if args.input_json != None: - if os.path.exists(args.input_json): - try: - arbjson = json.load(open(args.input_json,'r')) - print ("EXTRACTOR: arbjson",arbjson) - #arbjson.pop('name') - #arbjson.pop('namespace') - if DEBUG: "got here" - for key,val in arbjson["metadata"].items(): - - if DEBUG: print (key, val) - newval = os.path.expandvars(val) - if DEBUG: print (newval) - if key in mddict["metadata"]: - print ("EXTRACTOR: overriding ",key,mddict["metadata"][key],"with", newval, "from json file" ) - mddict["metadata"][key] = newval - except: - print('Error loading input json file.',args.input_json) - - else: - print('warning, could not open the input json file', args.input_json) + except TypeError: print('You have not implemented a defineMetaData function by providing an experiment.') diff --git a/_includes/makercds.sh b/_includes/makercds.sh index 101d1fc..257423f 100755 --- a/_includes/makercds.sh +++ b/_includes/makercds.sh @@ -1,18 +1,22 @@ # give me the directory name as argument echo "----------------------------------------------------------------" -echo "makercds.sh" -echo "first ensure you have a justin token" -justin time -justin get-token -export HERE=`pwd` -# put the tar file on a bigger disk -export THERE=/exp/dune/data/users/$USER/ -date -ls -lrt $THERE/$1.tar.gz -echo " upload tar file to cvmfs and store location in cvmfs.location file" -export INPUT_TAR_DIR_LOCAL=`justin-cvmfs-upload $THERE/$1.tar.gz` -echo "file uploaded to $INPUT_TAR_DIR_LOCAL" -echo $INPUT_TAR_DIR_LOCAL > $HERE/cvmfs.location -echo "return to previous directory" -cd $HERE -echo "----------------------------------------------------------------" \ No newline at end of file +echo "makercds.sh $1" +if [ "$1" == "" ] ; then + echo "need to enter the directory name that was used for your tar file" +else + echo "first ensure you have a justin token" + justin time + justin get-token + export HERE=`pwd` + # put the tar file on a bigger disk + export THERE=/exp/dune/data/users/$USER/ + date + ls -lrt $THERE/$1.tar.gz + echo " upload tar file to cvmfs and store location in cvmfs.location file" + export INPUT_TAR_DIR_LOCAL=`justin-cvmfs-upload $THERE/$1.tar.gz` + echo "file uploaded to $INPUT_TAR_DIR_LOCAL" + echo $INPUT_TAR_DIR_LOCAL > $HERE/cvmfs.location + echo "return to previous directory" + cd $HERE + echo "----------------------------------------------------------------" +fi \ No newline at end of file diff --git a/_includes/maketar.sh b/_includes/maketar.sh index d56bd03..b4365d2 100755 --- a/_includes/maketar.sh +++ b/_includes/maketar.sh @@ -6,11 +6,19 @@ export HERE=`pwd` export THERE=/exp/dune/data/users/$USER/ cd .. # go up one from current directory date -echo " make tar file" -tar --exclude '.git' --exclude build_slf7.x86_64 -cf $THERE/$1.tar $1 -date -echo " gzip step " -gzip -f $THERE/$1.tar -date -echo " tar file is at $THERE/$1.tar.gz" -echo "----------------------------------------------------------------" +if [ "$1" == "" ] ; then + echo "need to enter the directory name and be in that directory" +else + echo " make tar file from $1, excluding build_slf7... " + tarname=$(basename $1) + echo "$tarname" + tar --exclude '.git' --exclude build_slf7.x86_64 -cf ${THERE}/${tarname}.tar $1 + date + echo " gzip step " + gzip -f $THERE/${tarname}.tar + date + echo " tar file is at $THERE/${tarname}.tar.gz" + cd $HERE + echo "----------------------------------------------------------------" +fi + diff --git a/_includes/submit_local_code.jobscript.sh b/_includes/submit_local_code.jobscript.sh index d48d803..ed1bba8 100755 --- a/_includes/submit_local_code.jobscript.sh +++ b/_includes/submit_local_code.jobscript.sh @@ -133,6 +133,8 @@ lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile ) larExit=$? + + # Subshell exits with exit code of last command @@ -149,9 +151,11 @@ echo "--------make metadata---------" #sam_metadata_dumper ${outFile1} -echo "python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=${outFile1} --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} # > $outFile1.json" +FCL_FILE_NAME=$(basename $FCL_FILE) + +echo "python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=${outFile1} --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE_NAME} --namespace=${NAMESPACE} # > $outFile1.json" -python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile1 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} #> $outFile1.json +python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile1 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE_NAME} --namespace=${NAMESPACE} #> $outFile1.json file1Exit=$? @@ -162,9 +166,9 @@ echo "------------ non-artroot metadata -----------" oldjson=${outFile1}.json -echo " python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json" +echo " python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE_NAME} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json" -python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json +python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE_NAME} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json file2Exit=$? diff --git a/_includes/submit_workflow.sh b/_includes/submit_workflow.sh index cfe92ef..6bc6526 100755 --- a/_includes/submit_workflow.sh +++ b/_includes/submit_workflow.sh @@ -16,9 +16,12 @@ echo "DESCRIPTION=$DESCRIPTION" echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" echo "NAMESPACE=${NAMESPACE}" - -echo "---- do the submission ----" -justin simple-workflow \ ---mql "$MQL" \ ---jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope ${NAMESPACE} --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" \ No newline at end of file +if test -e "./${FCL_FILE}"; + echo "---- do the submission ----" + justin simple-workflow \ + --mql "$MQL" \ + --jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ + --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope ${NAMESPACE} --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" +else + echo "FCL_FILE must be in $DIRECTORY for now" +fi \ No newline at end of file diff --git a/_includes/test_workflow.sh b/_includes/test_workflow.sh index c93e945..e58f66f 100755 --- a/_includes/test_workflow.sh +++ b/_includes/test_workflow.sh @@ -1,7 +1,7 @@ # actual submission #export INPUT_TAR_DIR_LOCAL=`cat cvmfs.location` export INPUT_TAR_DIR_LOCAL=$DUNEDATA -source $DIRECTORY/job_config.sh +source ./job_config.sh export NUM_EVENTS=2 echo "DIRECTORY=$DIRECTORY" @@ -18,9 +18,9 @@ echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" echo "tardir $INPUT_TAR_DIR_LOCAL" export HERE=$PWD -echo " go up one directory " -cd .. +#echo " go up one directory " + justin-test-jobscript \ --mql "$MQL" \ ---jobscript $DIRECTORY/submit_local_code.jobscript.sh --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USER} --env APP_TAG=${APP_TAG} --env NAMESPACE=${NAMESPACE} -cd $HERE +--jobscript submit_local_code.jobscript.sh --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USER} --env APP_TAG=${APP_TAG} --env NAMESPACE=${NAMESPACE} +#cd $HERE diff --git a/files/usefulcode.tar b/files/usefulcode.tar index b8a5790916d98d0ecd5228ac37833d0745f8e8fd..6fd0f4c05a9a992a041c42f9452aa6fb4e8284ee 100644 GIT binary patch delta 1531 zcmb_cQEb~(6va)dR#^*VE6^5E-MAiWp-F8gwh<+uX-%}L?SL)?AvSex{G6I5b{0G9 z+L2|HAtAL=S)ar&@Uc-Qgr=Df1VVyId<)!QtLS9RD7R;xAw5ijJ9R8Q(tf6y5NFZu~{z=^&s7ch70~&%XWmLZOFp z{6xL`G|ezez>Y1U6|BrXhR+lRaCYhdZqEJc-kn_Pbt^C7&dYrC#>tKDxFjf&C<)}^ z8Cj9SYoYbXZYU$AL&#X~B#6-2gF2bNVE>Na8O zRt;9HI$^3c@cPZhP}xSXE2D1IYzJ>TiRZjiUaQ-VVS-~+5ak5_M7gA6EwxHHSG8&m z9L>$`2%A(ijC^mGq-^b*$T-*yRYQTAkDRJxVqaC_|?{<8&b&i-= zkQE_AwG_p+iC2g!8Bu7d(xULZ5USF6uw(K65JWXKSU})4)srTaYqt_O-t2oB^qN&6 zJ{Tj7D32xse6cMpSf87;QI+&>R}xgAfT#F1!yd(!og&aQJ=HQwAOw66RQecMjN2cfV$EI`h z#}Cis_H?L)jJoe#9b~#Fp#k`x?ML13KYW~tdK>QF+kXVBd)Ewqh3{P(cQ1YXI^!O{ z^9{4L*5!76^46eNE-wq7uVh|qmvW2vN=yf@V@Q!?q5V4E{<$MTeB~h9fg}PpMMP;8 z*I+07(Zs)eH|%@-ft`UzJdY=nl{01XILy2_&B96Abo|r1&vcM+dw*xZUEkSeCg^Na nD0nSed7(naI&dL3d2BX!r11RgEA(|Rtz|Z*8}8vd#m;{Lb>_p0 delta 921 zcmZXSO-vI(6vrKE1vX%)sA-AOVQKgVW1&G*xhuO3M49^ z2O>=rUgE{X#DhkG1F{}Gc;KkvV4}u&@#e{cUkBsKxLcrG^fsCI`_KE&|Gnwwf$7!2 z-BL(YWFiR?=7=P5(k{$_in0zuw*q0AU?LH*ir^KoLIi)G0x6@6|-rft1 z(3@O`m4RkxPqpuFQwvu{EJIJ2hS}Mnt7EjI3!WtpME);WraI_VfAojpPpXz-bB2wF zX@Rnypl--0;5?m-!D#9@G`5D|J5vV_(xH0q5*O9cE|^J2uuiYKXM7SHR>D;6;RKy9 zRXtl&3s&bg`L}wTq=XFC(581vev6 zR+LHtbUsjB5+y%AZ9W*{6;4uk(NhTuS5X-^0})>0%h6ULMB*!mHG^tc27cMGLnGss zt?Ag-#wfFO&QUvV8+oe3>~%GmRc*?^`gjCBjh}&7u^S4-IArbzQ50I2n&D+J3?GZp z`oS!Y_9Xj~%;H?y`A|BF7P#2rggbvi1cCSFFY?53s=X6TDUp3FIuSETOg4Qj6b+$ebWlhPUA*_irB} g$waOoZ@$zZ1{&VBIWO+*Mo#xTj9mYhf?y*3FUo@&mjD0& From 0a1b79e8a71ef54883703a9200e4abaaa4c2e413 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Tue, 17 Feb 2026 16:26:55 -0800 Subject: [PATCH 20/27] {MQL} --- _includes/test_workflow.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_includes/test_workflow.sh b/_includes/test_workflow.sh index e58f66f..440a608 100755 --- a/_includes/test_workflow.sh +++ b/_includes/test_workflow.sh @@ -8,7 +8,7 @@ echo "DIRECTORY=$DIRECTORY" echo "DUNE_VERSION=$DUNE_VERSION" echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" echo "FCL_FILE=$FCL_FILE" -echo "MQL=$MQL" +echo "MQL=${MQL}" echo "APP_TAG=$APP_TAG" echo "USERF=$USERF" echo "NUM_EVENTS=$NUM_EVENTS" From e3a2d0bd8f64b00ef76e8a148e477838402ad011 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Thu, 19 Feb 2026 14:56:14 -0800 Subject: [PATCH 21/27] fix bash stuff --- _includes/setup-grid.md | 1 - _includes/setup-local | 129 ----------------------------------- _includes/submit_workflow.sh | 2 +- _includes/test_workflow.sh | 6 +- files/usefulcode.tar | Bin 55808 -> 55808 bytes 5 files changed, 4 insertions(+), 134 deletions(-) delete mode 100644 _includes/setup-grid.md delete mode 100644 _includes/setup-local diff --git a/_includes/setup-grid.md b/_includes/setup-grid.md deleted file mode 100644 index 6b54c66..0000000 --- a/_includes/setup-grid.md +++ /dev/null @@ -1 +0,0 @@ -{% include setup-grid %} \ No newline at end of file diff --git a/_includes/setup-local b/_includes/setup-local deleted file mode 100644 index 205ff92..0000000 --- a/_includes/setup-local +++ /dev/null @@ -1,129 +0,0 @@ -# No magic #!, this script must be sourced! -# this script is part of mrb and gets renamed to setup when copied to the localProducts area - -# NOTICE: this script is not relocatable - -# -# Begin boilerplate. -# - -# Note: All the following special tricks for $_ must continue -# relaying the value to the next rule. Be careful! -# Special trick to nail the value of $_ down in a variety of shells. -echo $_ >& /dev/null -# Special trick for tcsh which is one-off on the command history stack. -: $_ -# Special trick to capture the value of $_ in zsh and bash -test $?shell$_ != 1$_ >& /dev/null && \ - dollar_underscore="$_" && \ - dollar_underscore=`expr "${dollar_underscore}" : ".\(.*\)"` -# Special trick to capture the value of $_ in tcsh -test $?shell = 1 && set dollar_underscore=`echo $_` - -# need to be able to check for mrb -test $?shell = 1 && set ss="csh" || ss="sh" -test "$ss" = "csh" && alias return exit - -test "$ss" = "csh" && \ - alias tnotnull "eval '"'test $?'"\!* -eq 1' && eval '"'test -n "$'"\!*"'"'"'" -test "$ss" = "sh" && \ - eval 'tnotnull() { eval "test -n \"\${$1-}\"" ;}' - -# check for mrb -tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS!" ; echo "ERROR:" ) -tnotnull UPS_DIR || unset ss -tnotnull UPS_DIR || return 1 -tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first setup mrb!"; echo "ERROR:" ) -tnotnull MRB_DIR || unset ss -tnotnull MRB_DIR || return 1 -test -f "$MRB_DIR/libexec/shell_independence" || \ - ( echo "ERROR:" ; echo "ERROR: this mrb area expects mrb >= v5_00_00 (found $MRB_VERSION)!" ; echo "ERROR:" ) -test -f "$MRB_DIR/libexec/shell_independence" || unset ss -test -f "$MRB_DIR/libexec/shell_independence" || return 1 - -# Get the shell independence aliases and functions. -source "$MRB_DIR/libexec/shell_independence" - -# Capture the value of $0 -set_ dollar_zed=`echo "${0}" | sed -e 's/^-//'` - -# Special tricks to figure out if this script has been sourced. -# Works for bash, tcsh, and in some cases for zsh. -set_ is_sourced=false -ifcsh_ - # Note: It is unfortunate that we must hard-code the name - # of this script here, but there is no other way - # that works, tcsh is brain-dead. - set base=`basename "${dollar_zed}"` - test "${base}" != "setup" && \ - set is_sourced=true -else - # Special trick for zsh. - test "${ZSH_NAME}" && test "${dollar_underscore}" = "${dollar_zed}" && \ - is_sourced=true - # If there were arguments then there is no safe way to find out - # whether or not the script was sourced in zsh. Pretend it was. - test "${ZSH_NAME}" && test "${#argv}" != "0" && \ - is_sourced=true - # Special trick for bash. - test "${BASH}" && test "${BASH_SOURCE}" != "${dollar_zed}" && \ - is_sourced=true -# Warning, this must be here because the tcsh parser is brain-dead. -endif -endifcsh_ - -# -# End of boilerplate. Begin of real work. -# - -tnotnull UPS_DIR || ( echo "ERROR:" ; echo "ERROR: you MUST set up UPS" ; echo "ERROR:" ) -tnotnull UPS_DIR || source "$MRB_DIR/libexec/unset_shell_independence" -tnotnull UPS_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav -tnotnull UPS_DIR || return 1 - - -tnotnull MRB_DIR || ( echo "ERROR:"; echo "ERROR: you MUST first set up mrb!"; echo "ERROR:" ) -tnotnull MRB_DIR || unset me db dollar_underscore dollar_zed is_sourced base msg1 flav -tnotnull MRB_DIR || return 1 - -setenv MRB_PROJECT "larsoft" -setenv MRB_PROJECT_VERSION "v09_91_02d01" -setenv MRB_QUALS "e26:prof" -setenv MRB_TOP "${DIRECTORY}" -setenv MRB_TOP_BUILD "${DIRECTORY}" -setenv MRB_SOURCE "${DIRECTORY}/srcs" -setenv MRB_INSTALL "${localProductsdir}" -setenv PRODUCTS "${MRB_INSTALL}:${PRODUCTS}" -setenv CETPKG_INSTALL "${localProductsdir}" - -echo "MRB_INSTALL is ${MRB_INSTALL}" -#--- begin middle boilerplate - -set_ flav=`get-directory-name subdir` -set_ buildDirName="build_${flav}" - -test "$ss" = sh && test -n "${MRB_BUILDDIR}" && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" -test "$ss" = csh && tnotnull MRB_BUILDDIR && setenv OLD_MRB_BUILDDIR "${MRB_BUILDDIR}" -setenv MRB_BUILDDIR ${MRB_TOP_BUILD}/${buildDirName} - -unset me dollar_underscore dollar_zed is_sourced base msg1 flav - -#--- end middle boilerplate -# report the environment -echo -echo MRB_PROJECT=$MRB_PROJECT -echo MRB_PROJECT_VERSION=$MRB_PROJECT_VERSION -echo MRB_QUALS=$MRB_QUALS -echo MRB_TOP=$MRB_TOP -echo MRB_SOURCE=$MRB_SOURCE -echo MRB_BUILDDIR=$MRB_BUILDDIR -echo MRB_INSTALL=$MRB_INSTALL -echo -echo PRODUCTS=$PRODUCTS -echo CETPKG_INSTALL=$CETPKG_INSTALL - -echo " got here" - -source "${MRB_DIR}/libexec/unset_shell_independence" -unset db buildDirName - diff --git a/_includes/submit_workflow.sh b/_includes/submit_workflow.sh index 6bc6526..fd8fd7c 100755 --- a/_includes/submit_workflow.sh +++ b/_includes/submit_workflow.sh @@ -16,7 +16,7 @@ echo "DESCRIPTION=$DESCRIPTION" echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" echo "NAMESPACE=${NAMESPACE}" -if test -e "./${FCL_FILE}"; +if test -e "./${FCL_FILE}"; then echo "---- do the submission ----" justin simple-workflow \ --mql "$MQL" \ diff --git a/_includes/test_workflow.sh b/_includes/test_workflow.sh index 440a608..5fde80e 100755 --- a/_includes/test_workflow.sh +++ b/_includes/test_workflow.sh @@ -1,8 +1,9 @@ # actual submission -#export INPUT_TAR_DIR_LOCAL=`cat cvmfs.location` +# tarball is in a local area on my machine (could also set to cvmfs location export INPUT_TAR_DIR_LOCAL=$DUNEDATA source ./job_config.sh +# these are things you need to set ahead of time to run/create metadata - see job_config.sh export NUM_EVENTS=2 echo "DIRECTORY=$DIRECTORY" echo "DUNE_VERSION=$DUNE_VERSION" @@ -18,9 +19,8 @@ echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" echo "tardir $INPUT_TAR_DIR_LOCAL" export HERE=$PWD -#echo " go up one directory " justin-test-jobscript \ --mql "$MQL" \ --jobscript submit_local_code.jobscript.sh --env PROCESS_TYPE=${PROCESS_TYPE} --env DIRECTORY=${DIRECTORY} --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USER} --env APP_TAG=${APP_TAG} --env NAMESPACE=${NAMESPACE} -#cd $HERE + diff --git a/files/usefulcode.tar b/files/usefulcode.tar index 6fd0f4c05a9a992a041c42f9452aa6fb4e8284ee..2d183305573a592ae79e3f1d69946dd50c1a771e 100644 GIT binary patch delta 130 zcmZqJ!rZWhc|xy%nX#dPv7v&2p{a?vA%lX! delta 128 zcmZqJ!rZWhc|xy%k%57kseyulp{a?v0fU0U Date: Thu, 19 Feb 2026 16:53:58 -0800 Subject: [PATCH 22/27] raise lifetime to 700 --- _includes/submit_workflow.sh | 4 +++- _includes/submit_workflow_rucio.sh | 25 ++++++++++++++++--------- files/usefulcode.tar | Bin 55808 -> 55808 bytes 3 files changed, 19 insertions(+), 10 deletions(-) diff --git a/_includes/submit_workflow.sh b/_includes/submit_workflow.sh index fd8fd7c..c6571d2 100755 --- a/_includes/submit_workflow.sh +++ b/_includes/submit_workflow.sh @@ -3,6 +3,8 @@ export INPUT_TAR_DIR_LOCAL=`cat cvmfs.location` source job_config.sh # pick up the configuration +# this sends output to scratch + echo "---- check the configuration ----" echo "DIRECTORY=$DIRECTORY" echo "DUNE_VERSION=$DUNE_VERSION" @@ -21,7 +23,7 @@ if test -e "./${FCL_FILE}"; then justin simple-workflow \ --mql "$MQL" \ --jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope ${NAMESPACE} --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" + --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope ${NAMESPACE} --lifetime 700 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" else echo "FCL_FILE must be in $DIRECTORY for now" fi \ No newline at end of file diff --git a/_includes/submit_workflow_rucio.sh b/_includes/submit_workflow_rucio.sh index 5964560..851d48b 100755 --- a/_includes/submit_workflow_rucio.sh +++ b/_includes/submit_workflow_rucio.sh @@ -1,7 +1,11 @@ -# actual submission - this one uses rucio as output location - +# actual submission export INPUT_TAR_DIR_LOCAL=`cat cvmfs.location` +# this sends output directly to $NAMESPACE in rucio + +source job_config.sh # pick up the configuration + +echo "---- check the configuration ----" echo "DIRECTORY=$DIRECTORY" echo "DUNE_VERSION=$DUNE_VERSION" echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" @@ -12,11 +16,14 @@ echo "USERF=$USERF" echo "NUM_EVENTS=$NUM_EVENTS" echo "DESCRIPTION=$DESCRIPTION" echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" -echo "Output NAMESPACE=$NAMESPACE" - +echo "NAMESPACE=${NAMESPACE}" -echo "tardir $INPUT_TAR_DIR_LOCAL" -justin simple-workflow \ ---mql "$MQL" \ ---jobscript submit_local_code_rucio.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${USER}-output" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope usertests --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" \ No newline at end of file +if test -e "./${FCL_FILE}"; then + echo "---- do the submission ----" + justin simple-workflow \ + --mql "$MQL" \ + --jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ + --output-pattern "*.root:${USER}-output" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope ${NAMESPACE} --lifetime 700 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" +else + echo "FCL_FILE must be in $DIRECTORY for now" +fi \ No newline at end of file diff --git a/files/usefulcode.tar b/files/usefulcode.tar index 2d183305573a592ae79e3f1d69946dd50c1a771e..fef1222904c1d7192323df8138982683583528f4 100644 GIT binary patch delta 113 zcmZqJ!rZWhc|+eReq#eeV*>>PLo)*-69xr?$%%|9n;B0xGfNs97%3Q<8k(4z!<3ks z85qNroG)XPQ&uR+$ShVUPR&axR>&_cDJU&bD9KkSPA*C;NzT|@df^2Vlexj>xXX=f E06e54P5=M^ delta 87 zcmZqJ!rZWhc|+eRenTT;Ljwf^Lo)*-69xr?$%%|9n;B0xGfNto7%CW=8k(4z!<3ks f8JNPAoG)YCJpIBmrp*_pb1`o2xKhD7(LfOZS2`O2 From 1c810ebe4c98ec81b28cc413bad6a00252514479 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Fri, 20 Feb 2026 14:58:26 -0800 Subject: [PATCH 23/27] add rucio submit to tarfile --- _includes/gitadd.sh | 3 +- _includes/sam2metacat.py | 195 ------------------ .../submit_local_code_patchrun.jobscript.sh | 189 ----------------- .../submit_local_code_rucio.jobscript.sh | 179 ---------------- files/usefulcode.tar | Bin 55808 -> 58880 bytes 5 files changed, 2 insertions(+), 564 deletions(-) delete mode 100644 _includes/sam2metacat.py delete mode 100644 _includes/submit_local_code_patchrun.jobscript.sh delete mode 100755 _includes/submit_local_code_rucio.jobscript.sh diff --git a/_includes/gitadd.sh b/_includes/gitadd.sh index 3c06329..cc49595 100644 --- a/_includes/gitadd.sh +++ b/_includes/gitadd.sh @@ -8,7 +8,8 @@ maketar.sh \ setup_before_submit.sh \ submit_local_code.jobscript.sh \ setup-grid \ -submit_workflow.sh +submit_workflow.sh \ +submit_workflow_rucio.sh #> ../files/usefulcode.tar git add ../files/usefulcode.tar git add ../_extras/short_submission.md diff --git a/_includes/sam2metacat.py b/_includes/sam2metacat.py deleted file mode 100644 index ddeced0..0000000 --- a/_includes/sam2metacat.py +++ /dev/null @@ -1,195 +0,0 @@ - #!/usr/bin/env python3 - # - # Convert protoDUNE metadata from extractor_prod.py into JSON suitable to be - # the value of the "metadata" key in the JSON sent to MetaCat - # - # THIS IS NOT REALLY PART OF justIN AND IT SHOULD BE INCORPORATED INTO THE - # SCRIPTS ASSOCIATED WITH APPLICATIONS, LIKE extractor_prod.py ! - # - # Adapted from - # https://github.com/ivmfnal/protodune/blob/main/tools/declare_meta.py and - # https://github.com/ivmfnal/protodune/blob/main/declad/mover.py - # to write out modified JSON rather than uploading it. This allows the script - # to be run inside jobscripts supplied by users. - # - # A version can be found in the jobutils area of the justIN UPS product - # in cvmfs - # - -import sys, json -import datetime -import argparse - -def getCoreAttribute(): - coreAttributes = { - "event_count": "core.event_count", - "file_type" : "core.file_type", - "file_format": "core.file_format", - "data_tier" : "core.data_tier", - "data_stream": "core.data_stream", - "events" : "core.events", - "first_event": "core.first_event_number", - "last_event" : "core.last_event_number", - "event_count": "core.event_count", - "user" : None, - "group" : None - } - return coreAttributes - -def doit(inputMetadataFile, allInputDidsFile=None, namespace="usertests", outputFormat=None, outputTier=None ): - coreAttributes = getCoreAttribute() - - try: - inputMetadata = json.load(open(inputMetadataFile, "r")) - except Exception as e: - print("Error reading metadata from file: " + str(e), file=sys.stderr) - sys.exit(1) - - allInputDids = [] - if allInputDidsFile is not None: - try: - for line in open(allInputDidsFile, "r").read().splitlines(): - allInputDids.append(line) - except Exception as e: - print("Error read all input DIDs file: " + str(e), file=sys.stderr) - sys.exit(2) - - - inputMetadata.pop("file_size", None) - inputMetadata.pop("checksum", None) - inputMetadata.pop("file_name", None) - - - - # Most of the metadata goes in "metadata" within the outer dictionary - outputMetadata = { "metadata": {}} - - runsSubruns = set() - runType = None - runs = set() - for run, subrun, rtype in inputMetadata.pop("runs", []): - runType = rtype - runs.add(run) - runsSubruns.add(100000 * run + subrun) - - outputMetadata["metadata"]["core.runs_subruns"] = sorted(list(runsSubruns)) - outputMetadata["metadata"]["core.runs"] = sorted(list(runs)) - outputMetadata["metadata"]["core.run_type"] = runType - - app = inputMetadata.pop("application", None) - if app: - if "name" in app: - outputMetadata["metadata"]["core.application.name"] = app["name"] - if "version" in app: - outputMetadata["metadata"]["core.application.version"] = app["version"] - if "family" in app: - outputMetadata["metadata"]["core.application.family"] = app["family"] - if "family" in app and "name" in app: - outputMetadata["metadata"]["core.application"] = \ - app["family"] + "." + app["name"] - - for k in ("start_time", "end_time"): - t = inputMetadata.pop(k, None) - if t is not None: - t = datetime.datetime.fromisoformat(t).replace( - tzinfo=datetime.timezone.utc).timestamp() - outputMetadata["metadata"]["core." + k] = t - - for name, value in inputMetadata.items(): - - - if name == 'parents': - parentDids = [] - for parent in value: - matchingDid = None - for did in allInputDids: - if did.endswith(parent["file_name"]): - matchingDid = did - break - name = did.split(':')[1] - namespace = did.split(':')[0] - if name != 'noparents.root': - outputMetadata["parents"] = [ - {"name": name, - "namespace": namespace - } - ] - - # if matchingDid: - # parentDids.append({ "did" : matchingDid }) - # else: - # print("No matching input DID for file %s with parent file_name %s- exiting" - # % (str(parent), str(parent["file_name"])), - # file=sys.stderr) - # sys.exit(3) - - # Add the list of { "did": "..." } dictionaries to top level - - - - # outputMetadata["parents"] = parentDids - - - - else: - if '.' in name: - outputMetadata["metadata"][name] = value - else: - if name in coreAttributes: - if coreAttributes[name]: - outputMetadata["metadata"][coreAttributes[name]] = value - else: - outputMetadata["metadata"]['x.' + name] = value - print ("setting " + name + " to " + str(value)) - -# patches added by HMS to make this more general - if namespace != None: - outputMetadata['namespace'] = namespace - print ("overriding namespace to " + namespace, file=sys.stdout) - -# add the parentage if not inherited - if allInputDidsFile is not None and outputMetadata.get("parents", None) is None: - parentDids = [] - for line in open(allInputDidsFile, "r").read().splitlines(): - ns = line.split(':')[0] - name = line.split(':')[1] - parentDids.append({ "name" : name, "namespace": ns }) - outputMetadata["parents"] = parentDids - - if outputFormat != None: - outputMetadata["metadata"]["core.file_format"] = outputFormat - print ("overriding output format to " + outputFormat, file=sys.stdout) - - if outputTier != None: - outputMetadata["metadata"]["core.data_tier"] = outputTier - print ("overriding output data_tier to " + outputTier, file=sys.stdout) - - - outputMetadata["metadata"].setdefault("core.event_count", - len(outputMetadata["metadata"].get("core.events", []))) - - - json.dump(outputMetadata, sys.stdout, indent=4, sort_keys=True) - - outname = inputMetadataFile.replace(".ext.json",".json") - with open(outname, "w") as f: - json.dump(outputMetadata, f, indent=4, sort_keys=True) - -if __name__ == "__main__": - parser = argparse.ArgumentParser( - description='Command line interface for merge_utils') - - parser.add_argument('--inputMetadataFile', type=str, help='Path to the input metadata JSON file') - - parser.add_argument('--namespace', type=str, help='Namespace to use for the metadata', default="usertests") - parser.add_argument('--outputFormat', type=str, default=None, - help='Output format (default: inherits from input)') - parser.add_argument('--outputTier', default=None, help='Output data tier (default: inherits)') - parser.add_argument('--InputDidsFile', type=str, default=None, - help='Optional path to a file containing all input DIDs, one per line') - args = parser.parse_args() - doit(inputMetadataFile=args.inputMetadataFile, - allInputDidsFile=args.allInputDidsFile, - namespace=args.namespace, - outputFormat=args.outputFormat, - outputTier=args.outputTier) \ No newline at end of file diff --git a/_includes/submit_local_code_patchrun.jobscript.sh b/_includes/submit_local_code_patchrun.jobscript.sh deleted file mode 100644 index 6169e27..0000000 --- a/_includes/submit_local_code_patchrun.jobscript.sh +++ /dev/null @@ -1,189 +0,0 @@ -#!/bin/bash -:<<'EOF' - -To use this jobscript to process 5 files from the dataset fardet-hd__fd_mc_2023a_reco2__full-reconstructed__v09_81_00d02__standard_reco2_dune10kt_nu_1x2x6__prodgenie_nu_dune10kt_1x2x6__out1__validation -data and put the output logs in the `usertests` namespace and saves the output in /scratch - -Use these commands to set up ahead of time: - -export DUNE_VERSION= -export DUNE_QUALIFIER= -export FCL_FILE= -export INPUT_TAR_DIR_LOCAL= -export MQL= -export DIRECTORY= - -(see jobs_config.sh for the full list) - -Use this command to create the workflow: - -justin simple-workflow \ ---mql "$MQL" \ ---jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ - --output-pattern "*.root:${FNALURL}/${USERF}" --output-pattern "*.root.json:${FNALURL}/${USERF}" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope $NAMESPACE --lifetime 30 --env INPUT_TAR_DIR_LOCAL=${INPUT_TAR_DIR_LOCAL} --env DUNE_VERSION=${DUNE_VERSION} --env DUNE_QUALIFIER=${DUNE_QUALIFIER} --env FCL_FILE=${FCL_FILE} --env NUM_EVENTS=${NUM_EVENTS} --env USERF=${USERF} --env NAMESPACE=${NAMESPACE} --description "${DESCRIPTION}" - -see job_config.sh for explanations - -EOF - -# fcl file and DUNE software version/qualifier to be used -FCL_FILE=${FCL_FILE:-${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/my_code/fcls/my_reco.fcl} -APP_TAG=${APP_TAG:-unknown} - -#DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} -#DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} - -echo "------ set things up -------" -echo "Check environment" -echo "DIRECTORY=$DIRECTORY" -echo "DUNE_VERSION=$DUNE_VERSION" -echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" -echo "FCL_FILE=$FCL_FILE" -echo "MQL=$MQL" -echo "APP_TAG=$APP_TAG" -echo "USERF=$USERF" -echo "NUM_EVENTS=$NUM_EVENTS" -echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" -echo "NAMESPACE=$NAMESPACE" - - - -echo "Current working directory is `pwd`" - - -# number of events to process from the input file -if [ "$NUM_EVENTS" != "" ] ; then - events_option="-n $NUM_EVENTS" -fi - -if [ "$PATCH_RUN" == "YES" ] ; then - runid=${JUSTIN_WORKFLOW_ID:99999} - run_option="-e ${runid}:0:0" -fi - -# First get an unprocessed file from this stage -did_pfn_rse=`$JUSTIN_PATH/justin-get-file` - - -if [ "$did_pfn_rse" = "" ] ; then - echo "Nothing to process - exit jobscript" - exit 0 -fi - -# Keep a record of all input DIDs, for pdjson2meta file -> DID mapping -echo "$did_pfn_rse" | cut -f1 -d' ' >>all-input-dids.txt - -# pfn is also needed when creating justin-processed-pfns.txt -pfn=`echo $did_pfn_rse | cut -f2 -d' '` -did=`echo $did_pfn_rse | cut -f1 -d' '` - -echo "Input PFN = $pfn" - -echo "TARDIR ${INPUT_TAR_DIR_LOCAL}" -echo "CODE DIR ${DIRECTORY}" - -# Setup DUNE environment -localProductsdir=`ls -c1d ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts*` - -echo "localProductsdir ${localProductsdir}" - - -# seems to require the right name for the setup script - -echo " check that there is a setup in ${localProductsdir}" -ls -lrt ${localProductsdir}/setup-grid -ls -lrt ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE -source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh -export PRODUCTS="${localProductsdir}/:$PRODUCTS" - -# Then we can set up our local products -setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" -setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" - -setup metacat -export METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app -export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune - -source ${localProductsdir}/setup-grid -mrbslp - -#echo "----- code is set up -----" - -# Construct outFile from input $pfn -now=$(date -u +"%Y%m%d%H%M%SZ") -Ffname=`echo $pfn | awk -F/ '{print $NF}'` -fname=`echo $Ffname | awk -F. '{print $1}'` -# outFile1 is artroot format -# outFile2 is root format for analysis -export outFile1=${fname}_${APP_TAG}_${now}.root -export outFile2=${fname}_${APP_TAG}_tuple_${now}.root - -# echo "make $outFile1" -campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" - -# Here is where the LArSoft command is call it -( -# Do the scary preload stuff in a subshell! -export LD_PRELOAD=${XROOTD_LIB}/libXrdPosixPreload.so -# echo "$LD_PRELOAD" - - - -#sam_metadata_dumper $pfn - -echo "----- now run lar ------" - -echo "lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option $run_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_TAG}_${now}.log 2>&1" - -lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option $run_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_TAG}_${now}.log 2>&1 -) - -larExit=$? -# Subshell exits with exit code of last command - - -echo "lar exit code $larExit" - -echo '=== Start last 1000 lines of lar log file ===' -tail -1000 ${fname}_${APP_TAG}_${now}.log -echo '=== End last 1000 lines of lar log file ===' - - -echo "$did" > justin-input-dids.txt - -echo "--------make metadata---------" - -#sam_metadata_dumper ${outFile1} - -echo "python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=${outFile1} --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} # > $outFile1.json" - -python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile1 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='full-reconstructed' --file_format='artroot' --fcl_file=${FCL_FILE} --namespace=${NAMESPACE} #> $outFile1.json - -file1Exit=$? - -#cat ${outFile1}.json - -echo "------------ non-artroot metadata -----------" -# here for non-artroot files, salvage what you can from outFile1 - -oldjson=${outFile1}.json - -echo " python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json" - -python ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_new.py --infile=$outFile2 --appversion=$DUNE_VERSION --appname=${APP_TAG} --appfamily=larsoft --no_crc --inputDidsFile=justin-input-dids.txt --data_tier='root-tuple' --file_format='root' --fcl_file=${FCL_FILE} --no_extract --input_json=${PWD}/${oldjson} --namespace=${NAMESPACE} # > ${outFile2}.json - -file2Exit=$? - -echo "------- finish up ------" -if [ $larExit -eq 0 ] ; then - # Success ! - echo "$pfn" > justin-processed-pfns.txt - jobscriptExit=0 -else - # Oh ! - jobscriptExit=1 -fi - -# Create compressed tar file with all log files -tar zcf `echo "$JUSTIN_JOBSUB_ID.logs.tgz" | sed 's/@/_/g'` *.log -exit $jobscriptExit diff --git a/_includes/submit_local_code_rucio.jobscript.sh b/_includes/submit_local_code_rucio.jobscript.sh deleted file mode 100755 index e8cadf8..0000000 --- a/_includes/submit_local_code_rucio.jobscript.sh +++ /dev/null @@ -1,179 +0,0 @@ -#!/bin/bash -:<<'EOF' - -To use this jobscript to process 5 files from the dataset fardet-hd__fd_mc_2023a_reco2__full-reconstructed__v09_81_00d02__standard_reco2_dune10kt_nu_1x2x6__prodgenie_nu_dune10kt_1x2x6__out1__validation -data and put the output logs in the `usertests` namespace and saves the output in /scratch - -Use these commands to set up ahead of time: - -export DUNE_VERSION= -export DUNE_QUALIFIER= -export FCL_FILE= -export INPUT_TAR_DIR_LOCAL= -export MQL= -export DIRECTORY= - -Use this command to create the workflow: - -justin simple-workflow \ ---mql "$MQL" \ ---jobscript submit_local_code.jobscript.sh --rss-mb 4000 \ ---output-pattern "*.root:${USER}-output" --env APP_TAG=${APP_TAG} --env DIRECTORY=${DIRECTORY} --scope $NAMESPACE --lifetime 30 - - -The following optional environment variables can be set when creating the -workflow/stage: FCL_FILE, APP_TAG, NUM_EVENTS, DUNE_VERSION, DUNE_QUALIFIER - -EOF - -# fcl file and DUNE software version/qualifier to be used -FCL_FILE=${FCL_FILE:-${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/my_code/fcls/my_reco.fcl} -APP_TAG=${APP_TAG:-reco2} -#DUNE_VERSION=${DUNE_VERSION:-v09_85_00d00} -#DUNE_QUALIFIER=${DUNE_QUALIFIER:-e26:prof} - -echo "Check environment" -echo "DIRECTORY=$DIRECTORY" -echo "DUNE_VERSION=$DUNE_VERSION" -echo "DUNE_QUALIFIER=$DUNE_QUALIFIER" -echo "FCL_FILE=$FCL_FILE" -echo "MQL=$MQL" -echo "APP_TAG=$APP_TAG" -echo "USERF=$USERF" -echo "NUM_EVENTS=$NUM_EVENTS" -echo "INPUT_TAR_DIR_LOCAL=$INPUT_TAR_DIR_LOCAL" - - - -echo "Current working directory is `pwd`" - - -# number of events to process from the input file -if [ "$NUM_EVENTS" != "" ] ; then - events_option="-n $NUM_EVENTS" -fi - -# First get an unprocessed file from this stage -did_pfn_rse=`$JUSTIN_PATH/justin-get-file` - -if [ "$did_pfn_rse" = "" ] ; then - echo "Nothing to process - exit jobscript" - exit 0 -fi - -# Keep a record of all input DIDs, for pdjson2meta file -> DID mapping -echo "$did_pfn_rse" | cut -f1 -d' ' >>all-input-dids.txt - -# pfn is also needed when creating justin-processed-pfns.txt -pfn=`echo $did_pfn_rse | cut -f2 -d' '` -echo "Input PFN = $pfn" - -echo "TARDIR ${INPUT_TAR_DIR_LOCAL}" -echo "CODE DIR ${DIRECTORY}" - -# Setup DUNE environment -localProductsdir=`ls -c1d ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts*` - -echo "localProductsdir ${localProductsdir}" - - -# seems to require the right name for the setup script - -echo " check that there is a setup in ${localProductsdir}" -ls -lrt ${localProductsdir}/setup-grid -ls -lrt ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE -source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh -export PRODUCTS="${localProductsdir}/:$PRODUCTS" - -# Then we can set up our local products -setup duneana "$DUNE_VERSION" -q "$DUNE_QUALIFIER" -setup dunesw "$DUNE_VERSION" -q "$DUNE_QUALIFIER" -source ${localProductsdir}/setup-grid -setup metacat -export METACAT_SERVER_URL=https://metacat.fnal.gov:9443/dune_meta_prod/app -export METACAT_AUTH_SERVER_URL=https://metacat.fnal.gov:8143/auth/dune - -mrbslp - -# Construct outFile from input $pfn -now=$(date -u +"%Y%m%d%H%M%SZ") -Ffname=`echo $pfn | awk -F/ '{print $NF}'` -fname=`echo $Ffname | awk -F. '{print $1}'` -# outFile1 is artroot format -# outFile2 is root format for analysis -outFile1=${fname}_${APP_TAG}_${now}.root -outFile2=${fname}_${APP_TAG}_tuple_${now}.root - - -campaign="justIN.w${JUSTIN_WORKFLOW_ID}s${JUSTIN_STAGE_ID}" - -# Here is where the LArSoft command is call it -( -# Do the scary preload stuff in a subshell! -export LD_PRELOAD=${XROOTD_LIB}/libXrdPosixPreload.so -echo "$LD_PRELOAD" - -echo "now run lar" - -lar -c ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/$FCL_FILE $events_option -o ${outFile1} -T ${outFile2} "$pfn" > ${fname}_${APP_TAG}_${now}.log 2>&1 -) -# Subshell exits with exit code of last command -larExit=$? - -echo "lar exit code $larExit" - - -echo '=== Start last 1000 lines of lar log file ===' -tail -1000 ${fname}_${APP_TAG}_${now}.log -echo '=== End last 1000 lines of lar log file ===' - - -if [ $larExit -eq 0 ] ; then - # Success ! - echo "$pfn" > justin-processed-pfns.txt - jobscriptExit=0 -else - # Oh :( - jobscriptExit=1 -fi - -# make metadata for both files - -${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_prod.py --infile ${outFile1} --appfamily duneana --appname ${APP_TAG} --appversion ${DUNE_VERSION} --no_crc > ${outFile1}.ext.json - -file1Exit=$? - -${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/extractor_prod.py --infile ${outFile2} --appfamily larsoft --appname ${APP_TAG} --appversion ${DUNE_VERSION} --no_crc --input_json ${DIRECTORY}/pdvd_input.json > ${outFile1}.ext.json - -file2Exit=$? - -if [ $? -ne 0 ] ;then - ex_code=$((file1Exit+2*file2Exit)) - echo "ERROR: metadata extraction (reco) exit code: $ex_code" - files=`ls *_${now}_*` - for f in $files - do - size=`stat -c %s $f` - echo "written output file: $f $size" - done - fi -fi - -if [ $? -ne 0 ] -then - echo "Exiting with error" - exit 1 -else - files=`ls *_${now}*` - for f in $files - do - size=`stat -c %s $f` - echo "written output file: $f $size" - done - - echo "$pfn" > justin-processed-pfns.txt -fi - -# Create compressed tar file with all log files -tar zcf `echo "$JUSTIN_JOBSUB_ID.logs.tgz" | sed 's/@/_/g'` *.log -exit $jobscriptExit diff --git a/files/usefulcode.tar b/files/usefulcode.tar index fef1222904c1d7192323df8138982683583528f4..9f4a9d3c464a1b780d651f737d9ab5d73940de1b 100644 GIT binary patch delta 294 zcmZqJ!rX9%c|%W(pqZhWv7w=vf`OrlfsrYLg2CiO#+1#Bu`SFJX66cpriLa66~<KN!VhxQa`Yax+Wf%kzt}({l35;3?n9DW`OJ zjSNhIHklh48^F!kc$JaeM_|u_=cjl!7e*;FPCCbr(B(flrAEdl$I!@WEQ0+m*i9`l;kU@_&NHz1_wAgyDDVn mDL`E5e8FV%^b1~0JfXp^LAAONU6ZS)J8n+9Jd16jfieJ`+hJ}1 delta 94 zcmZoT!`!fic|%W(poy88nX$Q%f`Orlfw4J*g2CiO)|AbRu`SFJ#wH4eriLa66~<;p g#&8vhKNu&AB)^$xAiXH1XtTq*1B_r|asg`t0NMH+vH$=8 From 72fa9a9637a08ec8de4eec3c6922004dc345b6de Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 21 Feb 2026 11:41:18 -0800 Subject: [PATCH 24/27] update explanation --- _extras/short_submission.md | 85 ++++++++++++++++++++++++++++++------- 1 file changed, 70 insertions(+), 15 deletions(-) diff --git a/_extras/short_submission.md b/_extras/short_submission.md index 295a86b..084e8d0 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -2,10 +2,15 @@ title: Short submission with your own code --- -## this collects the sequence of steps for a batch submission with local code which produces both an artroot and root file +## this collects the sequence of steps for a batch submission with local code which produces both an artroot and root file. + +It splits out different functions so you need to examine/adapt all of the support scripts which can be found [here](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/files/usefulcode.tar). + +This sequence assumes you are in your top level mrb directory. ### in your top level mrb directory + For example `/exp/dune/app/users/$USER/myworkarea` need to have a name for it as you will be making a tarball @@ -13,30 +18,53 @@ need to have a name for it as you will be making a tarball ~~~ export DIRECTORY=myworkarea ~~~ +{: ..language-bash} ### copy these scripts into that top level directory You can access a tarball with them all [here](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/files/usefulcode.tar). -Download that tarball into the top level directory for your build and `tar xBf usefulcode.tar` to get the code. +Download that tarball into the top level directory for your build and + +~~~ +tar xBf usefulcode.tar +~~~ +{: ..language-bash} + + to get the code. -Here are links to each of the scripts. +### here are the scripts.. -[setup-grid](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup-grid) (should not need to modify) +#### utilities you need -[maketar.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/maketar.sh) +- [setup-grid](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup-grid) (should not need to modify) +This replaces `setup` in your local_build directory. + +- [maketar.sh $DIRECTORY](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/maketar.sh) (should not need to modify) +This takes the contents of `$DIRECTORY` and makes a tarball in `/exp/data/users/$USER/` + +- [makerdcs.sh $DIRECTORY](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/makerdcs.sh) (should not need to modify) +This takes the tarball and copies it to /cvmfs/ where grid jobs can find it. It places the location in the file `$DIRECTORY/cvmfs.location` so you can find it. + +#### setup scripts + +- [setup_before_submit.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup_before_submit.sh) (customize versions for your code) +You need to modify this to reflect the code version you are setting up. Normally only need to run/modify this once/session. + + +- [job_config.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/job_config.sh) (you modify this to reflect your workflow. Sets things like $FCL_FILE). This sets up essential job parameters. You need to understand and modify these appropriately for your purpose. -[makerdcs.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/makerdcs.sh) (should not need to modify) +#### Script to test and submit jobs -[job_config.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/job_config.sh) (you modify this to choose things like MQL query, number of events..) +- [test_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/test_workflow.sh) (script to do interactive tests of your jobscript) -[setup_before_submit.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/setup_before_submit.sh) (customize versions for your code) +- [submit_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_workflow.sh) (writes output to scratch. Modify running time and memory) -[submit_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_workflow.sh) (modify running time and memory) +- [submit_workflow_rucio.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_workflow_rucio.sh) (writes output to rucio. Modify running time and memory) -[test_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/test_workflow.sh) (script to do interactive tests of your jobscript) +#### scripts that run on the remote machine [extractor_new.py](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/extractor_new.py) (this makes metadata for your files) @@ -44,16 +72,21 @@ Here are links to each of the scripts. -### modify two scripts (should not need to change the others) +## How to run these scripts -edit `setup_before_submit.sh` if you change code versions and `job_config.sh` if you change more temporary things like fcl files . +### modify two-three scripts (should not need to change the others) + +edit + +- `setup_before_submit.sh` if you change code versions and +- `job_config.sh` if you change more temporary things like fcl files. - choose your code version (code version has to match your build) - *make certain the fcl file is either in the fcl path or in `$DIRECTORY`* - add a string `APP_TAG` that will go in your filename - add a description in `DESCRIPTION` -Then run it to set things up +Then run those scripts to set things up ~~~ source setup_before_submit.sh # sets up larsoft @@ -67,20 +100,42 @@ If you have changed any scripts or code, you must redo this. ./maketar.sh $DIRECTORY ./makercds.sh $DIRECTORY ~~~ +{: ..language-bash} -will take a while, produce a tarball on `/exp/dune/data/users` and put the cvmfs location in cvmfs.location in `$DIRECTORY` +will take a while, produce a tarball on `/exp/dune/data/users/$USER/` and put the cvmfs location in cvmfs.location in `$DIRECTORY` Then edit `job_config.sh` to reflect the # of events you want and other run-time parameters. +### Test your jobscript interactively + +[test_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/test_workflow.sh) + +results will show up in the 'tmp' area on your local machine. + + ### Submit the job [submit_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_workflow.sh) +This one writes to `/pnfs/dune/scratch` + ~~~ ./submit_workflow.sh ~~~ +{: ..language-bash} + +[submit_workflow_rucio.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/submit_workflow_rucio.sh) + +This one writes to a rucio location specified by `$NAMESPACE` + +~~~ +./submit_workflow_rucio.sh +~~~ +{: ..language-bash} + +### after submission -should get a workflow number back +You should get a workflow number back go to [justin](https://dunejustin.fnal.gov/dashboard/?method=list-workflows) From 5cd056b20e377df48a18e99544da1842b95514fa Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Sat, 21 Feb 2026 11:41:53 -0800 Subject: [PATCH 25/27] justIN --- _episodes/02-submit-jobs-w-justin.md | 18 +- _episodes/08-justin-job-submission.md | 405 +------------------------- 2 files changed, 18 insertions(+), 405 deletions(-) diff --git a/_episodes/02-submit-jobs-w-justin.md b/_episodes/02-submit-jobs-w-justin.md index 17bc29a..5227522 100644 --- a/_episodes/02-submit-jobs-w-justin.md +++ b/_episodes/02-submit-jobs-w-justin.md @@ -1,27 +1,27 @@ --- -title: New Justin Job Submission System +title: New justIN Job Submission System teaching: 20 exercises: 0 questions: -- How to submit realistic grid jobs with JustIn +- How to submit realistic grid jobs with justIN objectives: -- Demonstrate use of [justIn](https://dunejustin.fnal.gov) for job submission with more complicated setups. +- Demonstrate use of [justIN](https://dunejustin.fnal.gov) for job submission with more complicated setups. keypoints: - Always, always, always prestage input datasets. No exceptions. --- -# PLEASE USE THE NEW [justIn](https://dunejustin.fnal.gov) SYSTEM INSTEAD OF POMS +# PLEASE USE THE NEW [justIN](https://dunejustin.fnal.gov) SYSTEM INSTEAD OF POMS -__A simple [justIn](https://dunejustin.fnal.gov) Tutorial is currently in docdb at: [JustIn Tutorial](https://docs.dunescience.org/cgi-bin/sso/RetrieveFile?docid=30145)__ +__A simple [justIN](https://dunejustin.fnal.gov) Tutorial is currently in docdb at: [justIN Tutorial](https://docs.dunescience.org/cgi-bin/sso/RetrieveFile?docid=30145)__ A more detailed tutorial is available at: -[JustIn Docs](https://dunejustin.fnal.gov/docs/) +[justIN Docs](https://dunejustin.fnal.gov/docs/) -The [justIn](https://dunejustin.fnal.gov) system is described in detail at: +The [justIN](https://dunejustin.fnal.gov) system is described in detail at: -__[JustIn Home](https://dunejustin.fnal.gov/dashboard/)__ +__[justIN Home](https://dunejustIN .fnal.gov/dashboard/)__ -__[JustIn Docs](https://dunejustin.fnal.gov/docs/)__ +__[justIN Docs](https://dunejustin.fnal.gov/docs/)__ > ## Note More documentation coming soon diff --git a/_episodes/08-justin-job-submission.md b/_episodes/08-justin-job-submission.md index c51bd4e..729b9dc 100644 --- a/_episodes/08-justin-job-submission.md +++ b/_episodes/08-justin-job-submission.md @@ -1,5 +1,5 @@ --- -title: justIn Grid Job Submission (UNDER CONSTRUCTION) +title: justIN Grid Job Submission (UNDER CONSTRUCTION) teaching: 65 exercises: 0 questions: @@ -65,7 +65,7 @@ For now, please look at the short version of this sequence at ## Submit a job -Go to [The Justin Tutorial](https://dunejustin.fnal.gov/docs/tutorials.dune.md) +Go to [The justIN Tutorial](https://dunejustin.fnal.gov/docs/tutorials.dune.md) and work up to ["run some hello world jobs"](https://dunejustin.fnal.gov/docs/tutorials.dune.md#run-some-hello-world-jobs) @@ -101,405 +101,18 @@ There are many ways of doing this but by far the best is to use the Rapid Code D If you have finished up the LArSoft follow-up and want to use your own code for this next attempt, feel free to tar it up (you won't need anything besides the localProducts* and work directories) and use your own tar ball in lieu of the one in this example. You will have to change the last line with your own submit file instead of the pre-made one. -First, we should make a tarball. Here is what we can do - -~~~ -cd mybuild -tar --exclude '.git' -czf mybuild.tar.gz * -~~~ - -Follow the instructions at: - -[rapid code distribution to jobs via cvmfs](https://dunejustin.fnal.gov/docs/tutorials.dune.md#rapid-code-distribution-to-jobs-via-cvmfs) - - - - - - -Before we continue, let's examine these files a bit. We will source the first one in our job script, and it will set up the environment for us. - -~~~ -#!/bin/bash - -DIRECTORY=mybuild -# we cannot rely on "whoami" in a grid job. We have no idea what the local username will be. -# Use the GRID_USER environment variable instead (set automatically by jobsub). -USERNAME=${GRID_USER} - -source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh -export WORKDIR=${_CONDOR_JOB_SCRATCH} # if we use the RCDS the our tarball will be placed in $INPUT_TAR_DIR_LOCAL. -if [ ! -d "$WORKDIR" ]; then - export WORKDIR=`echo .` -fi - -source ${INPUT_TAR_DIR_LOCAL}/${DIRECTORY}/localProducts*/setup-grid -mrbslp -~~~ -{: .source} - - - -As you can see, we have switched from the hard-coded directories to directories defined by environment variables; the `INPUT_TAR_DIR_LOCAL` variable will be set for us (see below). -Now, let's actually create our tar file. - -Note how we have excluded the contents of ".git" directories in the various packages, since we don't need any of that in our jobs. It turns out that the .git directory can sometimes account for a substantial fraction of a package's size on disk! - -Then submit another job (in the following we keep the same submit file as above): - -```bash -jobsub_submit -G dune --mail_always -N 1 --memory=2500MB --disk=2GB --expected-lifetime=3h --cpu=1 --tar_file_name=dropbox:///exp/dune/app/users//sep2025tutorial.tar.gz --singularity-image /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest --append_condor_requirements='(TARGET.HAS_Singularity==true&&TARGET.HAS_CVMFS_dune_opensciencegrid_org==true&&TARGET.HAS_CVMFS_larsoft_opensciencegrid_org==true&&TARGET.CVMFS_dune_opensciencegrid_org_REVISION>=1105&&TARGET.HAS_CVMFS_fifeuser1_opensciencegrid_org==true&&TARGET.HAS_CVMFS_fifeuser2_opensciencegrid_org==true&&TARGET.HAS_CVMFS_fifeuser3_opensciencegrid_org==true&&TARGET.HAS_CVMFS_fifeuser4_opensciencegrid_org==true)' -e GFAL_PLUGIN_DIR=/usr/lib64/gfal2-plugins -e GFAL_CONFIG_DIR=/etc/gfal2.d file:///exp/dune/app/users/kherner/run_sep2025tutorial.sh -``` - -You'll see this is very similar to the previous case, but there are some new options: - -* `--tar_file_name=dropbox://` automatically **copies and untars** the given tarball into a directory on the worker node, accessed via the INPUT_TAR_DIR_LOCAL environment variable in the job. The value of INPUT_TAR_DIR_LOCAL is by default $CONDOR_DIR_INPUT/name_of_tar_file_without_extension, so if you have a tar file named e.g. sep2025tutorial.tar.gz, it would be $CONDOR_DIR_INPUT/sep2025tutorial. -* Notice that the `--append_condor_requirements` line is longer now, because we also check for the fifeuser[1-4]. opensciencegrid.org CVMFS repositories. - -The submission output will look something like this: - -~~~ -Attempting to get token from https://htvaultprod.fnal.gov:8200 ... succeeded -Storing bearer token in /tmp/bt_token_dune_Analysis_11469 -Using bearer token located at /tmp/bt_token_dune_Analysis_11469 to authenticate to RCDS -Checking to see if uploaded file is published on RCDS -Could not locate uploaded file on RCDS. Will retry in 30 seconds. -Could not locate uploaded file on RCDS. Will retry in 30 seconds. -Could not locate uploaded file on RCDS. Will retry in 30 seconds. -Found uploaded file on RCDS. -Transferring files to web sandbox... -Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/run_sep2025tutorial.sh [DONE] after 0s -Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/simple.cmd [DONE] after 0s -Copying file:///nashome/k/kherner/.cache/jobsub_lite/js_2023_05_24_224713_9669e535-daf9-496f-8332-c6ec8a4238d9/simple.sh [DONE] after 0s -Submitting job(s). -1 job(s) submitted to cluster 62007523. - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - -Use job id 62007523.0@jobsub01.fnal.gov to retrieve output -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- -~~~ - -Note that the job submission will pause while it uploads the tarball to RCDS, and then it continues normally. - -Now, there's a very small gotcha when using the RCDS, and that is when your job runs, the files in the unzipped tarball are actually placed in your work area as symlinks from the CVMFS version of the file (which is what you want since the whole point is not to have N different copies of everything). -The catch is that if your job script expected to be able to edit one or more of those files within the job, it won't work because the link is to a read-only area. -Fortunately there's a very simple trick you can do in your script before trying to edit any such files: - -~~~ -cp ${INPUT_TAR_DIR_LOCAL}/file_I_want_to_edit mytmpfile # do a cp, not mv -rm ${INPUT_TAR_DIR_LOCAL}file_I_want_to_edit # This really just removes the link -mv mytmpfile file_I_want_to_edit # now it's available as an editable regular file. -~~~ - -You certainly don't want to do this for every file, but for a handful of small text files this is perfectly acceptable and the overall benefits of copying in code via the RCDS far outweigh this small cost. -This can get a little complicated when trying to do it for things several directories down, so it's easiest to have such files in the top level of your tar file. - - - - - -## Monitor your jobs -For all links below, log in with your FNAL Services credentials (FNAL email, not Kerberos password). - -* What DUNE is doing overall: -[https://fifemon.fnal.gov/monitor/d/000000053/experiment-batch-details?orgId=1&var-experiment=dune](https://fifemon.fnal.gov/monitor/d/000000053/experiment-batch-details?orgId=1&var-experiment=dune) - - -* What's going on with only your jobs: -Remember to change the url with your own username and adjust the time range to cover the region of interest. -[https://fifemon.fnal.gov/monitor/d/000000116/user-batch-details?orgId=1&var-cluster=fifebatch&var-user=kherner](https://fifemon.fnal.gov/monitor/d/000000116/user-batch-details?orgId=1&var-cluster=fifebatch&var-user=kherner) - -* Why your jobs are held: -Remember to choose your username in the upper left. -[https://fifemon.fnal.gov/monitor/d/000000146/why-are-my-jobs-held?orgId=1](https://fifemon.fnal.gov/monitor/d/000000146/why-are-my-jobs-held?orgId=1) - -## View the stdout/stderr of our jobs -Here's the link for the history page of the example job: [link](https://fifemon.fnal.gov/monitor/d/000000115/job-cluster-summary?orgId=1&var-cluster=40351757&var-schedd=jobsub01.fnal.gov&from=1611098894726&to=1611271694726). - -Feel free to sub in the link for your own jobs. - -Once there, click "View Sandbox files (job logs)". -In general you want the .out and .err files for stdout and stderr. -The .cmd file can sometimes be useful to see exactly what got passed in to your job. - -[Kibana][kibana] can also provide a lot of information. - -You can also download the job logs from the command line with jobsub_fetchlog: - -```bash -jobsub_fetchlog --jobid=12345678.0@jobsub0N.fnal.gov --unzipdir=some_appropriately_named_directory -``` - -That will download them as a tarball and unzip it into the directory specified by the --unzipdir option. -Of course replace 12345678.0@jobsub0N.fnal.gov with your own job ID. - -> ## Quiz -> -> Download the log of your last submission via jobsub_fetchlog or look it up on the monitoring pages. Then answer the following questions (all should be available in the .out or .err files): -> 1. On what site did your job run? -> 2. How much memory did it use? -> 3. Did it exit abnormally? If so, what was the exit code? -> -{: .solution} - -## Review of best practices in grid jobs (and a bit on the interactive machines) - -* When creating a new workflow or making changes to an existing one, **ALWAYS test with a single job first**. Then go up to 10, etc. Don't submit thousands of jobs immediately and expect things to work. -* **ALWAYS** be sure to prestage your input datasets before launching large sets of jobs. This may become less necesaary in the future as we move to distributed storage locations. -* **Use RCDS**; do not copy tarballs from places like scratch dCache. There's a finite amount of transfer bandwidth available from each dCache pool. If you absolutely cannot use RCDS for a given file, it's better to put it in resilient (but be sure to remove it when you're done!). The same goes for copying files from within your own job script: if you have a large number of jobs looking for a same file, get it from resilient. Remove the copy when no longer needed. Files in resilient dCache that go unaccessed for 45 days are automatically removed. -* Be careful about placing your output files. **NEVER place more than a few thousand files into any one directory inside dCache. That goes for all type of dCache (scratch, persistent, resilient, etc). Subdirectories also count against the total for these purposes, so don't put too many subdirectories at any one level. -* **AVOID** commands like `ifdh ls /some/path` inside grid jobs unless it is absolutely necessary. That is an expensive operation and can cause a lot of pain for many users, especially when a directory has large number of files in it. Remote listings take much, much longer than the corresponding op on a machine where the directory is mounted via NFS. If you just need to verify a directory exists, there are much better ways than ifdh ls, for example the gfal-stat command. Note also that ifdh cp will now, by default, create an output directory if it does not exist (so be careful that you've specified your output string properly). -* Use xrootd when opening files interactively; this is much more stable than simply doing `root /pnfs/dune/... (and in general, do NOT do that...)` -* **NEVER** copy job outputs to a directory in resilient dCache. Remember that they are replicated by a factor of 20! **Any such files are subject to deletion without warning**. -* **NEVER** do hadd on files in `/pnfs` areas unless you're using `xrootd`. I.e. do NOT do hadd out.root `/pnfs/dune/file1 /pnfs/dune/file2 ...` This can cause severe performance degradations. -* Generally aim for output file sizes of 1 GB or greater. dCache is really not a fan of small files. You may need to process multiple input files to get to that size (and we generally encourage that anyway!) -* Very short jobs (measured in minutes) are quite inefficient, especially if you have an input tarball. In general you want to run for at least a few hours, and 8-12 is probably ideal (and of course longer jobs are allowed). Again you may need to process multiple input files, depending on what you're doing, or run multiple workflow stages in the same job. See the POMS section of the tutorial for more details. - -**Side note:** Some people will pass file lists to their jobs instead of using a SAM dataset. We do not recommend that for two reasons: 1) Lists do not protect you from cases where files fall out of cache at the location(s) in your list. When that happens your jobs sit idle waiting for the files to be fetched from tape, which kills your efficiency and blocks resources for others. 2) You miss out on cases where there might be a local copy of the file at the site you're running on, or at least at closer one to your list. So you may end up unecessarily streaming across oceans, whereas using SAM (or later Rucio) will find you closer, local copies when they exist. - -**Another important side note:** If you are used to using other programs for your work such as project.py (which is **NOT** officially supported by DUNE or the Fermilab Scientific Computing Division), there is a helpful tool called [Project-py][project-py-guide] that you can use to convert existing xml into POMS configs, so you don't need to start from scratch! Then you can just switch to using POMS from that point forward. As a reminder, if you use unsupported tools, you are own your own and will receive NO SUPPORT WHATSOEVER. You are still responsible for making sure that your jobs satisfy Fermilab's policy for job efficiency: https://cd-docdb.fnal.gov/cgi-bin/sso/RetrieveFile?docid=7045&filename=FIFE_User_activity_mitigation_policy_20200625.pdf&version=1 - -## The cost of getting it wrong: a cautionary tale - -Earlier in May 2023 there was a fairly significant disruption to FNAL dCache, which resulted in at least five different tickets across four different experiments complaining of poor performance (resulting in jobs going held of exceeding time), timeouts, or other storage-related failures. It's unclear exactly how many jobs were affcted but it was likely in the many thousands. The root cause was a DUNE user running `ifdh ls $OUTDIR 0` to check the existence of a given directory. That command, though it only spits out the directory name, was indeed doing a full internal listing of the contents of $OUTDIR. Normally that's not the end of the world (see the comment in the best practices section), but this directory had over 100,000 files in it! The user was writing all job outputs into the same directory from what we could tell. - -Since the workflow was causing a systemwide disruption we immediately held all of the user's jobs and blocked new submissions until the workflow was re-engineered. Fortunately dCache performance recovered very quickly after that. The user's jobs are running again and they are also much more CPU-efficient than they were before the changes. - -*The bottom line: one single user not following best practices can disrupt the entire system if they get enough jobs running.* EVERYONE is responsible for following best practices. Getting it wrong affects not only you, but your collaborators! - -## A word on the DUNE Global Pool - -DUNE has also created a a global glideinWMS pool similar to the CMS Global Pool that is intended to serve as a single point through which multiple job submission systems (e.g. HTCondor schedulers at sites outside of Fermilab) can have access to the same resources. Jobs using the global pool still run in the exactly the same way as those that don't. We plan to move more and more work over to the global pool in 2023 and priority access to the FermiGrid quota will eventually be given to jobs submitted to the global pool. To switch to the global pool with jobsub, it's simply a matter of adding `--global-pool dune` as an option to your submission command. The only practical difference is that your jobs will come back with IDs of the form NNNNNNN.N@dunegpschedd0X.fnal.gov instead of NNNNNNN.N@jobsub0X.fnal.gov. Again, everything else is identical, so feel free to test it out. +### Temporary short version of an example for custom code. +We're working on a long version of this but please look at these [instructions for running a justIN workflow using your own code]({{ site.baseurl }}/short_submission) for now. -## Making subsets of metacat datasets +### Cool justIN feature -Running across very large number of files puts you at risk of system issues. It is often much nicer to run over several smaller subsets. -Many official metacat definitions are large data collections defined only by their properties and not really suitable for a single job. +justIN has a very useful interactive test command. -You can do the following. Submit your jobs using the skip and limit commands. Here 'namespace:official_dataset' describes the official dataset. - -See [the basics tutorial](https://dune.github.io/computing-basics/03-data-management/index.html#official-datasets-) for information on official datasets. +Here is a test from the short submission example. ~~~ -query="files from namespace:official_dataset skip 0 limit 1000" -query="files from namespace:official_dataset skip 1000 limit 1000" -query="files from namespace:official_dataset skip 2000 limit 1000" -.... +{% include test_workflow.sh %} ~~~ -{: ..language-bash} - - - - - - -## Verify Your Learning: - -> ## Question 01 -> -> What are the differences in environment between Fermilab worker nodes and those at other sites (assuming the site supports Singularity)? ->
    ->
  1. Fermilab workers have additional libraries available.
    2. ->
  2. Worker nodes at other sites have additional libraries installed.
    3. ->
  3. No difference.
    4. ->
-> -> > ## Answer -> > The correct answer is C - No difference. -> > {: .output} -> > Comment: -> {: .solution} -{: .challenge} - -> ## Question 02 -> -> After setting up a new workflow or preparing to run one that has not been exercised in a while, what is an that has not been exercised in a while, what is an appropriate number of test jobs to initially submit? ->
    ->
  1. 1.
    2. ->
  2. 10.
    3. ->
  3. 100.
    4. ->
  4. As many as needed.
    5. ->
-> -> > ## Answer -> > The correct answer is A - 1. -> > {: .output} -> > Comment: -> {: .solution} -{: .challenge} - - -> ## Question 03 -> -> project.py is supported by the Fermilab Scientific Computing Division ->
    ->
  1. True.
    2. ->
  2. False.
    3. ->
-> -> > ## Answer -> > The correct answer is B - False. -> > {: .output} -> > Comment: -> {: .solution} -{: .challenge} - - -> ## Question 04 -> -> What is generally the best way to read in a .root file for analysis within a grid job? ->
    ->
  1. Open with an xrootd URI (root://).
    2. ->
  2. Copy the entire file at the beginning of the job.
    3. ->
  3. Both A and B.
    4. ->
-> -> > ## Answer -> > The correct answer is A - Open with an xrootd URI (root://). -> > {: .output} -> > Comment: -> {: .solution} -{: .challenge} - - -> ## Question 05 -> -> What is the best way to specify your desired operating system and environment in a grid job? ->
    ->
  1. Use the --OS option in jobsub.
    2. ->
  2. Do not specify any OS, but control it with the SingularityImage classad.
    3. ->
  3. Don’t specify anything. The grid does it
    4. ->
  4. None of the Above
    5. ->
-> -> > ## Answer -> > The correct answer is B - Do not specify any OS, but control it with the SingularityImage classad. -> > {: .output} -> > Comment: -> {: .solution} -{: .challenge} - - -> ## Question 06 -> -> What is the best way to copy custom code into a grid job? ->
    ->
  1. Use the RCDS (i.e. --tar_file_name=dropbox://foo/bar/) and stage the file in via CVMFS.
    2. ->
  2. Copy a tarball to /pnfs/dune/scratch/users/username.
    3. ->
  3. Copy a tarball to /pnfs/dune/persistent/users/username.
    4. ->
  4. Copy a tarball to /pnfs/dune/resilient/users/username.
    5. ->
  5. None of the Above
    6. ->
-> -> > ## Answer -> > The correct answer is A - Use the RCDS (i.e. --tar_file_name=dropbox://foo/bar/) and stage the file in via CVMFS. -> > {: .output} -> > Comment: -> {: .solution} -{: .challenge} - - - -## Further Reading -Some more background material on these topics (including some examples of why certain things are bad) is in these links: - - -[December 2022 jobsub_lite demo and information session](https://indico.fnal.gov/event/57514/) - -[January 2023 additional experiment feedback session on jobsub_lite]( ) - -[Wiki page listing differences between jobsub_lite and legacy jobsub](https://fifewiki.fnal.gov/wiki/Differences_between_jobsub_lite_and_legacy_jobsub_client/server) - -[2021 Intensity Frontier Summer School](https://indico.fnal.gov/event/49414) - -[The Glidein-based Workflow Management System]( https://glideinwms.fnal.gov/doc.prd/index.html ) - -[Introduction to Docker](https://hsf-training.github.io/hsf-training-docker/index.html) - -[job-autorelease]: https://cdcvs.fnal.gov/redmine/projects/fife/wiki/Job_autorelease - -[redmine-wiki-jobsub]: https://cdcvs.fnal.gov/redmine/projects/jobsub/wiki - -[redmine-wiki-using-the-client]: https://cdcvs.fnal.gov/redmine/projects/jobsub/wiki/Using_the_Client - -[fifemon-dune]: https://fifemon.fnal.gov/monitor/d/000000053/experiment-batch-details?orgId=1&var-experiment=dune - -[fifemon-userjobs]: https://fifemon.fnal.gov/monitor/d/000000116/user-batch-details?orgId=1&var-cluster=fifebatch&var-user=kherner - -[fifemon-whyheld]: https://fifemon.fnal.gov/monitor/d/000000146/why-are-my-jobs-held?orgId=1 - -[kibana]: https://fifemon.fnal.gov/kibana/goto/8f432d2e4a40cbf81d3072d9c9d688a6 - -[poms-page-ana]: https://pomsgpvm01.fnal.gov/poms/index/dune/analysis/ - -[poms-user-doc]: https://cdcvs.fnal.gov/redmine/projects/prod_mgmt_db/wiki/POMS_User_Documentation - -[fife-launch-ref]: https://cdcvs.fnal.gov/redmine/projects/fife_utils/wiki/Fife_launch_Reference - -[poms-campaign-stage-info]: https://pomsgpvm01.fnal.gov/poms/campaign_stage_info/dune/analysis?campaign_stage_id=9023 - -[project-py-guide]: https://cdcvs.fnal.gov/redmine/projects/project-py/wiki/Project-py_guide - -[DUNE_computing_tutorial_advanced_topics_20200129]: https://indico.fnal.gov/event/20144/contributions/55932/attachments/34945/42690/DUNE_computing_tutorial_advanced_topics_and_best_practices_20200129.pdf - -{%include links.md%} +it reads in a tarball from an area `$DUNEDATA` and writes output to a tmp area on your interactive machine. It works very well at emulating a grid job. \ No newline at end of file From 7b068dfb95557fef16eb5b5077b793a11fe9ca49 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Mon, 23 Feb 2026 11:26:14 -0800 Subject: [PATCH 26/27] more instructions --- _extras/short_submission.md | 27 +++++++++++++++++++++------ index.md | 2 +- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/_extras/short_submission.md b/_extras/short_submission.md index 084e8d0..666adfa 100644 --- a/_extras/short_submission.md +++ b/_extras/short_submission.md @@ -104,8 +104,29 @@ If you have changed any scripts or code, you must redo this. will take a while, produce a tarball on `/exp/dune/data/users/$USER/` and put the cvmfs location in cvmfs.location in `$DIRECTORY` +### Configure your job with job_config.sh + Then edit `job_config.sh` to reflect the # of events you want and other run-time parameters. +#### Details of job_config.sh + +Here is what is in job_config.sh +~~~ +{% include job_config.sh %} +~~~ + +- `FCL_FILE`= the top level fcl file - assumes it is in DIRECTORY or in the `PHICL_FILE_PATH` +- `OUTPUT_DATA_TIER1` data_tier for artroot output (full-reconstructed, ...) +- `OUTPUT_DATA_TIER2` data_tier for root output - normally plain root-tuple +- `MQL` Metacat query that you wish to run over +- `APP_TAG` this is a tag that goes into the output filename, like reco2, ana ... +- `DESCRIPTION` the jobname that shows up in justIN +- `USERF` make certain the grid knows who your are without overwriting whatever internal `USER` it has +- `NUM_EVENTS` the `-n` argument of larsoft +- `FNALURL` sends output to subdirectories of your area on scratch +- `NAMESPACE` rucio/metacat namespace for your output, normally `usertests` or possibly your username or physics group unless you are doing production. + + ### Test your jobscript interactively [test_workflow.sh](https://github.com/hschellman/computing-basics-batch-devel/blob/gh-pages/_includes/test_workflow.sh) @@ -140,9 +161,3 @@ You should get a workflow number back go to [justin](https://dunejustin.fnal.gov/dashboard/?method=list-workflows) to track your job. - -[internal link](/files/setup-grid) - - - - diff --git a/index.md b/index.md index e00e8bf..8550362 100644 --- a/index.md +++ b/index.md @@ -23,7 +23,7 @@ This tutorial will teach you the basics of DUNE batch computing. Instructors will engage students with hands-on lessons focused in three areas: -1. The [justIn](https://dunejustin.fnal.gov) batch system +1. The [justIN](https://dunejustin.fnal.gov) batch system 2. The jobsub batch system From 115b39110072ae0a4a4bc3423976480787d54286 Mon Sep 17 00:00:00 2001 From: Heidi Schellman <33669005+hschellman@users.noreply.github.com> Date: Mon, 13 Apr 2026 13:34:37 -0700 Subject: [PATCH 27/27] cleaner version of part 8 --- _episodes/08-justin-job-submission.md | 38 +++++++-------------------- 1 file changed, 9 insertions(+), 29 deletions(-) diff --git a/_episodes/08-justin-job-submission.md b/_episodes/08-justin-job-submission.md index 729b9dc..9985b5d 100644 --- a/_episodes/08-justin-job-submission.md +++ b/_episodes/08-justin-job-submission.md @@ -31,39 +31,15 @@ The video from the two day version of this training in May 2022 is provided [her --> -#### Live Notes - - - - - -For now, please look at the short version of this sequence at -[Short Submission Runthrough]({{ site.baseurl }}/short_submission) - -## Submit a job +## First learn the basics of Justin Submit a job Go to [The justIN Tutorial](https://dunejustin.fnal.gov/docs/tutorials.dune.md) @@ -89,6 +65,8 @@ Then work through ## Submit a job using the tarball containing custom code + + First off, a very important point: for running analysis jobs, **you may not actually need to pass an input tarball**, especially if you are just using code from the base release and you don't actually modify any of it. In that case, it is much more efficient to use everything from the release and refrain from using a tarball. All you need to do is set up any required software from CVMFS (e.g. dunetpc and/or protoduneana), and you are ready to go. If you're just modifying a fcl file, for example, but no code, it's actually more efficient to copy just the fcl(s) you're changing to the scratch directory within the job, and edit them as part of your job script (copies of a fcl file in the current working directory have priority over others by default). @@ -98,8 +76,6 @@ We need a way to efficiently get code into jobs without overwhelming our data tr We have to make a few minor changes to the scripts you made in the previous tutorial section, generate a tarball, and invoke the proper jobsub options to get that into your job. There are many ways of doing this but by far the best is to use the Rapid Code Distribution Service (RCDS), as shown in our example. -If you have finished up the LArSoft follow-up and want to use your own code for this next attempt, feel free to tar it up (you won't need anything besides the localProducts* and work directories) and use your own tar ball in lieu of the one in this example. -You will have to change the last line with your own submit file instead of the pre-made one. ### Temporary short version of an example for custom code. @@ -115,4 +91,8 @@ Here is a test from the short submission example. {% include test_workflow.sh %} ~~~ -it reads in a tarball from an area `$DUNEDATA` and writes output to a tmp area on your interactive machine. It works very well at emulating a grid job. \ No newline at end of file +it reads in a tarball from an area `$DUNEDATA` and writes output to a tmp area on your interactive machine. It works very well at emulating a grid job. + +## Did your job work? + +If not please ask over at #computing-questions in Slack \ No newline at end of file