Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -62,59 +62,19 @@ following example.
:alt: A learner's view of a programming problem that uses an external grader,
with an incorrect result.

.. _External Grader Requirements:

External Grader Requirements
============================

The course team builds and deploys the external grader.
Your site operations team :ref:`builds and deploys the external grader <Enable XQueue>`.

When you build your external grader, keep the following requirements in mind.

* To communicate with XQueue, the grader must accept and return JSON objects.
For more information, see :ref:`XQueue JSON Objects`.

* The external grader must actively retrieve, or pull, submissions from the
XQueue through a RESTful interface at regular intervals.

* The grader must be scalable. Many learners might submit responses at one
time, such as shortly before an exam is due. The grader must be able to
process many submissions in a limited time, without failure or unexpected
delays.

* The grader must not constantly send queries unless the grader detects that
the ``queuelen`` value is growing. If XQueue receives too many requests from
a grader, the XQueue system administrator may apply throttling to the grader.

* The grader must implement security features.

You are responsible for the server that runs the code that learners submit.
Your system must protect against learners who might submit malicious code and
ensure that the external grader runs only code that is relevant to the course
problems. Your specific security implementation depends on the programming
language that you are using and your deployment architecture. You must ensure
that malicious code will not damage your server.


* You must have a plan to immediately notify the team that operates your
grader, as well as your Open edX system administrator, if the grader fails.
You must also have a process to quickly identify the cause of the failure,
whether the problem is with your grader or XQueue.

If you know the grader will be unavailable at a certain time for maintenance,
you should :ref:`add a course update <Add a Course Update>`.

* You must test the grader thoroughly before your course starts. Be sure to
test incorrect code as well as correct code to ensure that the grader
responds with appropriate scores and messages.

.. seealso::


:ref:`External Grader Problem Requirements` (reference)

:ref:`Add an External Grader Problem` (how-to)

:ref:`Enable XQueue` (site operators how-to)


**Maintenance chart**

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,24 +15,92 @@ To create a code response problem in Studio, follow these steps.

#. In the new component, select **Edit**.

#. In the problem editor, enter the online learning XML (OLX) definition for
#. In the problem editor, enter the :ref:`online learning XML (OLX) definition <OLX Definition>` for
the problem that you created.

#. Select **Save**.

#. Test the problem in the LMS.

.. note::

To validate your external grader and test a problem, you must view the
component in a published unit in the LMS. If you test a problem in Studio,
you receive an “Error: No grader has been set up for this problem” message.

.. _OLX Definition:

OLX Definition
==============

To create an external grader problem in Studio, you create an Open Learning XML
(OLX) definition of the problem, and then add the code to a problem component.

.. note::

Your site operations team will need to first :ref:`Enable XQueue`, and then give you the ``queuename`` to use in your problem.

The OLX definition of a problem contains the following information.

* A ``queuename`` attribute that specifies the name of the XQueue that your site operations team
created.

* A ``<label>`` element that contains the instructions for the problem.

* The type of input that the problem accepts, specified as one of two elements.

* ``<textbox>``: The learner enters code in a browser field while the learner
views the course unit.

* ``<filesubmission>``: The learner attaches and submits a code file in the
unit.

* (optional) An element that contains a JSON object that you send to the
external grader. For example, you can use the ``<grader_payload>`` element to
tell the grader which tests to run for a problem.

The following example shows the OLX definition of a problem that uses an
external grader.

::

<problem>
<coderesponse queuename="my_course_queue">
<label>Write a program that prints "hello world".</label>
<textbox rows="10" cols="80" mode="python" tabsize="4"/>
<codeparam>
<initial_display>
# students write your program here
print ""
</initial_display>
<answer_display>
print "hello world"
</answer_display>
<grader_payload>
{"output": "hello world", "max_length": 2}
</grader_payload>
</codeparam>
</coderesponse>
</problem>

.. _Testing the Grader:

Testing the Grader
==================

You must test the grader thoroughly, via the LMS interface, before your course
starts. Be sure to test incorrect code as well as correct code to ensure that
the grader responds with appropriate scores and messages.

If you know the grader will be unavailable at a certain time for maintenance,
you might consider :ref:`adding a course update <Add a Course Update>`.


.. seealso::


:ref:`About External Grader Problems` (concept)

:ref:`External Grader Problem Requirements` (reference)
:ref:`Enable XQueue` (site operators how-to)


**Maintenance chart**
Expand Down
1 change: 0 additions & 1 deletion source/educators/navigation/components_activities.rst
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,6 @@ External Grader
:glob:

../concepts/exercise_tools/about_external_graders.rst
../references/course_development/exercise_tools/external_grader_requirements.rst
../how-tos/course_development/exercise_tools/add_external_grader.rst

Gene Explorer Tool
Expand Down

This file was deleted.

Loading