Skip to content

README.md

Latest commit

 

History

History
133 lines (95 loc) · 4.31 KB

README.md

File metadata and controls

133 lines (95 loc) · 4.31 KB

BaseEvalutionFunctionLayer

Base docker image for evaluation functions coded in python. This layer cannot function alone, it needs to be extended in a specific way by the evaluation function it supports.

This layer implements the behaviour common to all evaluation functions:

  • Request and response schema validation (legacy JSON schemas and muEd OpenAPI spec)
  • Unit testing setup
  • Dual API support: legacy command-based API and muEd path-based API

Note: user-defined files are those provided by the evaluation function code meant to extend this layer

Behaviour and Usage

The handler inspects the incoming request path (rawPath / path) to decide which API to invoke:

Path Handler
/evaluate muEd handler (OpenAPI-validated)
/evaluate/health muEd healthcheck
anything else Legacy handler (JSON-schema-validated)

Legacy API

Commands are passed in the command header of each request. By default (if no header is present), the function will run the eval command.

  • eval: calls the evaluation function in the user-defined evaluation.py file.
  • preview: calls the preview function in the user-defined preview.py file.
  • healthcheck: runs all unittests for schema testing as well as user-defined tests in evaluation_tests.py.
  • docs: returns the docs.md user-defined file.

muEd API

Requests to /evaluate and /evaluate/health are validated against the muEd OpenAPI spec (schemas/muEd/openapi-v0_1_0.yml).

  • POST /evaluate: runs evaluation. If preSubmissionFeedback.enabled is true, returns pre-submission feedback instead of a full evaluation result.
  • GET /evaluate/health: runs the healthcheck suite.

Submission Types

The muEd API supports the following submission types. The handler extracts the submission content from the corresponding key in the content object:

Type Content key
MATH expression
TEXT text
CODE code
MODEL model
OTHER value

If the primary content key is absent, the handler falls back to the value key. An error is raised only if neither key is present.

Schemas

Schemas are organised into two trees:

  • schemas/legacy/ — JSON schemas for legacy request/response validation
  • schemas/muEd/openapi-v0_1_0.yml — OpenAPI spec for the muEd API

Note: title is not a required field in the Task schema.

Requirements from the superseding layer

This function makes references to files and functions which don't exist yet in this layer - those need to be provided by the superseding layer. They're shown here in the way a dockerfile might be extending it.

FROM ###THIS IMAGE###

WORKDIR /app

# Install requirements if your evaluation function needs them
COPY requirements.txt .
RUN pip3 install -r requirements.txt

# Copy the evaluation and testing scripts
COPY evaluation.py ./app/
COPY evaluation_tests.py ./app/

# Copy Documentation
COPY docs.md ./app/

# Set permissions so files and directories can be accessed on AWS
RUN chmod 644 $(find . -type f)
RUN chmod 755 $(find . -type d)

# The entrypoint for AWS is to invoke the handler function within the app package
CMD [ "/app/app.handler" ]

Operating Container Structure

Since this is only just a base layer for eval functions, the repo's file structure won't match the file structure inside the built image, which can get confusing at times. This is what the /app/ directory (where all our data is contained) will look like for an operational function:

|____requirements.txt
|____base_requirements.txt
|____app
   |______init__.py
   |____tests
   |  |______init__.py
   |  |____requests.py
   |  |____responses.py
   |  |____handling.py
   |  |____commands.py
   |  |____docs.py
   |  |____parse.py
   |____tools
   |  |______init__.py
   |  |____commands.py
   |  |____validate.py
   |  |____docs.py
   |  |____parse.py
   |  |____healthcheck.py
   |  |____utils.py
   |____docs.md
   |____handler.py
   |____evaluation_tests.py
   |____evaluation.py

Dev Notes

Can run the following command to look around the container of a running function

docker exec -it eval-function bash

From a container which exposes port 8080 to the real port 9000, requests can be made to the function using the url:

http://localhost:9000/2015-03-31/functions/function/invocations