Skip to content

Latest commit

 

History

History
425 lines (273 loc) · 21.2 KB

ARCHITECTURE.md

File metadata and controls

425 lines (273 loc) · 21.2 KB
Raw

Project structure

This document explains in brief the various components of KenDB3. You do not need to be familiar with the underlying technologies.

KenDB3 has a non-trivial project structure that combines backend-centric Django framework for Python with a modern frontend powered by TypeScript and Sass. It was inspired by the "hybrid" approach discussed in the well-known SaaS Pegasus article.

Overview

Overview diagram of KenDB3 architecture

KenDB3/
├── api_lib/               Django app; handles backend-frontend communication
├── autogenerated_static/  directory populated by frontend pipeline and auto-
│                            generators at startup
├── babel.config.json
├── .browserslistrc        list of target browsers for frontend pipeline
├── db.sqlite3             default location for developer database
├── Dockerfile
├── docs/                  top-level documentations
├── .git/
├── .gitea/                CI files for Gitea Actions (analogous to .github)
├── .gitignore
├── fixtures/              Sample data for the developer database, see
│                            MANAGING_SAMPLE_DATA.md for details
├── javascript_pipeline/   Django app; frontend pipeline implementation
├── kendb3/                Django project
├── LICENSE
├── manage.py              Django-provided site management script
├── node_modules/          default location for NPM (frontend) dependencies
├── package.json           NPM dependency list
├── package-lock.json      NPM dependency hashes
├── profiles/              Django app; implements user profiles
├── README.md
├── requirements-production.txt  Pip (backend) dependency list for production
├── requirements.txt       Pip dependency list for development
├── static/                static assets
├── submissions/           Django app; implements submissions
├── templates/             directory with shared Django templates
├── tsconfig.json          TypeScript configuration and frontend aliases
├── util/                  Django app; miscellaneous Django utilities
├── viewmodule/            Django app; implements viewmodules
├── webpack.config.js
└── .windcorp.ru           deployment settings

Backend

KenDB3 is a Django site. Django is a comprehensive framework for building websites; it comes with features such as ORM, HTML templates, user management, and an admin panel.

Django sites have a Django project and multiple Django apps, each in a directory under repository root. Some Django apps are external dependencies installed as Python packages.

Directories that are relevant to Django:

KenDB3/
│
│# Django project
├── kendb3/
│
│# Local Django apps that provide content
├── profiles/              implements user profiles
├── submissions/           implements submissions
│
│# Local Django apps that provide supporting libraries
├── api_lib/               handles backend-frontend communication
├── javascript_pipeline/   frontend pipeline implementation
├── viewmodule/            viewmodules for frontend
├── util/                  misc utilities that need to be inside a Django app
│
│# Templates
├── templates/             shared Django templates
├── <all Django apps>/
│   └── templates/         app-specific Django templates
│
│# Static files
├── autogenerated_static/  directory populated by frontend pipeline and auto-
│                            generators at startup
├── static/                static assets
│
│# Other
├── db.sqlite3             default location for developer database
├── manage.py              Django-provided site management script
├── requirements-production.txt  dependencies for production
└── requirements.txt       dependencies for development

Django project

The Django project resides in kendb3/.

  • urls.py defines the top-level router. It maps Django apps to their URL subpaths. Apps mapped to root URL ('') define their own subpaths. See the apps' own urls.py for more details on routing.

  • settings.py is a configuration file for Django. It contains a fair amount of logic for determining some of the values. It additionally declares the Django apps and other plugins (middleware, etc.) that are part of this Django site. All new Django apps must be activated via addition to this file.

  • wsgi.py and asgi.py provide support to correspondingly-named Python webserver interfaces; KenDB3 is designed to be deployed with WSGI.

Django content apps

profiles and submissions are content Django apps. They share a common structure:

  • templates/ contains Django templates used to generate HTML,
  • webpack_src/ contains relevant files for frontend pipeline,
  • migrations/ will contain database migration scripts in the future. These directories are gitignored except for migrations/__init__.py.
  • models.py defines Django models (entities, data structures),
  • views.py implements dynamic pages,
  • urls.py describes a mapping from URLs to Django views,
  • admin.py configures Django admin panel.

Django support apps

  • javascript_pipeline implements the frontend pipeline. It is an adapter for the frontend toolchain. See the section on frontend pipeline for details.

    javascript_pipeline additionally provides the interface for autogenerators.

    Note that the entrypoint to the logic of this app is in apps.py. This Django app does not serve any URLs.

  • api_lib is a fairly sophisticated, if a little inelegant, custom object-relational mapper for KenDB3. It simplifies retrieving data from the server for the frontend. See Middleware section for details.

    This Django app serves api/v0/get/ URLs.

  • viewmodules provides a pure frontend framework for creating pages with dynamic content. See the dedicated Viewmodules section for details.

    This Django app does not serve any URLs.

Frontend

KenDB3 frontend is a complex system.

The transpilation, pre-processing and packaging of the above is managed with Webpack. Webpack execution is controlled through the javascript_pipeline Django app. See Frontend pipeline for details on the build process for frontend.

Django templates

Django templates are used to render base HTML pages on the server. This includes injecting JSON payloads.

Django templates can be found in multiple locations:

KenDB3/
│
│# Templates
├── templates/             shared Django templates
└── <all Django apps>/
    └── templates/         app-specific Django templates

According to Django convention, HTML files reside inside directories named after the correspoding Django app. Therefore, template index.html of submissions Django app is located in submissions/templates/submissions/index.html.

All HTML pages must extend [templates/]common/base.html. Pages that use viewmodules must extend [viewmodule/templates/]viewmodules/viewmodule_base.html.

Aliases

TypeScript and Sass files are stored inside relevant Django apps in directories webpack_src. These folders are assigned convenient aliases through tsconfig.js when they are designed to be reused.

For example,

{
  "compilerOptions": {
    "paths": {
      "profiles": ["./profiles/webpack_src"],
    }
  }
}

resolves @use 'profiles/inline_profile'; to ./profiles/webpack_src/_inline_profile.scss.

TypeScript

TypeScript is JavaScript with type checking. All browsers scripts must be written in TypeScript.

util/common is a good starting point to get familiar with how TypeScript is used in KenDB3.

Every HTML page should have a single corresponding TypeScript file, even if it just consists of a few import statements.

TypeScript outputs are described explicitly in webpack.config.js. As its name suggests, insertEverywhere path must be inserted into every Webpack entry:

    entry: {
      submissions: [
        './submissions/webpack_src/submissions.ts',
        insertEverywhere,
      ]
    }

See Aliases above for file locations and import resolution rules.

Sass

Sass adds many quality-of-life features for writing CSS code. SCSS syntax is used in KenDB3; it is a superset of regular CSS.

Most SCSS files are named with a leading underscore: _filename.scss. This signals to preprocessors that these files are meant to be imported only. Files without a leading underscore, such as submissions.scss, each result in a packaged asset.

Every HTML page should have a single corresponding SCSS file, even if it just consists of a few use statements. util/base must be included in every page.

See Aliases above for file locations and @use resolution rules.

Viewmodules

Viewmodules is a custom framework for creating single-page applications. It uses JsRender to render HTML, although it does not use JsViews or JsObservable.

Viewmodule-managed pages serve a tree of URL paths. Subpath patterns are assigned TypeScript Viewmodule objects. When the user navigates to a subpath managed by the current viewmodule-enabled page, the framework erases the contents of the previously installed viewmodule and invokes the install method of the selected viewmodule.

Django templates of pages that use viewmodules must extend viewmodules/viewmodule_base.html.

See documentation for more details, see submissions.ts for a use example.

Middleware

In this document, "middleware" refers to the JSON communication layer between backend and frontend. Not to be confused with Django middleware.

This role is fulfilled by the custom api_lib Django app. It creates three convenient interfaces: a Python interface for the backend, a TypeScript interface for the frontend, and a set of HTTP endpoints that form a public API.

The interfaces operate on several models (submissions, submission revisions, Minecraft versions, etc.) Each model has a known set of fields. Each field is part of one or more field groups. Model queries specify the field group to retrieve.

For example, Submission has a basic field group that only includes data necessary to populate a submission list entry and a * field group that includes all supported fields.

Backend view

api_fields.py provides a Python interface to make Django models available via the API. See the file for documentation.

api_fields makes use of Python annotations in an unconventional manner: Django model fields should be annotated to be registered. This conflicts with the use of type hints in Python.

In addition to automatic data management, Django templates and views can inject objects into HTML using api_lib.api_server.inject. Injections reduce the number of API requests and speed up page render.

HTTP API

The models registered with api_fields are made available via HTTP endpoints api/v0/get/<model-name>. Interface of v0 endpoints is subject to backwards-incompatible changes and eventual removal in favor of stable v1 endpoints.

Frontend view

The models available via the HTTP API are exposed to TypeScript code via dataman. dataman processes injections and requests more objects from the server as they become needed.

dataman relies in part on autogenerated code for model definitions in api_lib/webpack_src/models.autogenerated.ts.

Database

At this moment the database used in both staging and development environments is SQLite3. It will eventually be replaced with Dolt, a SQL database with better performance and revision control.

Build system

Several actions happen before or during initialization of KenDB3 backend.

Django management

Django projects are managed using an unmodified Django-provided manage.py.

Run this Python script as an executable:

./manage.sh help

See the section on Dependency management for backend if the help command above produces an error.

Useful commands:

  • ./manage.py runserver: start a development server at localhost:8000
  • ALLOWED_HOSTS='*' ./manage.py runserver 0:8000: start a development server reachable from LAN (for testing with mobile devices)
  • ./manage.py startapp APP_NAME: create a new Django app named APP_NAME from a template; it has to be enabled in kendb3/settings.py separately.

In production, Django is served by Gunicorn. See DEPLOYMENT.md for details.

Migrations

Django database migrations are Python scripts that update the structure of a database when update is rolled out. They are usually autogenerated by Django but human involvement is not uncommon.

  • To generate migrations based on changes to models, run ./manage.py makemigrations.
  • To apply existing migrations that have not been applied yet, run ./manage.py migrate.

For developers, it is recommended to run makemigrations and migrate every time before starting a server.

Each Django app has a migrations directory where migration files are located:

# Example for submissions Django app
submissions/migrations/
├── 0001_initial.py
├── 0002_submissionrevision_intended_solution_url_and_more.py
├── ...
└── __init__.py

It is good practice to commit migration scripts to version control. In KenDB3, migration scripts are currently gitignored instead, because OLEGSHA believes this slows down development. When a "main" website instance is deployed, developers should start to commit migrations.

Frontend pipeline

TypeScript and SCSS files are transpiled and packaged by Webpack via NPM using numerous plugins.

See sections on TypeScript and Sass for layout of sources. Packaged assets are stored in autogenerated_static. Packaged assets are gitignored.

Webpack is automatically controlled by Django app javascript_pipeline. Its apps.py launches a blocking Webpack build (in production mode) or an async Webpack development daemon (in development mode) during Django initialization if it is deemed necessary. This system is referred to as the frontend pipeline.

The execution of frontend pipeline can be controlled manually if necessary. See javascript_pipeline/apps.py for details. However, developers should not normally need to interact with it.

Autogenerated files

In addition to various Django features and the frontend pipeline, a few custom asset generators are used for various tasks. See javascript_pipeline/autogenerators.py for details.

Autogenerators output files in autogenerated_static or add .autogenerated to the filename. Autogenerated resources are gitignored.

Autogenerators run every time Webpack is about to start. Resources generated by autogenerators are visible to Webpack.

All existing autogenerators:

Static files

Static files are served to HTTP clients without any modification. They are located in static.

Static files are handled by static_files Django app. In development mode, they are served by static_files directly. In production mode, WhiteNoise serves them instead.

Dependency management

To run or develop the project, both backend and frontend dependencies need to be installed. See main README.md for a copy-and-paste setup shell script.

Backend

Backend dependencies are managed with pip, the Python package manager. Development Python dependencies are stored in pip freeze format in requirements.txt. Production dependencies are found in requirements-production.txt.

After installing, updating or removing dependencies, use pip freeze >requirements.txt to save your changes. Update production requirements accordingly.

Frontend

Frontend dependencies are managed with NPM, a common web development tool and package manager. Dependencies and version requirements are stored in package.json, dependency hashes are stored in package-lock.json. Both of these files are committed to version control.

NPM dependency changes are saved automatically.

NPM may report detected vulnerabilities in dependencies. If you see such messages when interacting with NPM, please open an issue.

CI/CD

KenDB3 is designed to be deployed on kendb.windcorp.ru. Because of that, some deployment internals are stored in the repository. This approach is simpler and safer than managing host infrastructure manually, especially when dealing with rollbacks and differing staging and main instances.

See DEPLOYMENT.md for an overview of KenDB3's deployment.

Docker

KenDB3 is packaged as a Docker image. Docker containerizes the application using lightweight virtual machines known as containers, ensuring it is decoupled and isolated from its host.

Docker packaging is configured in Dockerfile. Although it is primarily designed for CI/CD usage, developers may find it useful in some circumstances.

Gitea Actions

Gitea Actions, similar and almost fully compatible with GitHub Actions, is the CI/CD system of choice.

Gitea repository at gitea.windcorp.ru uses the configuration in .gitea to build and deploy new commits automatically.

windcorp.ru internals

Gitea Actions is heavily sandboxed and thus limited. Another smaller system is deployed on host that receives the build artifacts and configurations from Gitea Actions and starts the two instances of the website.

Files in .windcorp.ru are only useful to the host deployment system: