SP00 - Sprint 0 - Final Project

Development Environment

Initial Development Environment

  1. In a terminal on your host system, not within VSCode, create a new directory with the name of your team’s final project repository and cd into it:
    • mkdir comp423-final-project
    • cd comp423-final-project
  2. Initialize the new empty directory as a git repository:
    • git init
  3. Add a git remote repository named upstream bound to the public, open source CSXL web app repository:
    • git remote add upstream https://github.com/unc-csxl/csxl.unc.edu.git
  4. Pull the main branch from upstream:
    • git pull upstream main
  5. Add a git remote repository named origin bound to your team’s private final project repository. You will need to find the correct repository URL. Open your team’s final project repository on GitHub in a web browser, click the down triangle on the green Code button, and copy the Clone URL. Use this URL in the placeholder below:
    • git remote add origin TEAM_REPO_URL_PLACEHOLDER
  6. Push main to origin. If you are the first member on the team to do this, you will see the repository’s history being pushed up to GitHub. If you are not the first member on the team to do this, you may see an error or “Everything up to date”:
    • git push origin main
  7. Your team will work off of a staging branch named stage, rather than the main branch. Go ahead and establish a stage branch locally and push it:
    • git fetch --all
    • git branch stage
    • git switch stage
    • git pull origin stage
    • git push origin stage

GitHub Repository Configuration

Only one member of your team needs to setup the following repository branch settings:

  1. Settings > Branches
  2. Change Default Branch to stage
    • Press the swap/switch button (not the pencil!) and select stage
    • If you do not see stage, be sure you completed all the steps above
  3. Add Branch Protection Rule to prevent any pushing to main
    1. Branch name pattern: main
    2. (Check) Lock branch
    3. (Check) Do not allow bypassing the above settings
    4. Save Changes
  4. Add Branch Protection Rule to enforce a professional team workflow on stage
    1. Branch name pattern: stage
    2. (Check) Require a pull request before merging
      1. (Check) Require approvals: 1 required
      2. (Check) Dismiss stale pull request approvals when new commits are pushed
      3. (Check) Require approval of the most recent reviewable push
    3. (Check) Require conversation resolution before merging
    4. (Check) Require linear history
    5. (Check) Do not allow bypassing the above settings
    6. Save Changes

Setting up your Development Environment

Open your project locally, without opening it in a Dev Container at first, and follow the instructions to get started found in the project’s docs directory: docs/get_started.md

Understanding User Authentication and Authorization

See the official documentation on CSXL.unc.edu Authentication and Authorization for feature development.

Sprint 0 Expectations

  1. “Minimum Viable” Data Models, Entities, and Dev Data merged into stage branch and added to database via backend/script/reset_database.py.

  2. Implement at least one story across all layers, running, and demoable on CloudApps.

    • Add an entry to NavigationComponent on the front-end so that you can reach your feature from the sidebar.
    • Aiming for a “view” story here, as opposed to an update/edit story, is advisable. For example, viewing upcoming workshops, a list of clubs, rooms with availability, etc.

  3. Everything committed and merged from #2 is properly documented, tested, and up to standards.

    • Backend service layer needs unit tests that cover methods all methods and functionality implemented.
    • Backend service layer classes should not have any HTTP concerns. Similarly, your FastAPI routes should not have any SQLAlchemy or Entity concerns. These two layers are integrated via Pydantic models.

  4. Everyone is expected to make at least one Pull Request. The Pull Request should have at least one additional improvement commit that improves based on a Code Reviewer’s suggestions. This PR should ultimatey be approved and merged into Stage ahead of the deadline. After final approval, it is your responsibility to merge the changes in and write the final commit message for your merge. When merging, you should “Squash and Merge” the commits into a single, well documented commit.

  5. Everyone is expected to serve as a Code Reviewer for at least one Pull Request. As a Code Reviewer, you are expected to find and request at least one improvement to the PR. Once the PR author addresses your concerns with improvement commit(s), you should review and approve the PR author to merge in the changes.

  6. Significant effort and care has gone into managing the project’s board with Cards/Issues. All members are actively participating in the GitHub project. Best practices are being followed, e.g. commit descriptions and change requests, per the assigned readings (Google Dev Engineering Practices).

Staging Server Environment (DevOps - At Least Two Members of Team Should Pair on This)

Choose one team member’s CloudApps namespace to setup as the staging environment where your team’s work will be hosted in the cloud. Other team members will be added to this namespace as collaborators so everyone has access.

In establishing the staging environment on our cloud infrastructure, we will work from the bottom up. We will start with the database server, then add secrets needed to build and run the application, then add the application, and finally add the route to expose the application to the internet.

Cleaning Up from EX03

  1. Login to OpenShift (remember: off-campus access requires connecting to the VPN)
  2. Open a Terminal on your Host Machine (not the VSCode Dev Env!)
    1. Login to the OpenShift Terminal on your Host Machine (not the VSCode DevEnv!)
    2. Navigate to your project’s directory in the terminal
    3. Login to OpenShift’s oc program you installed for EX03. You will need a new token from OpenShift: Click your name after logging in, select Copy Login Command, Display Token, copy the oc login ... line, paste into Terminal.
  3. Go ahead and delete your EX03 deployment now that everything is graded:
    • Run: oc delete all --selector app=comp423-ex03

Giving Team Members Access to Your Project

  1. Add team members to your workspace:
    • Navigate to the Administrator sidebar
    • Select: User Management > Role Bindings
    • For each team member, with their onyen:
      1. Create binding
      2. Name: admin-ONYEN (replace ONYEN with teammate’s ONYEN)
      3. Namespace: Your ONYEN
      4. Role name: admin
      5. Subject:
        • User
        • Name: your teammate’s ONYEN

Creating the Database Server

  1. Add a PostgreSQL database to your project:
    1. Developer View Sidebar
    2. Add (from the Sidebar)
    3. Database
    4. PostgreSQL Provided by Red Hat
    5. Instantiate Template
    6. Change the following settings:
      1. Database Service Name: db
      2. PostgreSQL Database Name: csxl
      3. Version of PostgreSQL Image: latest
    7. Create

Once the database is created, you can go to the Secrets page and view the generated credentials for the database under db (the name you gave it). If you select “Reveal Values” you can see the name, username, and password for the database. These secrets will be used as environment variables in your application in the next step.

Creating Secrets for your Application

Let’s create a secret for your application to use. This will be used to store the database credentials, and will ultimately be mounted as environment variables in your application.

Make note of the username and password for the database, from above. Additionally, you will need to generate a random string for the JWT_SECRET environment variable. This will be used to sign the JWT tokens that your application will use to authenticate users. You can generate a random string using the following command in your Dev Container:

openssl rand -hex 32

From your host machine’s terminal, run the following command to create the secret:

oc create secret generic final-project-environment \
    --from-literal=POSTGRES_HOST=db \
    --from-literal=POSTGRES_PORT=5432 \
    --from-literal=POSTGRES_DATABASE=csxl \
    --from-literal=POSTGRES_USER=<from-secret-above> \
    --from-literal=POSTGRES_PASSWORD=<from-secret-above> \
    --from-literal=JWT_SECRET=<generate-random-string>

From the OpenShift web console, you can verify that the secret was created by navigating to the Secrets page and selecting the final-project-environment secret.

Repository Deploy Key (Secret)

As with in EX03, you will need to add a deploy key to your repository so that OpenShift can pull your code from GitHub. This is the same process as EX03, but you will need to add the deploy key to your final project repository’s settings.

In your Dev Container’s terminal, run the following command to generate a new deploy key:

ssh-keygen -t ed25519 -C "Deploy Key for Final Project" -f ./deploy_key

Do NOT set a passphrase. When prompted for a passphrase, just press enter.

This will generate two files: deploy_key and deploy_key.pub. The public key (deploy_key.pub) will need to be added to your repository’s settings, and the private key (deploy_key) will need to be added to OpenShift as a secret.

Add the public key to your repository’s settings:

  1. Navigate to your repository’s settings
  2. Select Deploy Keys
  3. Add Deploy Key
  4. Title: CloudApps Deploy Key
  5. Key: Copy the contents of deploy_key.pub into the key field
  6. Check the box to allow write access
  7. Click Add Key

Create the secret in OpenShift:

  1. Back in your host machine’s terminal, not the Dev Container’s terminal, navigate to your project directory and run the following command to create the secret in OpenShift:
oc create secret generic final-project-deploy-key \
    --from-file=ssh-privatekey=./deploy_key \
    --type=kubernetes.io/ssh-auth

To verify the secret was correctly created, run the following command:

oc get secret final-project-deploy-key

Finally, you need to link the secret to the “builder” process of OpenShift. This will allow OpenShift to use the secret when it pulls your code from GitHub and builds your project.

oc secrets link builder final-project-deploy-key

This command will succeed silently.

Create the OpenShift Application

Back in your host’s terminal, since it has the oc command installed, run the following commands to create the application in OpenShift:

oc new-app python:3.11~git@github.com:comp423-23s/<your-final_repo_name>.git#stage \
  --source-secret=final-project-deploy-key \
  --name=final-project \
  --strategy=docker \
  --env=MODE=development \
  --env=HOST=team-k9-comp423-23s.apps.cloudapps.unc.edu

Notice the #stage at the end of the repository URL. This is the branch name that OpenShift will pull from. When setting up the final project, you created a branch named stage and established it as the primary branch for your repository. This notion of a staging branch is a common practice in DevOps, and is a good way to keep your production code separate (live at csxl.unc.edu) from your development code (which you are establishing right now).

While the project is building, add secrets to the environment variables of the deployment and verify their existence with list:

oc set env deployment/final-project --from=secret/final-project-environment
oc set env deployment/final-project --list

Exposing the Application

Once your application builds, it will be running on a pod that is not exposed to the internet. To establish a public route, first we need to expose it as a service, run the following command:

oc expose deployment final-project \
  --port=80 \
  --target-port=8080

Next, we can create a route to the service with a specifically chosen hostname. Please replace k9 with your team’s zone (lowercase) and table number:

oc create route edge \
  --service=final-project \
  --hostname=team-k9-comp423-23s.apps.cloudapps.unc.edu

You can now visit the hostname for your team and access it in the browser. If you see a message from OpenShift that says “Application is not available”, it means that the application is still building. Once your build completes, you should see the application running, but there is still one more important step: resetting the database.

Resetting the Database

The database that you created in the previous step is empty. You will need to reset the database to the state that it was in when you submitted your final project. To do this, you will need to run the reset_db.py script that is included in your final project repository.

This script needs to be run from within your pod, so in this section you will learn how to connect to your pod and run commands from within it.

First, you will need to find the name of your pod. Run the following command to get a list of pods running in your project:

oc get pods --selector deployment=final-project

You should see a single pod with a name like final-project-648fdff8d5-rr4fs. The letters and numbers at the end of the name are a unique identifier for the pod. This identifier changes every time a new build of your pod is deployed, environment variables change, the pod gets restarted, and in other instances. Copy the name of your running pod and run the following command to connect to it:

oc rsh final-project-YOUR-POD-IDENTIFIER

The rsh stands for “remote shell”. You are now connected to your pod running in the cloud via a secure shell (ssh)! The commands you run are not running on your host machine, but on the CloudApps infrastructure. If you ls you will see you are in your project’s built directory. Not everything is there, importantly not the frontend because it was compiled into the static directory as part of the build process.

To confirm you are logged into your pod, you can assure yourself with the following command:

hostname

You can now run the reset_database script to reset the database. Run the following command to do so:

python -m backend.script.reset_database

You should see the SQLAlchemy log messages creating tables, inserting dev data, etc. Your staging database is now reset!

Important: As you deploy new versions, add new entities, add new dev data, etc., this process of resetting the database in staging is one you and your team members will both need to be comfortable doing and remember to do.

Setting up Push-to-Deploy Webhooks

GitHub repositories can be configured with webhooks, which are URLs that get called when events occur in order to notify another service of the event. In our case, we want to set up a webhook so that when we push to the stage branch, OpenShift’s build configuration for our project will receive a webhook notification and kick off a new build.

To find the URL for the web hook, open up your project in OpenShift and navigate to the Admin sidebar, followed by Builds > BuildConfigs. Select final-project and look for the webhooks section at the bottom. Click the Copy URL with Secrets button for the GitHub webhook. This copies the URL to your clipboard.

Next, open your project’s settings in GitHub and navigate to Webhooks. Click Add Webhook and paste the URL into the Payload URL field. Be sure to set the content type to application/json and leave the secret field empty. Click Add Webhook.

From the “Webhooks” page you’re brought back to, click your webhook. Then go to the Recent Deliveries tab. You should see a successful delivery. Congratulations, your project is now set up to automatically build and deploy every time your team merges PRs into the stage branch! As a reminder, if your data entities change, you will need to reset the database in staging after the build and deploy completes.

Contributor(s): Kris Jordan