Table of Contents¶
Overview¶
Why?¶
The goal of the ScoringEngine is to keep track of service up time in a blue teams/red team competition.
How does it work?¶
The general idea of the ScoringEngine is broken up into 3 separate processes, Engine, Worker, and Web.
Engine¶
The engine is responsible for tasking Checks that are used to verify network services each round, and determining/saving their results to the database. This process runs for the entire competition, and will sleep for a certain amount of time before starting on to the next round.
Worker¶
The worker connects to Redis and waits for Checks to get tasked in order to run them against . Once it receives a Check, it executes the command and sends the output back to the Engine.
Web¶
The web application provides a graphical view of the Competition. This includes things like a bar graph of all team’s scores as well as a table of the current round’s results. This can also be used to configure the properties of each service per team.
External Resources¶
We currently use MySQL as the database, and Redis as the data store for tasks while they are getting scheduled.
Putting it all together¶
- The Engine starts
- The first Round starts
- The Engine tasks Checks out to the Workers
- The Workers execute the Checks and return the output to the Engine
- The Engine waits for all Checks to finish
- The Engine determines the results of each Check, and saves the results to the DB
- The Engine ends the Round
- The Engine sleeps for some time
- The second Round starts
- …
Installation¶
Docker¶
Note
It takes a minute or 2 for all of the containers to start up and get going!
TestBed Environment¶
make rebuild-testbed-new
This command will build, stop any pre-existing scoring engine containers, and start a new environment. As part of the environment, multiple containers will be used as part of the testbed environment.
Environment Variables¶
We use certain environment variables to control the functionality of certain docker containers.
| SCORINGENGINE_OVERWRITE_DB: | |
|---|---|
| If set to true, the database will be deleted and then recreated during startup. | |
| SCORINGENGINE_EXAMPLE: | |
| If set to true, the database is populated with sample db, and the engine and worker containers will be paused. This is useful for doing development on the web app. | |
You can set each environment variable before each command executed, for example:
SCORINGENGINE_EXAMPLE=true make rebuild-new
Production Environment¶
Modify the bin/competition.yaml file to configure the engine according to your competition environment. Then, run the following make command to build, and run the scoring engine.
Warning
This will delete the previous database, exclude the ‘new’ part from the command to not rebuild the db.
make rebuild-new
Then, to ‘pause’ the scoring engine (Ex: At the end of the day):
docker-compose -f docker-compose.yml stop engine
To ‘unpause’ the engine:
docker-compose -f docker-compose.yml start engine
Manual¶
Base Setup¶
Note
Currently, the only OS we have documentation on is Ubuntu 16.04.
Install dependencies via apt-get¶
apt-get update
apt-get install -y python3.5 wget git python3.5-dev build-essential libmysqlclient-dev
Create engine user¶
useradd -m engine
Download and Install pip¶
wget -O /root/get-pip.py https://bootstrap.pypa.io/get-pip.py
python3.5 /root/get-pip.py
rm /root/get-pip.py
Setup virtualenvironment¶
pip install virtualenv
su engine
cd ~/
mkdir /home/engine/scoring_engine
virtualenv -p /usr/bin/python3.5 /home/engine/scoring_engine/env
Setup src directory¶
git clone https://github.com/scoringengine/scoringengine /home/engine/scoring_engine/src
Install scoring_engine src python dependencies¶
source /home/engine/scoring_engine/env/bin/activate
pip install -e /home/engine/scoring_engine/src/
Copy/Modify configuration¶
cp /home/engine/scoring_engine/src/engine.conf.inc /home/engine/scoring_engine/src/engine.conf
vi /home/engine/scoring_engine/src/engine.conf
Create log file locations (run as root)¶
mkdir /var/log/scoring_engine
chown -R syslog:adm /var/log/scoring_engine
Copy rsyslog configuration¶
cp /home/engine/scoring_engine/src/configs/rsyslog.conf /etc/rsyslog.d/10-scoring_engine.conf
Restart rsyslog¶
systemctl restart rsyslog
Web¶
Install MySQL Server¶
apt-get install -y mysql-server
sed -i -e 's/127.0.0.1/0.0.0.0/g' /etc/mysql/mysql.conf.d/mysqld.cnf
systemctl restart mysql
Setup MySQL¶
mysql -u root -p<insert password set during installation>
CREATE DATABASE scoring_engine;
CREATE USER 'engineuser'@'%' IDENTIFIED BY 'enginepass';
GRANT ALL on scoring_engine.* to 'engineuser'@'%' IDENTIFIED by 'enginepass';
Install Nginx¶
apt-get install -y nginx
Setup SSL in Nginx¶
mkdir /etc/nginx/ssl
cd /etc/nginx/ssl
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt
Copy nginx config¶
cp /home/engine/scoring_engine/src/configs/nginx.conf /etc/nginx/sites-available/scoring_engine.conf
ln -s /etc/nginx/sites-available/scoring_engine.conf /etc/nginx/sites-enabled/
rm /etc/nginx/sites-enabled/default
systemctl restart nginx
Setup web service¶
cp /home/engine/scoring_engine/src/configs/web.service /etc/systemd/system/scoring_engine-web.service
Modify configuration¶
vi /home/engine/scoring_engine/src/engine.conf
Install uwsgi¶
pip install uwsgi
Start web¶
systemctl enable scoring_engine-web
systemctl start scoring_engine-web
Monitoring¶
journalctl -f _SYSTEMD_UNIT=scoring_engine-web.service
tail -f /var/log/scoring_engine/web.log
tail -f /var/log/scoring_engine/web-nginx.access.log
tail -f /var/log/scoring_engine/web-nginx.error.log
Engine¶
Install Redis¶
apt-get install -y redis-server
Setup Redis to listen on external interface¶
sed -i -e 's/bind 127.0.0.1/bind 0.0.0.0/g' /etc/redis/redis.conf
systemctl restart redis
Setup Engine service (run as root)¶
cp /home/engine/scoring_engine/src/configs/engine.service /etc/systemd/system/scoring_engine-engine.service
Modify configuration¶
su engine
vi /home/engine/scoring_engine/src/engine.conf
Setup scoring engine teams and services¶
su engine
vi /home/engine/scoring_engine/src/bin/competition.yaml
source /home/engine/scoring_engine/env/bin/activate
/home/engine/scoring_engine/src/bin/setup
Start engine service (must run as root)¶
systemctl start scoring_engine-engine
Monitor engine¶
journalctl -f _SYSTEMD_UNIT=scoring_engine-engine.service
tail -f /var/log/scoring_engine/engine.log
Worker¶
Modify hostname¶
hostname <INSERT CUSTOM HOSTNAME HERE>
Setup worker service (run as root)¶
cp /home/engine/scoring_engine/src/configs/worker.service /etc/systemd/system/scoring_engine-worker.service
Modify configuration¶
Change REDIS host/port/password fields to main engine host::
vi /home/engine/scoring_engine/src/engine.conf
Modify worker to customize number of processes. Append ‘–concurrency <num of processes>’ to the celery command line. If not specified, it defaults to # of CPUs.
vi /home/engine/scoring_engine/src/bin/worker
Start worker service (must run as root)¶
systemctl enable scoring_engine-worker
systemctl start scoring_engine-worker
Monitor worker¶
journalctl -f _SYSTEMD_UNIT=scoring_engine-worker.service
tail -f /var/log/scoring_engine/worker.log
Install dependencies for DNS check¶
apt-get install -y dnsutils
Install dependencies for HTTP/HTTPS check¶
apt-get install -y curl
Install dependencies for most of the checks¶
apt-get install -y medusa
Install dependencies for SSH check¶
pip install paramiko
Install dependencies for LDAP check¶
apt-get install -y ldap-utils
Install dependencies for Postgresql check¶
apt-get install -y postgresql-client
Install dependencies for Elasticsearch check¶
pip install requests
Install dependencies for SMB check¶
pip install pysmb
Install dependencies for RDP check¶
apt-get install -y freerdp-x11
Install dependencies for MSSQL check¶
apt-get install -y apt-transport-https
curl -s https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl -s https://packages.microsoft.com/config/ubuntu/16.04/prod.list | tee /etc/apt/sources.list.d/msprod.list
apt-get update
ACCEPT_EULA=Y apt-get install -y locales mssql-tools unixodbc-dev
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
Install dependencies for SMTP/SMTPS check¶
cp /home/engine/scoring_engine/src/scoring_engine/checks/bin/smtp_check /usr/bin/smtp_check
cp /home/engine/scoring_engine/src/scoring_engine/checks/bin/smtps_check /usr/bin/smtps_check
chmod a+x /usr/bin/smtp_check
chmod a+x /usr/bin/smtps_check
Configuration¶
Location to config file¶
Docker¶
Note
This file needs to be edited before running the make commands.
<path to source root>/docker/engine.conf.inc
Manual¶
Note
Need to restart each scoring engine service once the config is modified.
/home/engine/scoring_engine/src/engine.conf
Configuration Keys¶
Note
Each of these config keys can be expressed via environment variables (and take precendence over the values defined in the file). IE: To define round_time_sleep, I’d set SCORINGENGINE_ROUND_TIME_SLEEP=3.
| Key Name | Description |
|---|---|
| checks_location | Local path to directory of checks |
| round_time_sleep | Amount of time (seconds) the engine sleeps between rounds |
| worker_refresh_time | Amount of time (seconds) the engine will sleep for in-between polls of worker status |
| worker_num_concurrent_tasks | The number of concurrent tasks the worker will run. Set to -1 to default to number of processors. |
| worker_queue | The queue name for a worker to pull tasks from. This can be used to control which workers get which service checks. Default is ‘main’ |
| timezone | Local timezone of the competition |
| debug | Determines wether or not the engine should be run in debug mode (useful for development) |
| db_uri | Database connection URI |
| cache_type | The type of storage for the cache. Set to null to disable caching |
| redis_host | The hostname/ip of the redis server |
| redis_port | The port of the redis server |
| redis_password | The password used to connect to redis (if no password, leave empty) |
Implemented Checks¶
DNS¶
Queries a DNS server for a specific record
Custom Properties:
| qtype | type of record (A, AAAA, CNAME, etc) |
| domain | domain/host to query for |
Elasticsearch¶
Uses python requests to insert message and then query for same message
Custom Properties:
| index | index to use to insert the message |
| doc_type | type of the document |
FTP¶
Uses python ftplib to login to an FTP server, upload a file, login again to FTP and download file
Uses Accounts
Custom Properties:
| remotefilepath | absolute path of file on remote server to upload/download |
| filecontents | contents of the file that we upload/download |
HTTP(S)¶
Sends a GET request to an HTTP(S) server
Custom Properties:
| useragent | specific useragent to use in the request |
| vhost | vhost used in the request |
| uri | uri of the request |
IMAP(S)¶
Uses medusa to login to an imap server
Uses Accounts
Custom Properties:
| domain | domain of the username |
LDAP¶
Uses ldapsearch to login to ldap server. Once authenticated, it performs a lookup of all users in the same domain
Uses Accounts
Custom Properties:
| domain | domain of the username |
| base_dn | base dn value of the domain (Ex: dc=example,dc=com) |
MSSQL¶
Logs into a MSSQL server, uses a database, and executes a specific SQL command
Uses Accounts
Custom Properties:
| database | database to use before running command |
| command | SQL command that will execute |
MySQL¶
Logs into a MySQL server, uses a database, and executes a specific SQL command
Uses Accounts
Custom Properties:
| database | database to use before running command |
| command | SQL command that will execute |
POP3(S)¶
Uses medusa to login to an pop3 server
Uses Accounts
Custom Properties:
| domain | domain of the username |
PostgreSQL¶
Logs into a postgresql server, selects a database, and executes a SQL command
Uses Accounts
Custom Properties:
| database | database to use before running command |
| command | SQL command that will execute |
SMB¶
Logs into a system using SMB with an account/password, and hashes the contents of a specific file on a specific share
Uses Accounts
Custom Properties:
| share | name of the share to connect to |
| file | local path of the file to access |
| hash | SHA256 hash of the contents of the file |
SMTP(S)¶
Logs into an SMTP server and sends an email
Uses Accounts
Custom Properties:
| touser | address that the email will be sent to |
| subject | subject of the email |
| body | body of the email |
SSH¶
Logs into a system using SSH with an account/password, and executes command(s)
Note
Each command will be executed independently of each other in a separate ssh connection.
Uses Accounts
Custom Properties:
| commands | ‘;’ delimited list of commands to run (Ex: id;ps) |
VNC¶
Connects and if specified, will login to a VNC server
Uses Accounts (optional)
Custom Properties: none
Development¶
Note
Currently we support 2 ways of working on the Scoring Engine. You can either use the existing Docker environment, or you can run each service locally using python 3. If you choose to do your development locally, we recommend using virtual environments.
Initial Setup¶
These steps are for if you want to do your development locally and run each service locally as well.
Create Config File¶
cp engine.conf.inc engine.conf
sed -i '' 's/debug = False/debug = True/g' engine.conf
Hint
If debug is set to True, the web ui will automatically reload on changes to local file modifications, which can help speed up development.
Install Required Dependencies¶
pip install -e .
Populate Sample DB¶
python bin/setup --example --overwrite-db
Run Services¶
Web UI¶
python bin/web
Then, access localhost:5000
| Username | Password |
|---|---|
| whiteteamuser | testpass |
| redteamuser | testpass |
| team1user1 | testpass |
| team2user1 | testpass |
| team2user2 | testpass |
Note
The engine and worker do NOT need to be running in order to run the web UI.
Engine¶
Both the engine and worker services require a redis server to be running. Redis can be easily setup by using the existing docker environment.
python bin/engine
Worker¶
python bin/worker
Run Tests¶
We use the pytest testing framework.
Note
The tests use a separate db (sqlite in memory), so don’t worry about corrupting a production db when running the tests.
First, we need to install the dependencies required for testing.
pip install -r tests/requirements.txt
Next, we run our tests
pytest tests
Hint
Instead of specifying the tests directory, you can specify specific file(s) to run: pytest tests/scoring_engine/test_config.py