DEVELOPMENT NOTES

Azure DevOps

1. git repositories - SSH authentication
2. Service Principals
3. Creating and consuming Python packages

git repositories - SSH authentication

Azure DevOps provides git repositories, and to connect to these on macOS or Linux it is recommended to use SSH. The steps are as follows:

1. Create a public/private SSH key pair: ssh-keygen -f ~/.ssh/<newkeyname>; create a passphrase when prompted
2. Ensure that this key is added to your SSH Agent: ssh-add ~/.ssh/<newkeyname>; enter the passphrase when prompted
3. Copy the newly created public key to the Azure DevOps "SSH public keys"
4. Set the relevant git configuration settings at the local repository level: git config --add --local core.sshCommand 'ssh -i ~/.ssh/<newkeyname>'

Full instructions can be found at https://docs.microsoft.com/en-us/azure/devops/repos/git/use-ssh-keys-to-authenticate?view=azure-devops.

Service Principals

Sometimes when executing scripts remotely it is necessary to have access to certain Azure resources (e.g. access to KeyVault in order to get a connection string for a database to carry out a deploy as part of a release pipeline). When these automated scripts are not Azure resources themselves then they need be run under a "Service Principal" in order to be authenticated.

Create and use a Service Principal - Password-based authentication

In order to authenticate as a Service Principal, if attempting to authenticate in a dotnet executable using Azure.Identity.DefaultAzureCredential, then the shell executing the script / executable must have the following environment variables set, where the values are as per the newly created Service Principal:

• AZURE_CLIENT_ID
• AZURE_TENANT_ID
• AZURE_CLIENT_SECRET

Create and use a Service Principal - Certificate-based authentication

In order to authenticate as a Service Principal, if attempting to authenticate in a dotnet executable using Azure.Identity.DefaultAzureCredential, then the shell executing the script / executable must have the following environment variables set, where the values are as per the newly created Service Principal:

• AZURE_CLIENT_ID
• AZURE_TENANT_ID
• AZURE_CLIENT_CERTIFICATE_PATH

Creating and consuming Python packages

Useful resources:

• Get started with Python packages in Azure Artifacts
• Publish Python packages with Azure Pipelines

To build and publish a package:

1. Create a feed under Azure DevOps Artifacts, making a note of the assigned name
2. Ensure your Python project is in a fit state to be packaged :)
3. Create a YAML pipeline for the repository, and edit it to look (broadly) like the following:

variables:
  python.version: '3.10'
  major.version: '0'
  minor.version: '0'

name: $(major.version).$(minor.version).$(Rev:r)

trigger:
- master

pool:
  vmImage: ubuntu-latest

steps:
- task: UsePythonVersion@0
  inputs:
    versionSpec: '$(python.version)'
  displayName: 'Use Python $(python.version)'

- task: PythonScript@0
  inputs:
    scriptSource: 'inline'
    script: |
      with open("setup.py", "rt") as fin:
        data = fin.read()
        data = data.replace('{__BUILDNUMBER__}', '$(Build.BuildNumber)')
      with open("setup.py", "wt") as fin:
        fin.write(data)
  displayName: 'Update Version Number'

- script: |
    pip install build twine tox
  displayName: 'Install dependencies'

- script: tox
  displayName: 'Run tests'

- script: |
    python -m build
  displayName: 'Build distribution'

- task: TwineAuthenticate@1
  inputs:
    artifactFeed: '<PROJECT_NAME/FEED_NAME>'
  displayName: 'Twine Authenticate'

- script: |
    python -m twine upload -r <FEED_NAME> --config-file $(PYPIRC_PATH) dist/*.whl
  displayName: 'Upload package'

4. Note that the above assumes (a) that tests are configured to run using tox, and (b) a setup.py file exists, and the version property is set to '__BUILDNUMBER__' in order that the pipeline can replace this with a configured build number (else the publish action will fail if the version number is not updated with each build).

To consume a published package:

1. Create a Personal Access Token ("PAT") under the user's security settings within Azure DevOps
2. pip install the package and pass the PAT embedded into the repository URL through the --index-url or --extra-index-url arguments, where the URL takes the following form:
https://<feedname>:<pattoken>@pkgs.dev.azure.com/<org>/<project>/_packaging/<feed>/pypi/simple/

Databases

1. SQL Server

SQL Server

Users and Logins

The SQL Server access model uses the concepts of logins for authentication (at the server level) and users for authorization (at the database level).

To create a login: CREATE LOGIN <login> WITH PASSWORD = '<password>'[, CHECK_POLICY = ON|OFF];

To create a user and associate it to a login: USE <database>; CREATE USER <username> FROM LOGIN <login>;

To achieve anything with the newly created user permissions will need to be granted. A simple way to achieve this is to make use of SQL Server's in-built 'Fixed-database roles'. There are a number of roles; some of the most useful are db_ddladmin (allows user to run any DDL command), db_datawriter (allows user to add, delete, or change data in all user tables), and db_datareader (allows user to read all data from all user tables and views). The syntax to add a user to a role is as follows:

Note executing the above will require a user to themselves have permissions to execute the ALTER ROLE command.

.Net Development

1. Running services "in production" from Docker containers with TLS

Running services "in production" from Docker containers with TLS

Generate a self-signed certificate

To enable HTTPS use in local web app development use the dotnet dev-certs command.

To create a simple self-signed certificate simply run the following command: dotnet dev-certs https --trust. This will create a pfx file, typically within ~/.aspnet/dev-certs/https (on a Mac).

For greater control, and to allow other applications to be able to make use of the created certificate, create a self-signed certificate in a specified location and protected with a password with the following command: dotnet dev-certs https --trust --export-path <PATH> --password <PASSWORD>.

Pass self-signed certificate to Docker container

Assuming a Docker image encapsulating a .Net application, this can be run securely over TLS (only locally, as that is the only location that will trust the self-signed certificate), via a run command similar to the following:

docker run \
    --rm \
    -p <PUBLISHED PORT>:443 \
    -e ASPNETCORE_URLS="https://*"
    -e ASPNETCORE_Kestrel__Certificates__Default__Path=/https/<CERTIFICATE NAME>.pfx \
    -e ASPNETCORE_Kestrel__Certificates__Default__Password=<CERTIFICATE PASSWORD> \
    -v ~/.aspnet/dev-certs/https:/https/ \
    <IMAGE>:<TAG>

The above assumes that the self-signed certificate exists as a pfx file within ~/.aspnet/dev-certs/https.

The ASPNETCORE_URLS environment variable tells the application to listen to all IP addresses on the https protocol on the default port of 443.

The ASPNETCORE_Kestrel__Certificates__Default__Password environment variable is only required if the pfx certificate is password-protected.

OpenSSL and encryption

1. Hashing
2. Password-Based Key Derivation Functions
3. Symmetric Encryption
4. Asymmetric Encryption
5. Combining symmetric and asymmetric encryption
6. Digital Signatures

Hashing

Basic commands and documentation

Basic hashing commands are run using openssl dgst, and documentation can be viewed via man openssl-dgst.

A list of available hashing algorithms can be viewed via openssl dgst -list.

A sensible starting point is to use the sha256 hashing algorithm.

Hashing Files

To hash the file foo.txt run the command: openssl dgst -sha256 foo.txt

Password-Based Key Derivation Functions

Generating a key from a password using PBKDF2

Using password as a password, we can generate a key as follows:

openssl kdf \
    -keylen 32 \
    -kdfopt pass:password \
    -kdfopt salt:$(openssl rand 16) \
    PBKDF2

In the example above note the use of the salt parameter to ensure that the same password produces different keys when used multiple times.

Symmetric Encryption

Basic commands and documentation

Basic encryption commands are run using openssl enc, and documentation can be viewed via man openssl-enc.

A list of available encryption ciphers can be viewed via openssl enc -list.

For symmetric encryption a sensible starting point is to use the aes-256-cbc cipher.

Encrypting Files with self-generated Keys and Initialization Vectors

Fist generate a key and initialization vector via the openssl rand command as follows:

Next, for a given file (say foo.txt) encrypt as follows:

openssl enc \
    -aes-256-cbc \
    -base64 \
    -e \
    -in foo.txt \
    -K $K \
    -iv $IV \
    -out foo.txt.encrypted

Note: The -base64 flag simply base64 encodes the encrypted output for readability (but is not required).

To decrypt the encrypted file simply reverse the above command (by swapping the -in and -out parameters, and replacing the -e encrypt flag with a -d decrypt flag.

Encrypting Files with Keys and Initialization Vectors generated from user-supplied passwords

For a given file (say foo.txt) and password (say password) encrypt as follows:

openssl enc \
    -aes-256-cbc \
    -base64 \
    -e \
    -pbkdf2 \
    -pass pass:password \
    -salt \
    -in foo.txt \
    -out foo.txt.encrypted

Note: The -salt flag is "on" by default and so is technically not required but can be provided for visibility.

Decrypting works as follows:

openssl enc \
    -aes-256-cbc \
    -base64 \
    -d \
    -pbkdf2 \
    -pass pass:password \
    -in foo.txt.encrypted

Note: The password can be supplied in a number of different ways. As above, passing -pass pass:<PASSWORD> will pass a password directly from the command line. Similarly, passing -pass env:<ENV_VAR> will use the named environment variable. Finally, simply omit the -pass flag altogether to enter interactive mode whereby openssl will prompt the user for a password to encrypt/decrypt as required.

A minimal example to encrypt a file would look as follows (with the CLI prompting the user for a password):

Asymmetric Encryption

Basic commands and documentation

Documentation can be viewed via the following man pages:

• man openssl-genpkey (generate a private key)
• man openssl-pkey (public or private key processing command)
• man openssl-pkeyutl (public key algorithm command)

Generating and inspecting a keypair

A keypair derived using the RSA algorithm can be generated via the following command:

The above command generates a 2,048-bit private key by default. The number of bits can be specifed by appending -pkeyopt rsa_keygen_bits:4096 to the command (replacing 4,096 with the requisite number of bits).

For a keypair, say rsa_keypair.pem, we can view the structure via the following command:

If we wish to restrict the above to only the public component we can the following command:

Note: As always, simply append -out <FILENAME> to the above command to save the output to a specified file.

Encrypting and decrypting using a keypair

RSA is limited to encrypting data no longer than the key length during one operation. Therefore we typically apply RSA encryption to session keys. The following code shows the generation of an arbitrary session key, and how to encrypt it, assuming a public key saved at rsa_public_key.pem:

openssl rand -out session_key.bin 32
openssl pkeyutl \
  -encrypt \
  -in session_key.bin \
  -out session_key.bin.encrypted \
  -pubin \
  -inkey rsa_public_key.pem \
  -pkeyopt rsa_padding_mode:oaep

We can decrypt the above output assuming a private key saved at rsa_keypair.pem:

openssl pkeyutl \
  -decrypt \
  -in session_key.bin.encrypted \
  -inkey rsa_keypair.pem \
  -pkeyopt rsa_padding_mode:oaep

Combining symmetric and asymmetric encryption

Assume we wish to encrypt the contents of foo.txt.

1. Generate a symmetric encryption key: openssl rand -out symmetric_keyfile.key 128
2. Encrypt the file using the symmetric encryption key: openssl enc -aes-256-cbc -in foo.txt -out foo.txt.enc -pbkdf2 -pass file:symmetric_keyfile.key
3. Generate a keypair: openssl genpkey -algorithm RSA -out private_key.pem
4. Generate a public key: openssl pkey -pubout -in private_key.pem -out public_key.pem
5. Encrypt the symmetric encryption key: openssl pkeyutl -encrypt -in symmetric_keyfile.key -out symmetric_keyfile.key.enc -pubin -inkey public_key.pem -pkeyopt rsa_padding_mode:oaep

We can now distribute both the encrypted file foo.txt.enc and the encrypted key symmetric_keyfile.key.enc. Decryption requires knowledge of private_key.pem.

To decrypt the file:

1. Decrypt the symmetric encryption key: openssl pkeyutl -decrypt -in symmetric_keyfile.key.enc -out symmetric_keyfile.key -inkey private_key.pem -pkeyopt rsa_padding_mode:oaep
2. Decrypt the file: openssl enc -d -aes-256-cbc -in foo.txt.enc -out foo.txt -pbkdf2 -pass file:symmetric_keyfile.key

Digital Signatures

We can use Elliptic Curves to digitally sign a file. We generate a keypair, and use the private key to sign a file, and the corresponding public key to verify the signature.

View the available curves: openssl ecparam -list_curves

Generate a keypair: openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:secp521r1 -out ec_keypair.pem

Generate a public key: openssl pkey -in ec_keypair.pem -pubout -out ec_public_key.pem

Sign the contents of somefile.txt using the private key ec_keypair.pem:

openssl pkeyutl \
  -sign \
  -digest sha3-512 \
  -rawin \
  -in somefile.txt \
  -inkey ec_keypair.pem \
  -out somefile.txt.signature

Verify the contents of somefile.txt.signature using the public key ec_public_key.pem:

openssl pkeyutl \
  -verify \
  -digest sha3-512 \
  -rawin \
  -in somefile.txt \
  -pubin \
  -inkey ec_public_key.pem \
  -sigfile somefile.txt.signature