All posts by markcerv_admin

Python and Django and Pipenv and ASDF – Creating a new project in Oct 2023

Django and Python keep releasing new versions, so I wanted to take some notes as I start a new project in October of 2023.

Tools I’ll be using:

asdf – I use this to manage python versions (an easy way to have multiple version of python installed and be able to have different directories/projects use different versions)

pipenv – is “a Python virtualenv management tool that supports a multitude of systems and nicely bridges the gaps between pip, python (using system python, pyenv or asdf) and virtualenv”

python – your favorite program language

django – your favorite high level python web framework.

Let’s get started – python first

python --version
Python 3.9.14

While that’s a huge step ahead of Python2, for this new project I want the latest and most stable.

asdf latest python
3.12.0

I have no reason to have on the .0 version, but let’s see what asdf has for version 3.11

asdf latest python 3.11
3.11.6

OK – that sounds good. Let’s use that.

asdf update
asdf install python 3.11.6
asdf reshim python 3.11.6
asdf local python 3.11.6   # This 

Now that we have 3.11.6, let’s create the directory that we want to use for the project.

python --version  # let's see what the "global version is"
Python 3.9.14     # still set to the older version

mkdir my-new-project
cd my-new-project
asdf local python 3.11.6

python --version  # what about now?
Python 3.11.6     # OK, that's good enough for me

Time to get pipenv going

So, I had a bit of a hard time getting pipenv to work with the newest asdf version of python. Here’s what helped:

# make sure you are in the correct project directory
# and that the directory is empty

python -m pip install --upgrade pip
pip install pipenv

pipenv --python 3.11
Here's a screenshot for what happened.

I want to show you what happens if you have an existing Pipfile in the directory, and if that Pipfile contains a python [requires] section that looks like this:

...
[requires]
python_version = "3.9"
...
 pipenv --python 3.11
Creating a virtualenv for this project...
Pipfile: /mnt/c/home/dashrc/clients/brazen/events-mvp/Pipfile
Using /home/mark/.asdf/installs/python/3.11.6/bin/python3 (3.11.6) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.11.6.final.0-64 in 105ms
  creator CPython3Posix(dest=/home/mark/.local/share/virtualenvs/events-mvp-my-dZw-e, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/mark/.local/share/virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.2.2, wheel==0.41.2
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/mark/.local/share/virtualenvs/events-mvp-my-dZw-e
Warning: Your Pipfile requires python_version 3.9, but you are using 3.11.6 (/home/mark/.local/share/v/e/bin/python).
  $ pipenv --rm and rebuilding the virtual environment may resolve the issue.
  $ pipenv check will surely fail.

Now for some django

pipenv install "django>=4"
pipenv install "django>=4"
Installing django>=4...
Resolving django>=4...
Added django to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (97f99baf1b56a396a4dd34bb22964db55993d76c480132cfaabfef55aa8444c1)!
Installing dependencies from Pipfile.lock (8444c1)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

Once django is installed, you will need to fire it up:

pipenv shell
django-admin startproject config .

Note in the above I used the word “config” — I intentionally choose to do that so that all of my config files end up in that config directory. And actually I’ll do a little bit more than that by converting the settings.py file into a settings/ folder with:

mkdir config/settings
cp config/init.py config/settings
mv config/settings.py config/settings/base.py

# Create a new config file by running this cat command
# cut and paste from first "cut" to final "EOF" in a shell

cat <<EOF > config/settings/local.py
# Django settings
#

# These are the settings for a local environment
from .base import *
EOF

What about the database?

In my live environments I always use PostreSQL, so I mimic that behavior on my local machines by running on PostgreSQL locally on my Ubuntu server (via WSL2)

Don’t forget to add in the psycopg2 pip library

pipenv install psycopg2

After that, you’ll need to login as the postgres user on your machine, and create the database and assign a password.

sudo su - postgres

# do this inside of the postgres account

postgres_db_and_user=tvcodes
createuser $postgres_db_and_user
createdb -O $postgres_db_and_user  $postgres_db_and_user

psql
## these next commands are INSIDE the postgresql shell

alter user xxxxDBUSERNAMExxxx with encrypted password 'whatever_password_should_be';

(hint: https://bitwarden.com/password-generator/)

what does that look like?

psql
psql (14.9 (Ubuntu 14.9-0ubuntu0.22.04.1))
Type "help" for help.

postgres=# alter user my_awesome_project with encrypted password 'whatever_password_should_be';
ALTER ROLE
postgres=# quit
postgres@localhost:~$ 
logout

And what do the variables look like?

DATABASE_URL=postgres://my_awesome_project:whatever_password_should_be@localhost:5432/my_awesome_project

Let’s use local environment variables

We want to NOT put important variables directly in the code, so the modern day solution is to use environment variables. I’m influenced by: https://pypi.org/project/environs/

pipenv install environs[django]

Now create a .env file at the same level as your manage.py file that looks like this:

cat <<EOF > .env
#
DJANGO_SETTINGS_MODULE=config.settings.local
#
DEBUG=True
SECRET_KEY="not so secret on local"

DATABASE_URL=postgres://my_awesome_project:whatever_password_should_be@127.0.0.1:5432/flytest
EOF

Let’s login locally

Let’s create a user we can login with:

python manage.py migrate
python manage.py createsuperuser
python manage.py runserver 0.0.0.0:8000

NOTE: This might fail if you don’t have environment variables configured yet. If using VSCode,

Copy putty (ssh) sessions to new/other computer

I’m a huge fan of putty – available for download at: Download PuTTY: latest release (0.78) (greenend.org.uk)

Over time, I tend to accumulate a large number of saved “sessions” that I find it useful to either move to a new computer, or save/backup. He

regedit /ea c:\Users\mark\dropbox\_mark\puttycopy.reg HKEY_CURRENT_USER\Software\SimonTatham\PuTTY

Copied that puttycopy.reg to the new drive/computer.

To add the saved sessions, just double clicked on that .reg file on the new computer.

Issue w/ pipenv python versions after upgrading asdf’s python

I recently upgraded from python 3.9.12 to 3.9.14 using asdf, and it was pretty simple:

sudo apt-get install liblzma-dev
asdf install python 3.9.14
asdf global python 3.9.14

But when I tried to run my pipenv, it didn’t work as planned

pipenv shell
No preset version installed for command pipenv
Please install a version by running one of the following:

asdf install python 3.9.14

or add one of the following versions in your config file at /home/mark/.tool-versions
python 3.9.12

After a lot of failed attemps to use asdf reshim and asdf update, I finally remembered the magic steps to use:

python -m pip install --upgrade pip
pip install pipenv

asdf update
asdf reshim python 3.9.14   # or whatever newest version is

pipenv install
pipenv shell
pipenv update   # To actually install what was in Pipfile

The last trick that I needed to do was make sure that my .env gets read so that any Python environment variables I set are correctly read in.

vi ~/.local/share/virtualenvs/hostname-RANDOM/bin/activate

And add these 2 lines starting at line 4 of the activate script:

# Use this sequence to load up the environnment variables from your .env file in a Bash console:
#
set -a; source /home/mark/home/aei/aeicloud.22/.env; set +a

Ubuntu 22 LTS on WSL / asdf / python / django

I’ll be using asdf to help control what version of Python I’m running.

I like pipenv to help manage my Python virtual environments.

I like to use PostgreSQL with Django, so lets’ first make sure that it’s installed:

sudo apt install postgresql 
sudo apt install libpq-dev

# make sure it's running
# You should see it listening to lots of ports
ss -nlt

asdf

git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.10.0

# modify .bashrc files, exit terminal and restart terminal
sudo apt-get update
sudo apt-get upgrade

# install packages necessary to build stuff
sudo apt install wget build-essential libreadline-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev libffi-dev zlib1g-dev liblzma-dev

asdf plugin-add python

asdf install python 3.9.12
asdf global python 3.9.12

If the correct version of Python isn’t detected, you might need to reshim.

python -m pip install --upgrade pip
pip install pipenv

asdf update
asdf reshim python 3.9.12

pipenv install

[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
You can use $ pipenv install –skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Hint: try $ pipenv lock –pre if it is a pre-release dependency.
ERROR: metadata generation failed

Turned out the above error, was because I didn’t have postgres installed on the new box (and my django uses postgres)

 Error: pg_config executable not found.

I got much futher along

# how to know things are almost all the way there
./manage.py shell
python manage.py dbshell

# If you didn't setup your postgres, you'll get an error like:
# python manage.py dbshell
# psql: error: connection to server at "127.0.0.1", port 5432 failed: Connection refused
#         Is the server running on that host and accepting TCP/IP connections?

You’ll need to start the server:

sudo service postgresql start

Then try

python manage.py dbshell
psql: error: connection to server at "127.0.0.1", port 5432 failed: FATAL:  password authentication failed for user "abcuser"
connection to server at "127.0.0.1", port 5432 failed: FATAL:  password authentication failed for user "abcuser"

The user and database don’t appear to exist — so let’s create them:

sudo su - postgres
# skipping the lines as you login as user postgres


psql
# now you are in the postgres shell

CREATE DATABASE abcdb;
CREATE USER abcuser WITH PASSWORD 'abc123';

\q     # that exists the postgres shell

exit   # that logs out the postgres user

You should now be able to init the DB

python manage.py migrate
python manage.py createsuperuser

# then run any bootstrapping that you need to
# python manage.py loaddata  xyz.json
# needed for pango and weasyprint (creating PDFs)
sudo apt-get install libpangocairo-1.0-0

Now, to get django working in VSCode

  1. install the Python extension for Visual Studio Code
  2. Press F1, then choose Python: Select Interpreter and make sure you are pointing to your correct venv
  3. Press the Run and Debug Icon

Resetting the mysql password

Start server

$ sudo service mysql start

Go to sock folder

$ cd /var/run

Back up the sock

$ sudo cp -rp ./mysqld ./mysqld.bak

Stop server

$ sudo service mysql stop

Restore the sock

$ sudo mv ./mysqld.bak ./mysqld

Start mysqld_safe

$ sudo mysqld_safe --skip-grant-tables --skip-networking &

Init mysql shell

mysql -u root

Change password

FLUSH PRIVILEGES;

ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'root';

Stop and stop mysql

sudo service mysql stop
sudo service mysql start

This is based on the combination of a few suggestions from https://stackoverflow.com/questions/41984956/cant-reset-root-password-with-skip-grant-tables-on-ubuntu-16

WSL boot shell script

I’ve created this script to help me get what I need running on my WSL Ubuntu VMs

I’m calling it ~/.mrc_boot_script and I also have it saved it to: https://gist.github.com/markcerv/7ba608b69bf6edf57db456187ad8a4ff

#If we made it in here, then that's a good thing

read -r -t 15 -p "Run the only on boot commands? [y/N] " response
response=${response,,}    # tolower

if [[ "$response" =~ ^(yes|y)$ ]]
then
    echo "All of these commands need sudo, so be prepared to enter in a password"
    sleep 2

    #Need to do this to get screens running cleanly
    echo "Screen cleanup"
    sudo /etc/init.d/screen-cleanup start

    #Let's also make sure postgres is running
    echo "Fire up postgresql"
    sudo service postgresql start

    #Let's also make sure mysql is running
    echo "Fire up mysql"
    sudo service mysql start

    #Let's also make sure ssh is running
    echo "Fire up ssh"
    sudo service ssh --full-restart

    #Let's also make sure redis is running (for celery)
    echo "Fire up redis-server"
    sudo service redis-server start
else
    echo "Doing nothing"
fi

Issues running npm & WSL2 (bad interpreter)

I started a new project and wanted to use node/npm inside of WSL2. However, when I tried to run npm inside of either Ubuntu 18 or 20, I got this error message:

mark@LAPTOP:~$ npm
-bash: /mnt/c/Program Files/nodejs/npm: /bin/sh^M: bad interpreter: No such file or directory

After a whole lot of searching, I came across this web page, https://stackoverflow.com/questions/63716587/in-wsl2-ubuntu-20-04-for-windows-10-nodejs-is-installed-but-npm-is-not-working which suggested updating the ~/.bashrc file to look like:

# strip out problematic Windows %PATH%
PATH=$(echo "$PATH" | sed -e 's/:\/mnt.*//g')

I added that to the VERY end of the file, and rebooted.

The next time I tried to run npm, I got this successful message instead:

mark@LAPTOP:~$ npm

Usage: npm <command>

where <command> is one of:
    access, adduser, audit, bin, bugs, c, cache, ci, cit,
    clean-install, clean-install-test, completion, config,
    create, ddp, dedupe, deprecate, dist-tag, docs, doctor,
    edit, explore, fund, get, help, help-search, hook, i, init,
    install, install-ci-test, install-test, it, link, list, ln,
    login, logout, ls, org, outdated, owner, pack, ping, prefix,
    profile, prune, publish, rb, rebuild, repo, restart, root,
    run, run-script, s, se, search, set, shrinkwrap, star,
    stars, start, stop, t, team, test, token, tst, un,
    uninstall, unpublish, unstar, up, update, v, version, view,
    whoami

npm <command> -h  quick help on <command>
npm -l            display full usage info
npm help <term>   search for help on <term>
npm help npm      involved overview

Specify configs in the ini-formatted file:
    /home/mark/.npmrc
or on the command line via: npm <command> --key value
Config info can be viewed via: npm help config

npm@6.14.4 /usr/share/npm

Update ubuntu system clock (ntpdate)

Did you notice that the system clock on your Ubuntu server is wrong? On a production server this might not happen, but on a staging server (or on a WSL instance) the date/time can dift. Here’s how to fix it:

First, ask your server what time it is (so you can have a baseline)

$ date
Thu Mar  4 12:30:02 PST 2021  

If that seems wrong, you’ll want to run ntpdate…but first make sure it’s installed:

$ sudo apt install ntpdate

Next, run the command to update the time

$ sudo ntpdate time.nist.gov

# 5 Mar 10:58:20 ntpdate[11619]: step time server 132.163.97.3 offset 80783.696700 sec

Wow, I was off by over 80,000 seconds, which is 1,346 minutes or 22.4 hours!

Now that it’s been fixed, let’s check the date/time again:

$ date
Fri Mar  5 10:58:26 PST 2021

Perfect!

Quick mysql commands

# configure mysql 1st time
mysql -u root --skip-password
mysql>  ALTER USER 'root'@'localhost' IDENTIFIED BY 'xxx';
# create databases
mysqladmin -u root create wp_somename;

# now run mysql as root
mysql -u root mysql -p


CREATE USER 'example_user'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password';

CREATE USER 'example_user'@'localhost' IDENTIFIED BY 'password';  /* in case above doesn't work */



GRANT ALL ON example_database.* TO 'example_user'@'localhost';

flush privileges;

exit;

WSL2 – Adding distro not in store

What if you wanted to add an older linux distro to your WSL2 bullpen? Visiting https://docs.microsoft.com/en-us/windows/wsl/install-manual#installing-your-distro will show you a large number of options where you can download the appropriate .appx file.

This will cause the <distro>.appx packages to download to a folder of your choosing. Follow the installation instructions to install your downloaded distro(s).

If you’re using Windows 10 you can install your distro with PowerShell. Simply navigate to folder containing the distro downloaded from above, and in that directory run the following command where app_name is the name of your distro .appx file.PowershellCopy

Add-AppxPackage .\app_name.appx

If that doesn’t work as expected, you could try any of these links that will get the distro from the Microsoft Store

he following links will open the Microsoft store page for each distribution: