Docs in markdown - Joplin (bonus points: to static site)
why #
I need to maintain some documentation about home/personal projects, while sharing just the home docs with my house mate
Joplin ticks most of what I want:
- [x] a cross platform tool where I can organize my notes
- [ ] the notes need to be saved in markdown format for future-proofing
- [x] sane method of syncing (read: webdav)
- [x] encryption
- [x] solid tool not under heavy development (I want to adopt it once and keep it for some time)
Bonus points:
- [x] the notes need to be easily exportable
- [x] the export should be in a format that allows to share them in a static html site
and the winner is.. #
Joplin seems to cover most of what I want, the notes are not in .md format but I can export the with an automation.
What I don't like:
the noted are in markdown format inside the app but everything is saved on a sqlite database.
There's an app for every major platform, and also a cli for headless systems or unattended operations.
runner ups #
Logseq seemed to be the what I wanted but wanted to sync in ways I would not want to support (file sync on every device),
since I also want to support iOS devices a webdav sync is the way to go.
Joplin #
Installation #
Apps #
very simple:
install the app and configure it to sync via webdav
CLI #
install node and the install Joplin:
npm install -g joplin
run joplin
your config files are in ~/.config/joplin
source:
WebDav Sync #
Apps #
set it up in the apps
CLI #
enter joplin and use the interactive console via ":",
then run these commands
:config sync.target 6
:config sync.6.path https://example.com/something/webdav/Joplin
:config sync.6.username YOUR_USERNAME
:config sync.6.password YOUR_PASSWORD
then run the sync command:
:sync
E2EE #
(end 2 end encryption)
Apps #
on the first client setup the E2EE and resync,
on the !first clients then sync the items, you will be asked for the master pasword you set up on the first client.
CLI #
run
:e2ee decrypt
you will be asked the master password
afeter every sync you'll need to decrypt your notes:
joplin sync
joplin e2ee decrypt
otherwise when you export your notes you'll receive this error:
This item is currently encrypted: note "test01" (aaa....fff) and was not exported. You may wait for it to be decrypted and try again.
other useful resources #
plugins: https://github.com/joplin/plugins/blob/master/README.md#plugins
Joplin CLI #
useful commands: #
list all the notebooks: #
joplin ls -l /
aaaaa 5 14/01/2024 16:10 Welcome!
bbbbb 14/01/2024 21:48 Home
headless sync: #
joplin sync
Synchronisation target: (6)
Starting synchronisation...
Fetched items: 1/1.
Downloading resources...
Fetched items: 1/1. Completed: 14/01/2024 21:31 (3s)
Joplin to Mkdocs #
I want to be able to export the notebook "Home" and transform it to a static html site,
since we are exporting markdown files we can leverage the power and simplicity of Mkdocs to achieve what we want.
Mkdocs #
install mkdocs and create a new project:
pip install mkdocs
mkdocs new mkdocs-prj-name
a new folder called "mkdocs-prj-name" will be created, with this structure:
mkdocs-prj-name
|-- docs
| `-- index.md
`-- mkdocs.yml
then we can place a directory structure with md files in the folder "docs" and everything will be built
to serve the mkdocs site while working on it or for debugging purposes we can execute this in the project folder:
mkdocs serve -t readthedocs
our site will be accessible locally on http://127.0.0.1:8000/
if you just want to build the site use:
mkdocs build -t readthedocs
your built files will be in the /site folder in the mkdocs workdir:
.
|-- docs
| `-- index.md
|-- mkdocs.yml
`-- site
|-- 404.html
|-- css
| |-- fonts
| | |-- Roboto-Slab-Bold.woff
| | |-- Roboto-Slab-Bold.woff2
| | |-- Roboto-Slab-Regular.woff
| | |-- Roboto-Slab-Regular.woff2
| | |-- fontawesome-webfont.eot
| | |-- fontawesome-webfont.svg
| | |-- fontawesome-webfont.ttf
| | |-- fontawesome-webfont.woff
| | |-- fontawesome-webfont.woff2
| | |-- lato-bold-italic.woff
| | |-- lato-bold-italic.woff2
| | |-- lato-bold.woff
| | |-- lato-bold.woff2
| | |-- lato-normal-italic.woff
| | |-- lato-normal-italic.woff2
| | |-- lato-normal.woff
| | `-- lato-normal.woff2
| |-- theme.css
| `-- theme_extra.css
|-- img
| `-- favicon.ico
|-- index.html
|-- js
| |-- html5shiv.min.js
| |-- jquery-3.6.0.min.js
| |-- theme.js
| `-- theme_extra.js
|-- search
| |-- lunr.js
| |-- main.js
| |-- search_index.json
| `-- worker.js
|-- search.html
|-- sitemap.xml
`-- sitemap.xml.gz
source: https://www.mkdocs.org/getting-started/
themes #
I've found I like the "redthedocs" theme more than the "mkdocs" one
this is mostly due to the fact that we have a list of the Notebooks and Notes on the left instead of in chaotic and nested menus in the top bar
markupsafe errors #
While building, if you receive an odd error about markupsafe like:
from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.9/dist-packages/markupsafe/__init__.py)
downgrade markupsafe to 2.0.1:
pip install markupsafe==2.0.1 --force
Joplin CLI operations #
to export one notebook we'll need to sync, decrypt and the export:
joplin sync
joplin e2ee decrypt
joplin export --notebook "Home" --format md_frontmatter ~/mkdocs-prj-name/docs/Joplin/
keeping it together #
I chose to export Joplin's files in a specific folder inside mkdocs's "docs" folder,
due to the fact that if I execute multiple exports I'll have duplicate files, so I want to clean said folder before exporting.
after the export the mkdocs folder structure will look like this:
mkdocs-prj-name
|-- docs
| |-- Joplin
| | |-- Home
| | | |-- Electricty
| | | | `-- wiring.md
| | | `-- Water
| | | `-- water_meter.md
| `-- index.md
`-- mkdocs.yml
NB: the folder structure will be kept, in the mkdocs site menus you'll find the structure:
|-- Joplin
| |-- Home
| | |-- Electricty
| | | -- wiring | |-- Water
| | `-- water_meter
the script #
#!/usr/bin/env bash
ssh_key="/home/user/.ssh/id_rsa"
folder_base="/data/joplin-to-mkdocs/mkdocs_workdir"
joplin_dest="/docs/Joplin"
rsync_source="/site/"
rsync_user="rsync_user"
rsync_host="192.168.0.10"
rsync_dest="/var/www/html/your-static-site-vhost/"
logfile="/data/joplin-to-mkdocs/autoupdate_script.log"
#wake up ssh:
eval `ssh-agent`
ssh-add $ssh_key
rm -rf $folder_base$joplin_dest
joplin sync
joplin e2ee decrypt
joplin export --format md_frontmatter $folder_base$joplin_dest
# to export just one notebook use: --notebook "NotebookName"
#this substitutes one newline with to newlines in every .md file, it's used for better mkdocs compatibility
find $folder_base$joplin_dest -name "*.md" -exec gawk -i inplace 'BEGIN{RS="\n" ; ORS="\n\n";}; { print }' {} \;
cd $folder_base
mkdocs build -t readthedocs
rsync -avhz --progress $folder_base$rsync_source -e "ssh -i $ssh_key" $rsync_user@$rsync_host:$rsync_dest
rm -rf $folder_base$joplin_dest
echo "- - -" | tee -a $logfile
eval "$(ssh-agent -k)"
scheduling #
I do it via cron:
*/5 * * * * user bash /data/joplin-to-mkdocs/autoupdate_script.sh > /dev/null
workflow #
issues #
formatting #
what I see correctly in Joplin is displayed differently on mkdocs built site, for example the spacings between lines, take this example:
banana
mango
in Joplin a single new line is enough to have the text on different lines:
banana
mango
whilst in mkdocs will be displayed spaced on the same line:
banana mango
this issue should have been patched with the find and aws line in the script,
it seems to persist in lists where the indentation is not respected.
for what I've seen the rest of the formatting is ok