CLI

Here are a bunch of one-off CLI snippets that I use frequently.

Copying

Fast copying lots of data between servers on same lan

You can use scp, sftp, or rsync to copy files pretty easily, but there is quite a bit of overhead. If you are in a situation where you have to copy lots and lots of data between servers and security isn’t a big concern, try the following method.

First, on your destination machine, open up a port to accept the bytes.

socat tcp4-listen:<DEST PORT> stdout | tar xvpf -

Second, on your source machine, create an archive of the files you want to transfer and then shoot them over.

tar cvf - ./<directory to archive> | socat stdin tcp4:<DEST IP>:<DEST PORT>

Fast copying lots of files between servers on same lan

Maybe instead of large number of bytes, you want to transfer lots of small files between servers. Give this a try.

First, on your source machine, open up a port.

tar -cf - -C /home/nali/sf-back . | pv | nc -l <SRC PORT>

Next, on your destination machine:

nc <SRC IP> <SRC PORT> | pv | tar -xf - -C

pv is used to show progress.

Mirror or Download Website

wget --recursive --page-requisites --adjust-extension \
     --span-hosts --convert-links --restrict-file-names=windows \
     -t 1 -T 5 -c -D <URL> --no-parent <URL>

-t: number of times to try
-T: number of seconds to wait per try

VPN

Check VPN IP when using with Gluetun

docker exec gluetun sh -c "wget http://ipecho.net/plain -O - -q ; echo"

That’s assuming your container is named gluetun.

Pruning

Some commands tested on Ubuntu.

Find large files

sudo find . -xdev -type f -size +100M

Backblaze

Determine bucket size

b2 get-bucket --show-size <BUCKET NAME> \
| jq -r '
    def h: [while(length>0; .[:-3]) | .[-3:]] | reverse | join(",");
    .totalSize | tostring | h'

Ubuntu

Update packages and upgrade to next point release

Install the deborphan package first which removes unused packages and then run the following.

apt -y update \
&& apt -y upgrade \
&& apt -y dist-upgrade \
&& apt -y autoremove \
&& apt -y autoclean \
&& apt -y clean \
&& apt -y remove $(deborphan)

Oracle (OCI)

Open a port in Ubuntu

sudo apt-get install firewalld
sudo firewall-cmd --permanent --zone=public --add-port=<PORT>/udp
sudo firewall-cmd --reload
service start firewalld

Remember to allow ingress for the port in the network settings.

Creating Encryption Key

First create the vault. Then when creating the key, select Import External Key, copy the Public Wrapping key, and save it in a file called wrappingKey.pub.

ECDSA keys aren’t allowed, so you probably can’t use your existing ssh keys. If so, you’ll have to create new keys:

openssl rand 32 > my_aes_key.bin

Then wrap this key with the Oracle wrapping key:

openssl pkeyutl -encrypt -in my_aes_key.bin -pubin -inkey wrappingKey.pub -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -out my_wrapped_key.bin

Upload the resulting my_wrapped_key.bin.

Policies

After creating a vault and wrapped key, the policy should look like:

Allow service blockstorage, compute to use keys in tenancy where target.key.id = 'ocid1.key.oc1.iad....'

Regular Expressions

Convert ISO 8601 to MM/DD/YYYY

For a test run:

sed -i bak  -E 's/([0-9]{4})-([0-9]{2})-([0-9]{2})T[0-9]{2}:[0-9]{2}:[0-9]{2}Z/\2\/\3\/\1/p' *

• -i bak: replace in-place and backup originals with .bak extensions.
  
• -E: extended regex

To actually run the command, replace /p with /g.

Cloudflare

Purge Website Cache (all pages)

You need an API token that has permission to purge the cache for the specific website. API tokens can be found in Cloudflare’s My Profile dropdown.

You could use the global API key, but it’s better to create one specific for this type of usage.

When creating a custom token, choose the following:

Permissions | Zone | Cache Purge | Purge<br/>
Zone Resources | Include | Specific Zone | <DOMAIN>

Use the API token below.

curl --request POST "https://api.cloudflare.com/client/v4/zones/<ZONE ID>/purge_cache" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <API TOKEN>" \
--data '{"purge_everything":true}'

Deploy Astro

Assuming wrangler is already installed:

npm run astro build && npx wrangler pages deploy

Expanding Shortened URLs

urlex.org is a great service for expand URLs. The only problem is that it’s a bit tedious. So I wrote a small Python program to submit a shortened URL to urlex.org and return the expanded URL. You can get it here.

Mac

Changing DNS

List the network hardware installed:

sudo networksetup -listallnetworkservices

See DNS settings by device:

Replace <device> with one of the items listed above.

networksetup -getdnsservers <device>

Update DNS settings:

sudo networksetup -setdnsservers <device> 1.1.1.1 8.8.8.8

Then verify with -getdnsservers command.

GPG

GPG Agent Forwarding with Yubikey

This is not the most straightforward process and it should really be in a script more generalized.

Run gpgconf --list-dir agent-extra-socket on the local machine.

Run gpgconf --list-dir agent-socket on the server.

For the host entry in ~/.ssh/config, add RemoteForward <server-agent-socket> <local-extra-agent-socket>.

Get the long id of your GPG key: gpg --list-secret-keys --keyid-format=long. It’s the part after rsa4096/ or ed25519/.

Login into the server.

Run gpg --keyserver keys.openpgp.org --recv-keys <YOUR_GPG_KEY_LONG_FORMAT>.

Run gpgconf --kill gpg-agent

Add StreamLocalBindUnlink yes to /etc/ssh/sshd_config on the server.

If the path from gpgconf --list-dir agent-socket exists on the server already, delete it.

Exit out of the server and log back in.

Now try encrypting and decrypting a file to test.

Chezmoi

List tracked files not updated in Chezmoi

chezmoi status

UFW

Reset iptables

Sometimes iptables and ufw seem to go nuts. Reset them:

udo apt update && \
sudo apt install -y ufw && \
sudo iptables -F && \
sudo iptables -X && \
sudo iptables -t nat -F && \
sudo iptables -t nat -X && \
sudo iptables -t mangle -F && \
sudo iptables -t mangle -X && \
sudo ufw --force reset && \
sudo ufw default deny incoming && \
sudo ufw default allow outgoing && \
sudo ufw allow 22/tcp && \
sudo ufw allow 80/tcp && \
sudo ufw allow 443/tcp && \
sudo ufw --force enable && \
sudo ufw status verbose

Apple

Checking temperature

sudo powermetrics -n 1 --samplers smc | egrep -i "FAN|CPU die|GPU die"

Also check out chezmoi git commands

More

gh/mitchweaver/bin - POSIX-only useful scripts

CLI application alternatives

curl → wcurl - saner defaults for common uses cases. And probably has the shortest man page ever.

Processes Accessing External Drive

lsof +D /Volumes/MyExternalDrive