I always wanted to learn how to pass a value to xargs somewhere in the middle of the command. The -I option can do it where the {} is just a token replacer. For example, here is a way to search for an IP prefix in a bunch of text files:

echo 192.168.1. | xargs -I{} grep {} *.txt

Here is how to pass -l to the middle of an ls {} -F command:

$ echo "-l" | xargs -I{} ls {} -F
total 0
drwxr-xr-x@ 7 emartin  staff  224 Nov  6 22:58 Folder1/
drwxr-xr-x@ 7 emartin  staff  224 Nov  6 23:58 Folder2/

I am really going to find this handy to do things like doing a find and then copying the item to a base folder.

You can actually use almost any token for the -I option.

(HT: Stack Exchange)

Earlier this week, I created a new Blogger tool to support this new Pelican-based blog. It is a simple Python3 tool that accepts a variable number of arguments and then creates a Pelican friendly Markdown file. It will then launch that new file in the Markdown editor of your choice. In my case it launches Visual Studio Code. Here is an example:

$ blogger.py New Python Blogger Tool
Creating: 2021-04-17-new-python-blogger-tool.md

And since I hate typing repetitive words, in my .zshrc I have the following line to simplify launching the blog editor to simply b:

alias b=blogger.py

Sample call to the tool:

$ b New Python Blogger Tool
Creating: 2021-04-17-new-python-blogger-tool.md

The code is not too advanced so I will just put it below. The script requires:

• Python 3.6+ for f-strings
• pyyaml from pip

You will probably notice a custom util and path module. Those are just simple helper modules. The main functions used.

• path.canonize() - Converts paths like ~/folder/../folder2 into /Users/emartin/folder2. See os.path.normpath() and os.path.expanduser() in the built-in os module.
• util.log() - My console logger with time stamps. Easily replaceable with a print() statement.

I would post the link to my Github repo, but it is a private and it would take me a while to extract secrets. Maybe one day soon. Here is the code:

#!/usr/bin/env python3
import argparse
import os

import path
import util
import run

# pip install pyyaml
import yaml

# Backup configuration needs to live in the same location as script
__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))
config_yaml = "blogger_config.yaml"
config_yaml_path = os.path.join(__location__, config_yaml)

def blogger(config, title):
  source_folder = path.canonize(config['source_folder'])

  # Validate source folder exists
  if not os.path.exists(source_folder):
    util.log(f'ERROR: Source folder does not exist: {source_folder}')
    return -1

  content_folder =  os.path.join(source_folder, 'content')

  # Validate content folder exists
  if not os.path.exists(content_folder):
    util.log(f'ERROR: Content folder does not exist: {content_folder}')
    return -1

  # Generate the file name from the date and passed in title
  today = util.todaysDate()
  year = '{:%Y}'.format(today)
  month = '{:%m}'.format(today)
  day = '{:%d}'.format(today)
  name = title.lower().replace(" ","-")
  filename = f'{year}-{month}-{day}-{name}.md'

  new_blog_file = os.path.join(content_folder, filename)

  # Only create the file if the file doesn't exist otherwise just open it
  if not os.path.exists(new_blog_file):
    # Get the metadata defaults from the config file
    category = config['category']
    tag = config['tag']
    author = config['author']

    # Write metadata to top of the file in yaml format
    print(f'Creating: {filename}')
    with open(new_blog_file, 'a') as f:
      f.write(f'Title: {title}\n')
      f.write(f'Date: {today}\n')
      f.write(f'Category: {category}\n')
      f.write(f'Tags: {tag}\n')
      f.write(f'Authors: {author}\n')

  # Launch Markdown Tool
  markdown_tool = path.canonize(config['markdown_tool'])
  run.launch_app(markdown_tool, new_blog_file)

# Main function
if __name__ == '__main__':
  # Parse arguments
  parser = argparse.ArgumentParser(description='Script to automate blogging.')
  parser.add_argument('title', help=f'The title of the blog.', nargs='*', default='')
  args = parser.parse_args()

  # Dump Help if no parameters were passed
  if (len(args.title) == 0):
    parser.print_help()
    exit(0)

  # Load configuration file
  config = {}
  with open(config_yaml_path) as file:
   config = yaml.full_load(file)

  # Run Blogger
  title = " ".join(args.title)
  return_code = blogger(config, title)
  exit(return_code)

And the blogger_config.yaml looks like this:

source_folder: ~/Source/blog.emrt.in
category: "Misc|Media|Technology"
tag: "Blog"
author: Erik Martin
markdown_tool: /Applications/Visual Studio Code.app/Contents/Resources/app/bin/code

Descriptions of the configuration:

• source_folder - Where the Pelican blog source is located. Files are created in the content folder.
• category - category of the post (blog entry metadata)
• tag - comma separated tags of the post (blog entry metadata)
• author - default author of the post (blog entry metadata)
• markdown_tool - The path to the markdown tool that launches the created .md file

A number of years ago I started The Expanse series of books. I read Leviathan Wakes the year it was published on a recommedation from the virtual book club The series is space opera sci-fi that I felt started out with a bang, but lost me around book 3 (Abaddon's Gate). Looking back I was probably simply overloaded with genre fiction and I never re-engaged, until this year.

Now that there are signs that we are slowly getting out of the pandemic, I was looking for a book series that was a bit more noir-ish, so I restarted the series. I have now passed where I stopped. I am about 10% into Cibola Burn and I can honestly say that it was a mistake to not continue with the series. The nuanced characterizations, the good people put in bad situations (Holden), there is no need to tell everyones back story in book 1. If I would have a complaint about anything is that the authors, James S.A. Corey, pull in new major characters every book and you have to learn the series dynamics from new. That may have been why I backed away from the series.

The cool bit is now there is a 5 season TV series on Amazon Prime to watch to complement the books. I find myself getting ahead in the book series and then playing catchup with the TV series. As usual, some of the differences are a bit jarring or disappointing, but I guess I am aged enough to not let that turn me off the TV series completely. The logistics of having a cast be busy throughout a season necessitates some fluff or some consolidation of storylines. I do like how the TV series introduced Avasarala early in season 1. It also helps that Shohreh Aghdashloo is a fantastic actress with such screen presence. The addition of the Drummer character was probably the most jarring of the changes for me because I wasn't sure at first who she was supposed to be. At first I thought she was Sam, but then Bull in season 3. Oh well, the character actor, Cara Gee is so wonderful that I learned to sit back and enjoy the ride with her storyline.

I am also listening to the series on audio book. The voice actor is really good at capturing the Belter language. When I read the words in the ebook and then listen to the same dialog in the audio book, it is not how I would have pictured it being said.

Today I finished off the first draft of my new blogger tool that I wrote in python to simplify the workflow of creating new Markdown files for the blog. Features:

• Loads a yaml configuration file
• Parses the command-line for the new blog title
• Creates the new blog file in the Pelican content folder
• Add the Pelican metadata at the top of the Markdown file with the Title, Date, Category, and Author.
• Launches the configured Markdown tool. In this case Visual Studio Code.

It took me in total about 20 minutes to build and test it because it is very similar to my customized Journal scripts. I want to make some changes and add some polish to make it easy to handle titles. I'll do another post soon with more details.

Update: Done. Added polish.

After 7 long years of using Octopress on this blog, today I moved it to Pelican.

I love static site generators. No real hacking threat like Wordpress. No need to have a database. All you need is a little markdown, a little metadata, and an engine to generate the site.

I like Pelican because it is based on Python instead of Ruby. Also, I could never really get used to Jekyll. The dependency handling was just way too complicated for my needs.

The command line to publish a new blog post is right up my alley. Here is all you have to d to create a new post:

cd blog.emrt.in
# create/edit a blog post
vi content/2021-04-11.md
pelican content
make rsync_upload

In typical Python fashion, the code is really easy to decipher what is going on. I found a nice Octopress theme for Pelican that made it super simple to change my old blog to the new theme. The most difficult part of the whole process is I had to convert the Markdown from using a yaml based header to a simple text header.

I do need to create some quick and dirty Python scripts to help with creating the skeleton for a new blog post. I also need to get all of this moved to Github.

My cousin, Chris, and I were talking last night about how I need to ramp up on Git. I use Git for this blog and I know how to do basic stuff, but I also need to come up to speed on some of the technologies and methodologies surrounding the software. Chris suggested that I check out git-flow first outlined in this blog post:

http://jeffkreeftmeijer.com/2010/why-arent-you-using-git-flow/

The main idea to git-flow is that you have a main develop branch and master is only for production releases. Features go on a feature branch and are merged back to the develop branch. There is also a release branch where you can put bug fixes and a hotfix branch where a branch is created from master and then it is commited back to master on release and back to develop.

There are Git extensions you can install with Brew:

$ brew install git-flow

Or manually integrate from github:

https://github.com/nvie/gitflow

With the extension you can create an empty repository that follows the git-flow methodology:

$ git flow init

It will then ask to create the branch prefix for each of the Flow branches. Then you can continue to use the extensions to do things like start a feature branch:

$ git flow feature start hot-new-feature
Switched to a new branch 'feature/hot-new-feature'

And then finish it, merging it back to develop and deleting the feature branch

$ git flow feature finish hot-new-feature
Switched to branch 'develop'

To create a release branch that tags it with a version:

$ git flow relase start 1.0
Switched to a new branch 'release/0.1.0'

You can then manually merge to master, or you can finish the release:

$ git flow release finish 1.0
Switched to branch 'master'

And from the blog post, here is all of what happens:

Boom. git-flow pulls from origin, merges the release branch into master, tags the release and back-merges everything back into develop before removing the release branch.

Hotfixes have a similar syntax, the difference is they are based off of master not develop.

Considering how powerful Git is when it comes to branching and merging compared to Subversion, it seems like a great idea to have these transient branches.

Maven is a project manager, not a build manager like Ant, hence the pom.xml file (Project object model). It tries to use a certain format to where the files are on the system to make it simpler to include dependencies.

$ mvn archetype:generate

This generates a simple template. archetype is the plugin, generate is the goal.

$ mvn help:effective-pom

This generates the entire POM that includes the minimal pom.xml including any parent POMs, user settings, and active profiles.

Maven doesn’t know how to compile your code or make a JAR file, but the plugins it uses do. Basic maven is a basic shell that knows how to:

parse the command-line manage a classpath parse a POM file download Maven plugins Maven Lifecycle Plugin goals can be attached to lifecycle phases (e.g. install, test). Each phase may have 0 or more goals attached to it. Example phases:

• process-resources
• compile
• process-classes
• process-test-resources
• test-compile
• test
• prepare-package
• package

When mvn fires is launches all phases before it in the lifecycle.

You can also just specify the plugin goals, but that is a lot more tedious

$ mvn resources:resources compiler:compile resources:testResources surefire:test jar:jar install:install

4 (really 5) definitions create the coordinate system for a product:

1. groupId - reverse domain name
2. artifactId - unique identifier under the group representing a single project
3. packaging - package type (jar, war, ear)
4. version - a specific release of a project
5. classifier (rarely used)

Maven download artifacts and plugins from a remote repository to a your local machine and stores them in your local Maven repository (~/.m2/repository). install phase installs a .jar into the local repository. A repository contains the .jar and the .pom of what it depends on.

You can generate a site and documentation by doing the following:

$ mvn site

Generates javadoc, and other custom reports

This can tell you what all of the dependencies are:

$ mvn dependency:resolve

A prettier way to see the dependency tree

$ mvn dependency:tree

I got a lot of useful information for this post from Maven By Example

With a desire to keep some of my personal scripts neat and tidy, I added some of them to GitHub here:

https://github.com/ez1222/MacScripts

Last night I installed and have been using BitTorrent Sync, a syncing utility to keep files and folders up to date on all of your machines.

In the past I was using DropBox, but for a number of reasons I dropped them. One of the big ones was that I found myself not storing as many files in their service. Even though they claimed that files were encrypted on their server and could not be read, I never felt comfortable keeping my secure files there.

Last night I saw a tweet by Wil Wheaton. He was talking about how he had a seemless switch to BitTorrent Sync, a product I have heard of, but never researched. The interesting thing about BitTorrent Sync is that it doesn't use a central server to store your files. It uses a peer-to-peer mechanism to sync files from one computer to another. It does this using the BitTorrent protocol.

There is also a pretty handy way to setup an iOS device with the product. All you have to do is download the app from the App Store and then scan a QR code. This connects up your device. The mobile app also has a password you can enter to prevent others from getting your files.

Is BitTorrent Sync perfect? No. The iOS app does not automatically sync files. There isn't even a setting for that. You have to manually select a file and it will start a transfer from one of your other machines. Also, to get around Firewall and NAT issues, BitTorrent Sync makes use of relay servers to get your files from one machine behind a firewall to another machine. Luckily you can disable the usage of relay servers in the settings, but that means that if BitTorrent Sync can't access inside your firewall at home, you can't sync.

For my casual usage of syncing, the pros outweigh the cons. I have been very happy with it so far. I have been able to unify my old DropBox sync and manual sync folders. I also like the name of the default sync folder. It is ~/Sync on Mac. In my opinion that is nicer than the branded ~/DropBox folder.

I have been using Git a lot in the last few weeks, for setting up this blog and for some research at work. Here is a brain dump for mainly my own personal reference.

Initial Setup

One thing you’ll probably want to setup on first run, make sure you fill in your name and email address in the Git configuration so that commit notes are assigned to you. In UI clients, it will probably ask you the first time you launch or try to download a repository. At the Terminal, you will want to do this:

$ git config --global user.name "Erik Martin"
$ git config --global user.email emartin@myemailservice.com

Getting a repository

Downloading an existing repository:

$ git clone http://git.server.com/git/myproduct.git

It will create a local folder named myproduct and start copying the files. This will retrieve the master branch. The above URL will be configured as the origin.

If you want to grab a specific branch called feature_branch and place it in the local product_feature_branch folder, you do the following:

$ git clone http://git.server.com/git/myproduct.git -b feature_branch product_feature_branch

Committing and Pushing

After you clone the tree, you can do commits just like Subversion:

$ git commit -m “This is a commit message” file.cpp

Assuming you are still on the master branch, push back to the origin remote like this:

$ git push origin master

Or push all local branches back to the origin:

$ git push origin --all

Remotes

You can easily setup another “backup” remote to a folder on your same machine by doing something like this in the local repository folder:

$ git remote add backup /Users/emartin/Source/backup/myproduct.git

A push of the master branch to the backup remote would look like this:

$ git push backup master

You can list all of a repository’s remotes by going to a local repository folder and typing:

$ git remote -v
backup /Users/emartin/Source/git/backup/myproduct.git (fetch)
backup /Users/emartin/Source/git/backup/myproduct.git (push)
origin http://git.server.com/git/myproduct.git (fetch)
origin http://git.server.com/git/myproduct.git (push)

New Repository Setup

If you want to create a new repository from an existing set of files, in the top level folder do this:

$ git init
$ git add .
$ git commit -m “Initial checkin” .

Then on the server or a local remote, you setup a bare repository, say in a folder named myproduct.git

$ mkdir myproduct.git
$ cd myproduct.git
$ git init --bare

Server or local remotes should be bare or you will get a warning during your push. See more info below in the research section of what a bare repo is and why a push must be bare.

On your local machine, you setup the remote in the new repository top level folder:

$ git remote add origin http://git.server.com/git/myproduct.git

And then assuming the authentication is correctly setup, push:

$ git push origin master

Branching

If you want to create a new branch and set it to the current branch, you just do the following while in the local sandbox:

$ git branch new_feature_branch
$ git checkout new_feature_branch

Quick way to create and set the branch:

$ git checkout -b new_feature_branch

To switch back to the master branch:

$ git checkout master

Git Rebase

Sometimes it makes sense to take commits from a feature branch and 'rebase' them into another branch, like the main development branch. That makes the log look more linear when looking back in the history. Here is an example of a rebase:

http://git-scm.com/book/en/Git-Branching-Rebasing

Submodules

Submodules are analogous to Subversion externals, a way to "attach" external repositories to another repository. Submodules work differently and are not as easy to use as svn externals. More info below in the research section.

To add a new submodule to an existing git repository:

$ git submodule add http://git.server.com/git/third_pary_library.git third_pary_library

This creates a folder called third_party_library and updates a .gitmodules file. .gitmodules is version controlled in the parent repository.

After adding, you have to commit the submodule:

$ git commit -m “Committing the submodule third_party_library” .

This commit locks the submodule to that revision of third_party_library. So if someone clones your parent repository, they get the committed revision of the submodule.

If a repository has submodules, there are two ways to check out. The legacy way:

$ git clone http://git.server.com/git/myproduct.git
$ cd repository
$ git submodule init
$ git submodule update

And the easy way:

$ git clone --recursive http://git.server.com/git/myproduct.git