Migrating Google Groups Archives Between Accounts

Recently I had to move a lot of data between an old GSuite Business account and a new GSuite Business account. Google support for such a migration is… well… can stand to be improved. The main pain points are email(1), Drive(2), Team Drives(3), Calendar(4), and for everything else – there is no migration. Google+ posts, password storage, Sites, Forms have to be recreated manually, and my pet peeve: Google Groups Archives.

If you’re not familiar with this awesome GSuite feature, which is based on the Google Groups usenet-like service, Google Groups for Business is a mailing list manager that in addition to distributing emails to recipients, also stores each email in an accessible archive – so new users can have access to old communications (this is great for accounting and support) and on top of that you can get forum like features with posting replies to topics and such. Unfortunately Google Groups has no export feature and because the archive is all about past communications, you can’t actually reproduce that data manually when you move to a new GSuite account.

(more…)


  1. The built-in data migration tool in the admin console – which is the only data migration tool available – only moves emails, and is not 100% reliable with that, and doesn’t move rules or other settings []
  2. There are external tools available, I’m using Multcloud, but sharing is a problem – the best you can get is to get a copy of each shared file and Google Docs without any sharing information attached, so that breaks the sharing. Other tools may convert all your Google Docs to Microsoft formats []
  3. Which surprisingly works very well – you just share the team drive to a user on the new domain and they can move all the files to a new team drive they create on the new account. Sharing information is lost and you have to reshare, but documents retain comments by the old users and there is no duplications []
  4. You can manually export all calendars to ical format and then manually import them one by one. Also not 100% []

Script Day: “secure” password generated one liner

Ever needed to create a “secure” password to register to a web site(1) and you couldn’t be bothered to invent a secure password? Just paste this command line to your terminal:

ruby -e 'puts [*"a".."z",*"A".."Z",*"0".."9",
  "!@#$%^&*()_+[]/-=.,".split("")
  ].shuffle[0..(ARGV.shift.to_i)].join' 16

The last argument is the number of characters to put into the password.


  1. that probably annoyingly require “at least 1 upper case letter, 1 lower case letter, 1 number and 1 special character” []

Script Day: AWS CLI with multiple accounts with ease

Maybe you are a consultant and juggle multiple clients with Amazon Web Services deployments, maybe you just have accounts for all the start-ups you ever worked for, or maybe you just like to use 17 different AWS accounts for the free-tier usage, but eventually your ~/.aws/credentials file looks like an MS-Windows INI file.

At this point, running the AWS CLI is kind of annoying – you need to remember the correct --profile flag to set for each scenario, and bash will not complete these for you…

Bash aliases to the rescue!

(more…)

Script day: persistent memoize in bash

One type of task that I often find myself implementing as a bash script, is to periodically generate some data and display or operate on it – maybe through a cron job, watch or simply a loop. Sometimes part of the process is an expensive computation (could be network based, IO intensive or simply subject to throttling by another entity). The way to deal with issues like that in modern programming languages is a caching technique known as “memoization” (based on the word “memorandum”) in the results of an expensive call is retained in memory after the first time, and returned for future calls instead of running the expensive calculation. We also need to clear the cache every once in a while, but that’s another issue.

So, how to implement in bash?

(more…)

Script Day: Cloud-init for MS-Windows, The Poor Man’s Version

Cloud-init is a Linux technology that allows easy setup and automation of virtual machines. The concept is very simple – the VM infrastructure provides some way of setting some custom data for each virtual machine (many providers call this “user data”), and when the operating system starts the cloud-init service reads that configuration, loads a bunch of modules to handle various parts and let them configure the system. As a user it is very convenient – you write a setup scenario using the variety of tools offered by cloud-init, you can store the scenario in a source control to allow to develop the scenario further, then just launch a bunch of machines with the specified scenario and watch them configure themselves.

The situation is much worse on the MS-Windows side of the fence: want to have an MS-Windows server configured and ready to go? Start a virtual machine, connect to is using RDP and Next, Next, Finish until your fingers are sore. Need to deploy a new version? either retrofit an existing image (again, manually) and risk deployment side effects, or do the whole process again from scratch.

Here’s a script to try to help a bit with the problem – at least on Amazon Web Services: a poor man’s cloud-init-like for MS-Windows server automation.

(more…)

Fix another ‘curl|sh’ bogus installation – Heroku

The Heroku toolbelt (which I don’t remember if its mentioned in the “curlpipesh” tumbler mentioned by Amir in his response to my “Fix RVM” post), is a CLI to manage applications on the Heroku PaaS platform. As is common (and horrible) in this day and age they also offer a ‘curl|sh’ type install on their home page.

While the Debian/Ubuntu specific installer is not entirely horrible – it basically adds the Heroku Toolbelt debian repository to the APT source list, updates the package list and installs it, the “standalone” version is as horrible as it can get: download an unsigned binary from the internet, get root permissions and then do something.

For users of Fedora and other distributions, or just Ubuntu users who don’t like installing external repositories on their system, here is a simpler method to get the Heroku Toolbelt running on your system without root permissions and downloading scripts off the internet:

(more…)

Fix RVM “run script from the internet to install”

On Wednesday I complained about the latest UN*X fad of installing software by running scripts from the internet, without any regard to how your operating system handles software installation.

Docker, that I complained about last time, at least has a script that takes into account the local software management solution (uses apt for Ubuntu, yum for Fedora, etc), but RVM – the Ruby Version Manager which is a popular tool among rubyists everywhere, just downloads a bunch of executable stuff (granted, most of it are scripts, but the difference is lost on most people) into arbitrary location on your file system. At least it doesn’t install system software, oh wait – it does.

While I can’t help with RVM’s desire to install system level software (that it actually needs because one of the things you want RVM to do for you is to compile ruby versions from source), I can try to help you figure out how to install RVM where you want it and use it how you want it.

(more…)

Docker and the horrible “one line installation” fad

One of the weird things that sane (or some would say “old skool”) system administrators complain about lately is that with the rising popularity of UN*X systems (mostly Mac OS X and Linux) in the world, and in particular in the software development world, people using UN*X system want less and less to understand how to manage their systems and the culmination is the

to install this complicated system level software, just copy and paste simple wget command to your terminal

with Docker being the most horrible example of that behavior. No sane person (who understand UN*X) will ever think that installing Docker by feeding the content of a URL to bash is a good idea, but for some reason this is the documented and recommended way by the Docker people. Other examples are abound, but lets concentrate on fixing the Docker scenario.

(more…)

Script Day: Upload Files to Amazon S3 Using Bash

Here is a very simple Bash script that uploads a file to Amazon’s S3. I’ve looked for a simple explanation on how to do that without perl scripts or C# code, and could find none. So after a bit of experimentation and some reverse engineering, here’s the simple sample code:

(more…)

Script Day: SSH to a host behind a NAT

I use SSH daily to work with different remote services, and its always a very straight-forward process… unless the remote server you want to work with is on LAN somewhere behind NAT(1). When you need to access such an internal server, the only option is to SSH into the firewall(2), and then SSH again to your server of choice.

But there’s a better way, and you don’t even have to fiddle with the firewall server!

(this is not actually a script, though minimal text editing is required)

The solution is actually quite simple: set up an alias in your .ssh/config file that you can use to call the remote server when you are outside the LAN (if you are inside the LAN its better to access it directly), and for that alias we will set up a ProxyCommand that will tell SSH to first access the firewall server and open a tunnel to the target LAN server.

It looks like this:

Host remote-alias
ProxyCommand ssh firewall-user@firewally-server nc lan-server 22

This set up works best if your access to the firewall-user account is without password or passphrase (using an SSH private key that is either without passphrase or already loaded in the agent), then the login is as streamlines as a direct access – but the worst is that you’d need to type in two passwords.

Enhanced by Zemanta

  1. router that does Network Address Translation so the servers address is not accessible from outside the LAN []
  2. or some other server that has legs both inside and outside the LAN – I’m using a DNATed server, what most off-the-shelf routers incorrectly call “DMZ“ []

Spam prevention powered by Akismet