The new Smartform updater
A cross-platform updater for all Smartform data products
To coincide with the development of the new beta database products, we are introducing a new cross-platform updater application that can be used to update all smartform products in a single database.
The application is currently available for:
- Windows (32-bit and 64-bit) tested on Windows 10
- Linux (amd64 and i386), tested on recent Ubuntu
- macOS, tested on macOS 10.15 Catalina but should work appropriately from at least High Sierra onwards
- Raspberry Pi (a new beta! Tested on a Pi 4 using Raspberry Pi OS and Ubuntu)
To get started, follow the above links for installation instructions.
Please note that the updater binaries are currently unsigned; you can read more about this.
N.B. the Go (golang) source code for the updater is not yet generally available but we may be able to make it available for inspection on request.
How the updater works
The updater is a small cross-platform binary. The basic process of using it is the same on all platforms, and in regular use its operation is effectively the same as the Linux bash script or Windows batch file currently used.
First run configuration
When the utility is run for the first time, it will ask for your Betwise ID and download credentials.
Once these are entered, the utility checks that the account they refer to exists and is a current subscriber.
If so it then asks for the connection details of the database to update, and once those credentials are entered and verified, it can immediately offer fetch data for any products installed in the database. You can choose the feed to fetch interactively or close the window (or press Control-C to do so).
Some notes on database configurations
- If you are using the updater to connect to a remote database server, the database updater will prefer TLS/SSL-secured connections where possible. Your remote server must support TLS 1.2; earlier versions are insecure and are unsupported by the updater. MySQL 8.x fully supports TLS 1.2, as do versions of MySQL 5.7 above 5.7.35.
- The utility can interactively configure and interactively update a single database, and for practicality reasons we recommend you do use a single database. If you do need to maintain and update multiple databases, the utility can be manually configured for this. Please get in touch if you do have this requirement.
Fetching feeds
This configuration only needs to be done once; the utility saves its configuration for subsequent reuse.
After it is installed and configured, the operation is essentially the same as the current scripts.
Fetching individual feeds
You can fetch an individual feed by naming it on the command line, if necessary specifying the full path to the smartform-updater binary instead of the relevant fetch-updates script, so on Linux or Mac you might use:
smartform-updater historic
or on Windows:
smartform-updater.exe historic
Fetching all feeds at once
While it remains most sensible to fetch the data feeds individually when they become available on any given day, the updater can also update all configured feeds at once:
smartform-updater all-feeds
We ask that you do this infrequently; no more regularly than every three hours.
Scheduling tasks
This utility can be run from your crontab or Windows task scheduler in the same way as the batch files currently run.
On Windows we also have an installer that can configure your scheduled tasks for you; more on this below.
Automatic detection of new data products
The updater can automatically identify when a data product has been added to the database.
As long as you install all your products in the same database, the updater will offer to update any products it detects.
So the process of arranging updates for an additional data product is as simple as downloading and installing the initial data set.
There is no further need to reconfigure the updater or write new scripts.
An installer for Windows
We have built a standard installer application for Windows (tested on Windows 10).
This updater will:
- install the correct (32-bit or 64-bit) binary for your system
- run it for the first time to help you configure
- help you set up recommended scheduled tasks.
Note that if you have previously installed scheduled tasks to run the fetch_updates.bat tool, you should remove those yourself manually.
A note on the existing updater scripts
The scripts you are already using will continue to work for the forseeable future. We will no longer be providing new Linux bash or Windows batch scripts for additional database products, but if you are unable to run the new updater or would prefer not to, we should be able to support you in the creation of your own updater scripts.