Family Tree investigations #2: FamilySearch: Family Tree and associated data stores

Please see this PDF: Family Tree investigations #2 FamilySearch Family Tree and associated data stores.pdf

Family Tree investigations #3: How I use the FamilySearch APIs

Please see this PDF: Family Tree investigations #3 How I use the FamilySearch APIs.pdf

Family Tree investigations #1: Discovery

Please see this PDF: Family Tree investigations #1 Discovery.pdf

Retirement projects update

General tidying up:

1. Using Google Calendar as my day to day diary (plus Google Contacts as my main address book, and Google mail for more and more purposes).

• This is finally working adequately.
• Calendar integration with Outlook was hard.
• I ended up having to use a free Outlook plugin called CalDav Synchronizer. Outlook's OOTB 'integration' allows you to subscribe to a Google calendar (one way sync from Google to Outlook) but does not support two-way sync (doh).
• I still haven't got category colours synchronising properly for mail or calendar.
• Outlook won't support category colours at all for Google mail because it's accessed via IMAP. That means that the UI has no 'categorisation' elements present at all unless you really dig for them.
• I had to implement another free 3rd party solution (GO Contact Sync Mod) to synchronise contacts; Outlook doesn't support this at all OOTB.
• Getting distribution lists to synchronise and using them effectively, for which I find you have to use hidden category colours and spurious meeting drafts, is basically a magic trick (I shall write a blog post on this). 
• Google mail's wacky lack of folders and realistic lack of any proper Archive function don't help at all. Why design it differently from every other email system on the planet? It's not as if it provided any fab new functionality (and Google's implementation of mail rules is a joke). Plus, Outlook hides Google mail items with no labels from view, presumably because IMAP can't see them.
• I have to say that the real problem with becoming more Google-centric is the support situation. When I think how much I have complained in the past about Microsoft Community, I am embarrassed. Google don't even pay lip service; their 'Help Communities' appear to be moribund and uncurated. (Having said which, Google Analytics does have a lively Discord channel with active and effective, if under-resourced, Google input, but I only found out about it by accident on Twitter.)

2. Regaining access to Teams.

• Done, by paying Microsoft an extra approx. £3 a month, but since I have used it a total of once I am not sure why I bothered.

3. Moving my cloud storage from Dropbox to OneDrive.

• Big success. Bye bye Dropbox, just too many annoyances.
• For less than my previous Dropbox subscription I can get Office 365 (including on the desktop), Teams (as above), and 1TB of OneDrive space.
• OneDrive integrates far better into Windows, e.g. you can set a folder to 'online only' automatically via PowerShell.
• It has effective support channels that don't assume you know nothing and/or that your question is daft.
• Plus, search in OneDrive is on another planet compared with Dropbox, which can't even see inside a .msg file, while OneDrive even indexes text on JPGs.
• And a shout out to the robocopy Low Disk Space Mode which allowed me to shift nearly 200GB of data from Dropbox to OneDrive fairly painlessly. 

4. Automation of mySQL backups.

• Done (for my locally hosted databases, anyway) via mysqldump.

5. Tidy up my cpanel hosting environment (deletion of loads of ancient experiments, etc.)

• Done.

6. Sort out non-mySQL backups for my hosting (I have never yet found a fully usable solution for this)

• Sync'd via winscp, which is slow but does a far better job than FTP.

i-Community:

7. Transferring the website across to Meridian to look after.

• Done.
• End of an era ... I had been involved in i-Community, and its predecessor the Catalyst/Notability/Logicalis IT Forum, ever since the summer of 2000. Lots of nice messages from members remembering IT Forum and i-Community Rochester trips.

www.notamos.co.uk:

8. Updating Google Analytics usage to use their new offering.

• Done. I shall be publishing a specific blogpost on this. (To say that Google don't make it easy is an understatement.)

Retirement Projects

 I've now retired from active customer work (on 30th March), but still have a fair few techie projects on the go, so I thought I would document them here.

General tidying up:

• Using Google Calendar as my day to day diary
• Regaining access to Teams
• Moving my cloud storage from Dropbox to OneDrive
• Digitising a lot of family photographs
• Audio and video
• Using Visual Studio and Azure DevOps properly
• Cleaning up various horrible scripting mechanisms to use PowerShell
• Automation of mySQL backups
• Tidy up my cpanel hosting environment  (deletion of loads of ancient experiments, etc.)
• Sort out non-mySQL backups for my hosting (I have never yet found a fully usable solution for this)

i-Community:

• Transferring the website across to Meridian to look after

www.notamos.co.uk:

• Updating Google Analytics usage to use their new offering

Genealogy:

• Various proposals concerning research and data quality aids (a lot of thinking and discussion still needed here)

Thoughts on StackOverflow

I've been having an interesting time lately engaging with the StackOverflow community at https://stackoverflow.com/.

I got involved because I had found lots of useful code on there when rewriting notamos.co.uk, and thought I should try to give something back.

It's been a mixed experience, mainly because I expected it to work like Microsoft Community, an environment with which I am very familiar from past Windows Phone forum involvement; wrong, because StackOverflow is explicitly aimed at building a knowledge base rather than at helping people work through problems. On StackOverflow you are supposed to ask an intelligent question, get an intelligent answer, and move on, with discussion/teaching discouraged (though by no means absent in practice).

The most assumption-challenging question I have encountered is (almost literally) 'I have written a complex Java-based web application, I want to put it live, what server should I put it on?'. But, far worse, so many developers appear never to have been taught exception handling, problem determination, basic relational database design, web application security, or even how to articulate a problem clearly. No wonder (to reiterate a regular rant) there is so much bad production software out there.

Using Blogger as a source of website content

For the last few years I have been webmaster for a local residents' association.

I put together a basic website using what I knew, which was html and shtml, and I created and maintained all the content (about 250 entries altogether) using that well known development tool Notepad.

Now it is time to hand the task over to others.

I realised a while ago that I had to find a better method of content editing, for speed, accuracy, and consistency, and also so that I could share the load with others with less technical skills.

Most of the entries are very short; a lot of them involve images; and many contain links to uploaded PDFs. Entries usually start off as 'highlights', linked to from the home page and notified to members via email and Twitter. Over time they lose their 'highlight' status; finally they are moved to the archive page.

I had a play with MODX, which we use for the i-Community website, but it's really far too complicated for the purpose (it's unnecessarily complicated for i-Community, really).

I had spent a bit of time in late 2017 learning some PHP (forced into it by the sudden demise of the Twitter feed mechanism on the i-Community home page) and a bit of Javascript (in order to implement an urgently needed non-Flash MP3 player for www.notamos.co.uk).

It occurred to me that I might solve the content editing problem via a Blogger blog like this one, pulling the content into the website via the RSS feed that is automatically provided for any Blogger blog.

Here's an example blog post (for a 'highlight'):

Here's its rendition on the website:

And here's the link on the home page:

The basics turned out to be pretty easy to do, using an open source tool called Feed2JS. This uses a second open source tool, Magpie, to read the RSS feed; the shipped version of Feed2JS, as the name suggests, then exposes the content via Javascript, with various formatting options.

I separated the content into the many necessary categories (News 'highlight', older News, News archive, Local Events, etc.) using Blogger's tags called Labels (each of which is available as an individual RSS feed), and automated the generation of the Feed2JS scripts (one per content category) via batch files.

So far so good.

The three remaining problems were: migrating all the existing content; implementing the 'highlight' links on the home page; and automating the 'last amended' date on each page.

Migration was a long and boring task. Eventually (and absolutely no thanks to Google's Blogger documentation or forum) I found some sensible import xml examples and succeeded in getting the content migrated, including an appropriate publication date for each entry, via a combination of manual editing, much changing of relative to absolute paths, SQL Server csv-xml conversion, Blogger import (not helped by some ridiculous daily import attempt limits), and manual application of Labels.

Here's an example (containing 2 entries) of the xml import.

The other two requirements turned out to be easiest to achieve by installing my own instance of Feed2JS. (I did have to put a redirect in place, effectively interpreting the Blogger site as a subfolder of the main website, as otherwise the website's PHP server was not prepared to access the RSS feed, for completely appropriate security reasons.)

I eventually used two much simplified copies of the Feed2JS code, with no Javascript; en route I took a considerable liking to PHP - a very friendly and easily learned language, it seems to me (I particularly admire the array handling).

The only remaining problem was the need to wait for Magpie's hour-long cache period to elapse before blog changes were reflected on the main website. In practice we just wait, although occasionally I do set the cache period to 1 minute temporarily (which has pretty awful effects on website performance, which is usually acceptable, if not as fast as the pre-blog version).

Finally I created a couple of little PHP utilities permitting the upload of PDFs and full size images to the website without FTP. (I subsequently had problems with these being used for malware purposes, which I should have anticipated - I have now moved them into a password-protected directory.)

Anyway it's all had the required effect: editing is now much easier, and my colleagues are happy to share the load.

Hints and Tips update (aka, SQL on #ibmi and #sqlserver: divided by a common language)

Decided to take a nice 'split string' table function I'd borrowed from some kind person on the internet for the Coordinate My Care data warehouse (SQL Server), and use it as a basis for a similar function to split multi-value attributes on IBM i, e.g. list of special authorities on a user profile. Cue a frustrating, if eventually successful, afternoon.

I really don't like #ibmi table functions: for starters, why can't I INSERT direct into the table to be returned, as in #sqlserver ?

However I did learn a few useful things that are worth sharing I think:

#ibmi Hints and Tips #18: create/replace table in QTEMP from SQL:
DECLARE GLOBAL TEMPORARY TABLE [table] ([col] [type], ... ) WITH REPLACE
(You can then use the table absolutely normally as QTEMP.[table].)

#ibmi Hints and Tips #19: FOR var1 as cur1 CURSOR FOR [SELECT statement] DO
[whatever, using column names direct from result set]; END FOR;
(I don't know whether you can do this in SQL Server, but I am definitely going to investigate.)

#ibmi Hints and Tips #20: table function usage: esp. note the final identifier (x here):
SELECT * FROM TABLE(MyTableFunction(parm1, ...)) x
(Don't like this super complicated syntax, either!)

Bath: The Graveyard of Ambition: Opera by Malcolm Hill: Saturday 19th August, 6.30pm

Making what you have work better: IBM DB2 for i SQL Performance Monitoring, Analysis and Tuning Workshop, IBM South Bank, 14-16 November 2017: not to be missed!

Mike Cain, of the IBM Labs, Rochester, Minnesota, a long term good friend of i-Community, will be coming to IBM South Bank on the 14^th, 15^th and 16^th November to run his fabulous “IBM DB2 for i SQL Performance Monitoring, Analysis and Tuning Workshop”.

This workshop will ensure that those staff using SQL will understand the impact of doing it right and wrong and understand the best practice methodology for using SQL and the supporting tools to write efficient and performant code.

I attended this course myself in 2009 and cannot praise it too highly.

The agenda includes:

·       Introduction to DB2 for i Query Optimizer and Database Engine

·       Database engine methods for data access and data processing

·       Indexing and Statistics strategies

·       SQL application design and programming best practices

·       Behaviour of static SQL and dynamic SQL

·       Best practices for popular SQL interfaces, such as CLI, ODBC and JDBC

·       Parallel database processing - DB2 Symmetric Multiprocessor (SMP)

·       State of the art tools, including System i Navigator Performance Centre

If you are interested, drop me an email at mandy.shaw@iperimeter.co.uk and I will put you in touch with the course organisers.

Still here and still busy!

Apologies for yet another big gap in posting. Again both work and domestic matters have got in the way, but I am looking forward to a less manic period when some of the blog posts I have in my head might make it here.

Sorry not to have followed up on the 'auction' post promised last time but I decided it might not be a public domain matter.

Quick update July 2015 to July 2016

Apologies for the complete silence - since my last post I have been busier than I ever remember.

Highlights of the last year (well, highlights in retrospect, in some cases):

• Fantastic trip to Brisbane (son's wedding), Sydney, and San Francisco last July (re wonderful SF public transport system with spot on approach to customer service, see right)
• Yet another excellent i-Community Rochester trip, September
• Successful go-live of POWER 8/IBM i V7R2/Flash storage at one of my insurance customers, November
• Magna Carta 800th anniversary concert, November
• Go-live of new Coordinate My Care system, also in November, after an 18-month procurement process and an 8-month implementation project (shouldn't it be the other way round?, I hear you say)
• Demonstration of CMC 2-way interoperability proof of concept at eHealth Week at Olympia, April 2016
• The Bath unexploded bomb (left under future school playground by Baedeker raids in 1942)

Next blog post will cover something rather alarming (broadly IT related) that my husband encountered when endeavouring to bid at an auction.

Calling PASE functionality from ILE #ibmi

I have recently carried out a proof of concept in this area - see this link for details and example ILE and PASE (AIX) source code. Any questions, drop me a note via my website.
UPDATE 21.10.17: the example code and PowerPoint are now on GitHub.com here: https://github.com/MandyShaw/iPerimeter/tree/master/IBMi/CallPASE

Unmissable course coming up! IBM DB2 FOR I SQL PERFORMANCE MONITORING AND TUNING WORKSHOP starting 24th September in London

If you have anything whatever to do with DB2 for i and you haven't previously attended this fantastic course, can I encourage you to consider attending? I did so myself in 2009 and have benefited greatly ever since.

Here's some more information:

IBM DB2 FOR i SQL PERFORMANCE MONITORING AND TUNING WORKSHOP (OD40AGB) 

This class will be taught by Jim Denton who is an experienced and highly skilled specialist from the IBM DB2 for i Centre of Excellence based in Rochester, US. Jim is one of the original authors of the SQL Performance Workshop and has been delivering this course, around the world since 2003.

Learn the information, tools and strategies for monitoring, analyzing and tuning IBM DB2 for i Structured Query Language (SQL). Explore the science and art behind the DB2 for i query optimizer and database engine, including database performance and scalability. Discuss performance issues affecting database design and data-centric applications. Get hands on experience with database performance parameters and options, as well as the various tools that assist in performance monitoring, analysis and tuning. Knowledge and skill gained by attending this course are considered critical success factors when building and deploying SQL based solutions.

More details:
http://edu.arrowecs.eu/uk/trainings/open/ibm-db2-for-i-sql-performance-monitoring-and-tuning-workshop/395921/training_sheet.html

Duration: 3 days
RRP: £1,700.00 per person (exc.VAT)
Date: 24th September 2013
Location: Arrow ECS London Office, Entrance 2, Fourth Floor, The Royal Exchange, London EC3V 3LN

i-Community Trip to IBM Rochester: Fantastic Agenda

Looking forward to setting off for what promises to be a superb event as always.

Wednesday 11 September, Executive Briefing Center

08:45     Continental Breakfast

09:00     Welcome and Introductions, John Rathke, World Wide Briefing Project Manager

09:15     IBM i Update and Directions, Steve Will, Chief Architect, IBM i

10:15     Break

10:30     DB2 for IBM i Discussion, Mike Cain, Team Leader DB2 for i Center of Excellence

11:45     Luncheon

12:45     PowerVM, Kris Whitney, Power Virtual I/O, SAN, Comm Development

13:45     PowerHA, Steve Finnes, Power Systems HA Product Offering Manager

14:45     Break

14:55     IBM i Storage Solutions, Nick Harris, SWG CPO, Storage Team Lead

15:55     Break

16:05     POWER Systems Update and Directions, Mark Olsen, Power System WW Product Management

Thursday 12 September, Executive Briefing Center

08:45     Continental Breakfast

09:00     IBM Director for IBM i, Tim Rowe, Business Architect of Application Development for IBM i

10:00     Break

10:15     Application Development Directions, Tim Rowe

11:30     Luncheon

12:30     Energy Management, Brad L. Brech, IBM Distinguished Engineer, STG Strategy Team and Architecture Board

13:30     Security – TKLM, Key Manager

14:30     Break

14:45     Performance Tuning – Awareness of iDoctor

Friday 13 September, Executive Briefing Center

08:45     Continental Breakfast

09:00     i25 and Lab Services Responsibilities, Ian Jarman, Apollo Category Marketing

09:45     BRMS – Backup and Recovery, Parallel Saves, Encrypted Backups, David R. Bhaskaran, BRMS Development

11:00     Electronic Support, Paula R. Fulton, Business Analyst, Electronic Support Transformation

12:00     Luncheon

12:45     Fix Support, Paula R. Fulton

Music of a Killer Prince: Chandos Singers of Bath, Saturday 6th July

Music of a Killer Prince
A concert marking the 400th anniversary of the death of the great Italian composer Carlo Gesualdo

The Chandos Singers, conducted by Malcolm Hill, are one of Bath’s leading chamber choirs.

The concert will begin with a celebration of the 850th anniversary of the laying of the foundation stone of Notre Dame Cathedral. The Notre Dame School produced the first multi-part sacred works to be known internationally, thereby having a significant influence on the development of Western music.

Carlo Gesualdo (1560-1613), Prince of Venosa, is known for two things: the fact that in 1590 he murdered his wife, and the passionately expressive and extraordinarily innovative choral music on which he worked for the rest of his life. Many have suggested that the two are linked. The Chandos Singers will be performing some of the most beautiful of his sacred works: Deus refugium, Reminiscere miserationum, Ave, dulcissima Maria, Gaudeamus omnes, Peccantem me, and Ave sanctissima.

The choir will celebrate the 60th anniversary of the Coronation of Queen Elizabeth II by singing a selection from A Garland for the Queen, commissioned from British composers by the Arts Council for performance on the eve of the Coronation. The Chandos Singers have chosen to perform the works by Ralph Vaughan Williams, Gerald Finzi, Edmund Rubbra, and John Ireland.

The programme is completed by a dramatic setting from the contemporary Finnish composer and linguist Jaakko Mäntyjärvi of Double, double, toil and trouble from Macbeth.

The concert will be held in the serene surroundings of the Magdalen Chapel on Holloway, Bath. A chapel is known to have existed on this site in the 11th century, and a leper hospital was built close by in the 12th. Both were under the care of the Abbey monks. The current building is 15th century. Despite severe bomb damage in 1942, it remains an active centre of worship.

Interval refreshments can be enjoyed in the Chapel garden, which has spectacular views over Bath.

Saturday 6th July 2013, 7.30pm
The Magdalen Chapel, Holloway, Bath BA2 4PX

Tickets: £12, students £6
Bath Box Office: 01225 463362 | boxoffice@bathfestivals.org.uk

Online bookings
https://uk.patronbase.com/_BathFestivals/Productions/CS2/Performances

Further details
Mandy Shaw 01225 436302 | mandy.shaw@iperimeter.co.uk

Chandos Singers
www.chandos-singers.co.uk
Magdalen Chapel
www.magdalenchapelbath.co.uk

Fantastic opportunity for IBM i people in the UK

The IBM DB2 for i SQL Performance Monitoring, Analysis and Tuning workshop will be held in London the week of 25 Feb. The excellent Mike Cain from the DB2 for i Center of Excellence - Rochester, Minnesota will be the instructor. More information about enrollment is here: http://www.edu.arrowecs.eu/uk/trainings/open/ibm-db2-for-i-sql-performance-monitoring-and-tuning-workshop/1672/training_sheet.html (I went on this course in 2009 and cannot recommend it too highly.)

#ibmi hints and tips #17: About reusing deleted records

I have been caught out so many times on this one ... If you create a physical file with CRTPF (whether using DDS or not), the default setting is REUSEDLT(*NO). If you create it using SQL, the default is REUSEDLT(*YES).

Interested in becoming a member of the IBM Electronic Support Client Advisory Panel?

I've been asked to publicise this IBM initiative for those with an interest in online technical support.

The Electronic Customer Support Advisory Panel meets on the 'phone every other month to review future plans for electronic support and to provide input about what they need from the Support web site.

This would take up about an hour every other month.

Participating gives members the opportunity to influence the enhancements and changes IBM makes to its electronic support; it also gives IBM a better understanding of what customers need and want in the next generation of support, and how IBM can address them.

If you're interested, I have more details and the necessary IBM contact information - you can get in touch with me at mandy.shaw@iperimeter.co.uk.

#ibmi hints and tips #16: A small fact about CPYTOSTMF and CPYTOIMPF

I have only recently realised the simple fact that CPYTOSTMF and CPYFRMSTMF work with 'program described files' (as IBM i still, scarily, calls them), i.e. physical files created using CRTPF without DDS, while CPYTOIMPF and CPYFRMIMPF work with proper database files, i.e. physical files created using either DDS or a SQL CREATE TABLE. I'm sure you were all aware of this, but just in case ...