Thoughts from a life in software

A while back I got to play with vectorisation of terms using a transformer model & Facebook Fais and I think there is a much better way. The idea is each document (block of unstructured text) within a corpus (collection of documents), is tokenised (group of terms). For example, if we had the following document:  The cat in the hat sat on the mat and drank milk from a jug. The child stared in alarm at the cat in the hat as that was his milk! In Natural Language Processing the first step is to remove stop words. Stop words are commonly used words within a language. They are typically used to join adjective, nouns, etc.. and so quickly dominate statistical analysis. For example "the" is a stop word. So removing stop words from our example gives the following: Cat hat sat mat drank milk from jug. Child stared alarm cat hat his milk Now we want to convert this into a series of tokens, the token size is dependent on your document (you don't want it to be larger than ...

After 10 years in DevSecOps, there is an assumption that that every project is unique and needs to deploy their own Continuous Integration (CI) instance and write their own Continuous Integration / Continuous Deployment (CI/CD) pipelines   However within a CI Pipeline you should be producing a build artefact which is supplied into a CD pipeline. Many CD Pipelines can be triggered by various means. This allows you to manage CI and CD pipelines separately.  While the mechanism and configuration of a deployed product can vary greatly between software products the CI pipelines face the same constraints which mean each project implementation is highly limited in the process it must implement. This blog will outline the reasoning behind the last statement There are only so many build systems Modern software languages have build automation systems and Dependency Management Systems, these aim to automate the mundane tasks in building, testing and releasing a software project. The ...

Many years ago I worked on a refresh of a simulation test tool. The tool presented a map display and you could place events on the map (e.g. a boat appears at location 50.1,97.3 at time 13:00 and reached 58.3,96.45 at 15:00) to construct test scenarios. The idea is you would build a set of scenarios to confirm a system correctly responded to events (system test tool).  The point of the refresh was to take the 'engine' of the tool (written in C++) and replace the ancient UI with a internal pluggable platform that embedded a Map interface into Eclipse Rich Client Program (RCP). We ported the requirements from the original tool and one of those was: The tool can open Scenario XYZ We obtained a copy of the scenario and added it to our weekly test pack runs, but we quickly ran into problems. Getting Scenario XYZ To Run  Scenario XYZ had thousands of events and the first few times we tried to open it the entire UI would crash. Single Threaded Application Each service was designed ...

As a Software engineer you will often come across positional coordinates, typically latitude, longitude & altitude. There are a number of coordinate systems, but we typically convert them into Decimal Degrees, which we handle as doubles. Typically the focus is on holding all data supplied for example if we received the following numbers we would store the entire value: Field Value Latitude 52.452383475328974532 Longitude 120.38794932875423 The issue is a latitude/longitude isn't a point in space, it represents an area. The way we measure a position we have a certain amount of uncertainty this is commonly known as circle error probable (CEP). If we have a device with a GPS receiver it might have a CEP of 10 metres, this means rather than the point we provide in the left image the actual position could be any point within the circle on the right. The circle centre position is no more likely to be the device position than an...

When we develop libraries & tools we define various means to interact with them. We do this by thinking about use cases and so we build in assumptions in how we expect our target users to typically interact with our work. In this sense everything built has an intended way of being used, a workflow.  When we want to achieve a goal, we should be asking ourselves how we want to solve the problem and looking for things which enable that approach. If we are constrained in the tools/libraries we can use, we should focus on how those tools are designed to solve the problem. The problem I've found is a lot of teams chase workflows which make no sense when combined with their tool choice. For me a great example of this was GitFlow branching strategy, because I have seen so many teams independently repeat exactly the same mistake. I'm going to take you through a bit of history to explain the issues. Trunk Based Development Before git existed, every source control management solution ...

On a Monday morning I went into work to find one of the systems down and it reminded me of one of the rules I had developed working out the 'Ops' side of DevSecOps. A lot of people gain experience in Operations, managing applications, these tend to be fairly limited in their scope (e.g. UI -> Middleware -> Database). this kind of system is typically event triggered and so changing the system and making a few calls against the UI will quickly test the entire chain of components. This allows you to immediately confirm the change was successful.  However as systems become more complex, a typical pattern I have seen in a few Open source and proprietary systems will create distributed state machines, typically there will be multiple services which require data to have certain fields with specific values. The service will then perform an action and once complete update the data object. Once you have a sequence of services operating in this manor you have a state machi...