anarchitekt

Mt. Kolmogorov: A Complex Climb

Risav Karna — Sat, 31 Aug 2024 11:03:51 GMT

In the second part of this series, we will take a quick quasi-philosophical detour and think a bit more formally about the core essence of code-generation and abstractions. Then we will quickly recap what we have built so far to compose a minimal description for an enterprise data CRUD+Search app.

1 Kolmogorov Complexity & MDL

Kolmogorov complexity measures the complexity of a sequence of symbols by determining the length of the shortest computer program that can generate that sequence. It can be understood, loosely, as a measure of how efficiently a sequence can be described. The lower the Kolmogorov complexity, the simpler the sequence is to describe. Now let's think of that sequence as being that of our finally generated program itself. My hypothesis is that most usual software, even with 100K+ LoC, have a more minimal description that can be used as a seed to generate the rest of the code. In few bespoke cases, this minimal description for the larger program is just the project specs and C4/UML diagrams. Often times, it something much simpler than that. Borrowing from NLP and computation linguistics, we will call these minimal descriptions and the length of such minimal descriptions as MDL.

2 (Meta) Kolmogorov Complexity of our Apps

Minimum description length (MDL), as a principle, states that the best model for a dataset is the one that minimizes the combined length of the model description and the data encoded using that model. This means that the MDL principle seeks to find the simplest model that can adequately explain the data.

N.B. "Data" is code in the context of code-generators.

MDL is thus essentially an application of the abstract concept of Kolmogorov Complexity, which on its own can only provide ordinal comparisons as opposed to more precise cardinal rankings from minimal description lengths.

In a data driven application, schema for data at rest and data over the wire dictate the kinds of services that can be possible. At the very least, the type signature, if not the behavior, of our data-facing service methods are bound to adhere to our schema's constraints and types. Given the MDL principle, and given that schema are generally ubiquitous, we can pivot to our schema to achieve the minimal length seed for our large programs. In cases where schema is implicit and needs to be inferred per collection, document, row or column, we would need the shape of some representative data records. This alone can make the seed description arbitrarily large but together with schema's constraints and guidance, the data representations can be a helpful addition to our MDL to infer data-shape-related interactions.

3 Schema and API

Say we have a database table or collection called user_profile and we use it to update user's preferences and profile information. How should the REST API providing interface look like? Now let's say that you have a table or collection about which you know nothing yet.

Can you still write the REST API for data create, update, read, search, evict or delete for that entity?

For a seasoned developer, across all major languages and frameworks, the answer is yes - in theory. Here's one example of what is possible with Prisma and its DMMF (from the last article) for a query controller:

export interface Queryextends NTTKey> {
    findUnique: (
        ntt: NTT,
        args: FindUniqueArgType,
    ) => ReturnType['findUnique']>;

    findFirst: (
        ntt: NTT,
        args: FindFirstArgType,
    ) => ReturnType['findFirst']>;

    findMany: (
        ntt: NTT,
        args: FindManyArgType,
    ) => ReturnType['findMany']>;

    count: (
        ntt: NTT,
        args: CountArgType,
    ) => ReturnType['count']>;

    aggregate: (
        ntt: NTT,
        args: AggregateArgType,
    ) => ReturnType['aggregate']>;

    fields: (ntt: NTTKey) => FieldRefType;

    groupBy: (
        ntt: NTT,
        args: GroupByArgType,
    ) => ReturnType['groupBy']>;
}

This interface is meant for a controller class exposing all of the ORM capabilities for querying the database. It is not tied to any particular entity, which I am calling NTT here. Now a frontend dev can directly use prisma to compose their own queries. Should a frontend dev be unaware of prisma? In that case, the backend dev can simply write a client sdk for the the frontend engineers. The main advantage is that no matter how many entities are there, there can be just one controller. Alternatively, there can be as many CRUD controllers as there are entities but we would need ts-morph for that. This will be the core theme for the next article in this series.

Can such a generalized query controller be implemented?

Yes. Fairly trivially it seems. Here's an example:

@Controller('query')
export class DBQueryController implements Query
{

  @Put(':ntt/find/unique')
  findUniqueextends NTTKey>(
    @Param('ntt') ntt: NTT,
    @Body() args: FindUniqArgType
  ): ReturnType['findUnique'] {
    return getDelegate(ntt).findUnique(args);
  }

  @Put(':ntt/find/first')
  findFirstextends NTTKey>(
    @Param('ntt') ntt: NTT,
    @Body() args: FindFirstArgType
  ): ReturnType['findFirst'] {
    return getDelegate(ntt).findFirst(args);
  }

  @Put(':ntt/find/many')
  findManyextends NTTKey>(
    @Param('ntt') ntt: NTT,
    @Body() args: FindManyArgType
  ): ReturnType['findMany'] {
    return getDelegate(ntt).findMany(args);
  }

  @Put(':ntt/count')
  countextends NTTKey>(
    @Param('ntt') ntt: NTT, 
    @Body() args: CountArgType
  ): ReturnType['count'] {
    return getDelegate(ntt).count(args);
  }

  @Put(':ntt/aggregate')
  aggregateextends NTTKey>(
    @Param('ntt') ntt: NTT,
    @Body() args: AggregateArgType
  ): ReturnType['aggregate'] {
    return getDelegate(ntt).aggregate(args);
  }

  @Put(':ntt/fields')
  fieldsextends NTTKey>(@Param('ntt') ntt: NTT): FieldRefType {
    return getDelegate(ntt).fields;
  }

  @Put(':ntt/groupBy')
  groupByextends NTTKey>(@Param('ntt') ntt: NTT, @Body() args: GroupByArgType): ReturnType['groupBy'] {
    return getDelegate(ntt).groupBy(args);
  }
}

NTTKey was introduced in the first part of the series and is simply a union of literal types representing all our table names. The client side SDK can essentially wrap over the Query interface shown above to make isomorphic type-safety like tRPC possible. For aiding with generics and type-safety, getDelegate finds us the right Prisma entity delegate needed to dispatch the JSON query to the DB query engine. Here's an example implementation of getDelegate:

import type { NTTKey } from './zen/entities-type';

export const prisma = new PrismaClient({
    errorFormat: 'pretty'
});

export const mapper = {
    Principal: prisma.principal,
    Account: prisma.account,
    Session: prisma.session,
    VerificationToken: prisma.verificationToken,
    Authenticator: prisma.authenticator,
    Payment: prisma.payment,
    Appointment: prisma.appointment,
    AppointmentType: prisma.appointmentType,
    Location: prisma.location,
    Patient: prisma.patient,
    Equipment: prisma.equipment,
    Service: prisma.service,
    Provider: prisma.provider,
    Form: prisma.form
} as const;

export type Delegateextends NTTKey> = typeof mapper[T];

export function getDelegate<T extends NTTKey>(key: T): Delegate<T> {
    const delegate = mapper[key];
    return delegate;
}

Notice how the Delegate type is constructed using our mapping object as a compile time constant. This is a fairly advanced usage of Typescript and the explanations for how I independently discovered this pattern would be a subject for an upcoming article in this same series. Meanwhile, I would like to show how something like FindUniqArgType from the interface above is built using the same type-mapping technique. In this case we are mapping over an interface instead of a compile time constant:

import type { Prisma } from "@prisma/client";
import type { NTTKey } from "./entities-type";

export interface FindUniqueArgs {
    Principal: Prisma.PrincipalFindUniqueArgs;
    Account: Prisma.AccountFindUniqueArgs;
    Session: Prisma.SessionFindUniqueArgs;
    VerificationToken: Prisma.VerificationTokenFindUniqueArgs;
    Authenticator: Prisma.AuthenticatorFindUniqueArgs;
    Payment: Prisma.PaymentFindUniqueArgs;
    Appointment: Prisma.AppointmentFindUniqueArgs;
    AppointmentType: Prisma.AppointmentTypeFindUniqueArgs;
    Location: Prisma.LocationFindUniqueArgs;
    Patient: Prisma.PatientFindUniqueArgs;
    Equipment: Prisma.EquipmentFindUniqueArgs;
    Service: Prisma.ServiceFindUniqueArgs;
    Provider: Prisma.ProviderFindUniqueArgs;
    Form: Prisma.FormFindUniqueArgs;
}

export type FindUniqueArgTypeextends NTTKey> = FindUniqueArgs[T];

This allows us to have a single generic type (FindUniqueArgType) in our interface for both the controller and the client-side SDK.

How is this code generated and kept up to date with migrations to the database? Here's a bit of ts-morph before the next article full of ts-morph:

 export function createTypeFile(
    project: Project,
    fileName: string,
    interfaceName: string,
    typeName: string,
    properties: { name: string; type: string }[],
) {
    const typeFile = project.createSourceFile(`${zenPath}/${fileName}.ts`, '', {
        overwrite: true,
    });

    /**
    * Adds:
    * import type { Prisma } from "@prisma/client";
    * import type { NTTKey } from "./entities-type";
    */
    typeFile.addImportDeclarations(essentialImports);

    /**
    * Adds 
    * export interface FindUniqueArgs {...}
    */
    typeFile.addInterface({
        name: interfaceName,
        isExported: true,
        properties,
    });

    /**
     * Adds
     * export type FindUniqueArgType = FindUniqueArgs[T];      
     */
    typeFile.addTypeAlias({
        name: typeName,
        typeParameters: [`T extends NTTKey`],
        type: `${interfaceName}[T]`,
        isExported: true,
    });

    typeFile.saveSync();
}

export const ModelNames: string[] = Prisma.dmmf.datamodel.models.map(
    (m: DMMF.Model) => m.name,
);

export function createProperties(suffix: string) {
    // Adds interface properties to the interface e.g. 
    // Patient: Prisma.PatientFindUniqueArgs;
    return ModelNames.map((ModelName) => ({
        name: ModelName,
        type: `Prisma.${ModelName}${suffix}`,
    }));
}

function createArgs(project: Project) {
    createTypeFile(
        project,
        'findUniqueArgs',
        'FindUniqueArgs',
        'FindUniqueArgType',
        createProperties('FindUniqueArgs'),
    );
    // Other interfaces go here in a similar way
}

export function initiateQb(): void {
    const project = new Project({
        tsConfigFilePath: './tsconfig.json',
        skipAddingFilesFromTsConfig: true,
    });

    createEntities(project);
    createEntitiesType(project);
    createArgs(project);
    // other generators go here

    project.saveSync();
}
initiateQb();

In all this, we have used information from the schema's AST only. Kolmogorov complexity is an idea "so abstract that it could just as well have been a prayer" - borrowing from the Foundation series.

So far we have generated a generic controller and seen a glimpse of how code can generate more code. In the next installment in the series, we will generate a separate query controller and a mutation controller for each entity. This will help us later create a CQRS system. Having a separate controller per entity will also allow us to apply various auth(n/z) guards and administrative policies for each API endpoint. Such guards and policy-enforcers can also be created by the code-generator itself as we will soon see in future installments in this same series.

Mountains & Software: Similar Peaks

Risav Karna — Mon, 26 Aug 2024 11:42:42 GMT

We have climbed similar mountains many times over across our usual projects. Going through these repetitions, I too have put together my toolkit for abstracting over implementation details each time, so as to scale the usual hurdles faster. This series explores such meta-programming, special trees, and the resulting code-generators I have built for my enterprise cloud automation and CI/CD integration use-cases so far.

Builders & Generators

In the world of rapid application development and 'no-code'/'low-code' app builders, code generators are a necessarily evil in the tool-chain. Code should be as DRY or repetition free as possible for maintainability. But that applies only for 'same code'.

When it comes to 'similar code', a lot of our code can be seen as blatant repetitions of "the same kinds of things" - if not "the same things". If your framework needs a new file for a new controller, your code-generator schematics can and should be generating that controller and its test files. This is possible and quite routine with nx, ng and similar cli tools. What if you know that you are going to need to run that generator for each of your hundred controllers?

A build system that integrates code-generators can add, update or remove parts of the code-base itself. Seen this way, code-generators are the 'actors' that generate code while builders are the 'directors' that supervise the action of code-generation itself. As for knowing which controllers to generate, there usually is a signal or annotation to find out. For instance, if you are writing a new controller layer for each of your user facing entities, you can mark them as such in your database schema itself or a configuration file somewhere. Same for per-folder view files in folder-based routers used by frameworks and meta-frameworks like SvelteKit, Remix, Next, AnalogJS and Astro.

Prisma ORM provides a way to annotate any part of your database schema (entity model, enum, attribute) with an active comment. Any comment marked with three slashes /// are picked up by cli-tools and libraries that tap into the capabilities of the ORM. In the case of a code-generator, this can be your own ts-morph scripts based on Prisma's database meta model format or DMMF.

Prisma & The Tree

Prisma's DMMF is an abstract syntax tree (AST) and an AST is one your best friends as a meta-programmer. Builders and generators can be guided by the AST in their internal logic. To work with your database's Prisma AST/DMMF, this is one way to expose it for programmatic use:

import { Prisma } from "@prisma/client";
import type { BaseDMMF } from "@prisma/client/runtime/library";

function getDMMF(): BaseDMMF {
    return Prisma.dmmf;
}

It's a one-liner to abstract us from how Prisma decides to expose the DMMF in a future version as these parts of the library are not public or expected to stay the same. This utility can now be used every where else in our code, say, to introspect which tables are present in the database of the currently running system:

export function getModelNames(): string[] {
    return getDMMF().datamodel.models
        .map(m => m.dbName ?? m.name);
}

Or to find metadata about the fields or columns in our table/model of interest:

export function getFieldMetadata(model: string) {
    const allModels = getDMMF().datamodel.models;
    const selectedModel = allModels.find(mod => mod.dbName === model);
    if (selectedModel) {
        return selectedModel.fields;
    }

    throw new Error("Model not found");
}

This isn't a particularly type-safe code code as the input is not all possible strings and the output is quite a complex type. Let's constrain the types as an exercise to get to the know the AST better and to make the above function safer to use. Let's click through the definitions in @prisma/client/runtime/library.d.ts as a starter:

// not exported, but you are advised to inline this in your code
type ReadonlyDeep_2 = {
    +readonly [K in keyof O]: ReadonlyDeep_2;
}

// can be used as a reference for how to deal with tables (models) and columns (fields)
export type Model = ReadonlyDeep_2<{
        name: string;
        dbName: string | null;
        fields: Field[];
        uniqueFields: string[][];
        uniqueIndexes: uniqueIndex[];
        documentation?: string;
        primaryKey: PrimaryKey | null;
        isGenerated?: boolean;
}>;

At this point we can type the getFieldMetadata function as follows:

export function getFieldMetaData(model: string): ReadonlyDeep_2<DMMF.Field[]> {..}

The string input type is still too wide but the output is well-typed for advanced use-cases. Wouldn't it be better if we knew our model names at the type-level, ahead of time? Thanks to the trivial getModelNames function, we can dump this information in a typescript file itself. That file would look something like this:

// zen/entities.const.ts
export const Principal = "Principal" as const;
export const Account = "Account" as const;
export const Session = "Session" as const;
export const VerificationToken = "VerificationToken" as const;
export const Authenticator = "Authenticator" as const;
export const Payment = "Payment" as const;
export const Appointment = "Appointment" as const;
export const AppointmentType = "AppointmentType" as const;
export const Location = "Location" as const;
export const Patient = "Patient" as const;
export const Equipment = "Equipment" as const;
export const Service = "Service" as const;
export const Provider = "Provider" as const;
export const Form = "Form" as const;

💡

The code snippets in this series are from in-production codebases from various business domains and verticals. This one above shows some tables from a clinic management app I recently built.

An exhaustive list of all models that need to be represented in our user interfaces can be generated like this into a file and can then be used as a type-level information in our future functions.

// zen/entities.type.ts
import { Principal, Account, Session, VerificationToken, Authenticator, Payment, Appointment, AppointmentType, Location, Patient, Equipment, Service, Provider, Form } from "./entities";
const Entities = {
    Principal, 
    Account, 
    Session, 
    VerificationToken, 
    Authenticator, 
    Payment, 
    Appointment, 
    AppointmentType, 
    Location, 
    Patient, 
    Equipment, 
    Service, 
    Provider, 
    Form
    // 100+ similar table names
} as const;

export type EntitiesType = typeof Entities;
export type NTTKey = keyof EntitiesType;

Now we can have a much more type-safe signature for our getFieldMetadata function:

export function getFieldMetaData(model: NTTKey): ReadonlyDeep_2<DMMF.Field[]> {..}

A big part of code-generation - or any advanced library code - is making illegal states impossible. In this case we have ensured that we only allow valid string names for our input and a read-only Field[] type as our output.

ts-morph & Morphisms

In this series, ts-morph will be our library of choice for demonstrating the file generation part of code-gen. The above file with table names as const can be generated as follows with ts-morph:

export const ModelNames: string[] = Prisma.dmmf.datamodel.models.map(
    (m: DMMF.Model) => m.name,
);

function statements(lCase = false) {
    return ModelNames.map((ModelName) => {
        const name = lCase ? firstCharLower(ModelName) : ModelName;
        return {
            declarationKind: VariableDeclarationKind.Const,
            declarations: [
                {
                    name,
                    initializer: `"${name}" as const`,
                },
            ],
            isExported: true,
        };
    });
}


function createEntities(project: Project): void {
    const entitiesTsFile = project.createSourceFile(`${zenPath}/entities.ts`, '', {
        overwrite: true,
    });

    entitiesTsFile.addVariableStatements(statements());
    entitiesTsFile.saveSync();
}

export function initiateQb() {
    const project = new Project({
        tsConfigFilePath: './tsconfig.json',
        skipAddingFilesFromTsConfig: true,
    });

    createEntities(project);
    createEntitiesType(project);
    // ... other createXXX functions
    project.saveSync();
}
initiateQb();

Generating the file zen/entities.type.ts from above works similarly, except for we have to import the constants from the earlier const file:

function createEntitiesType(project: Project): void {
    const entitiesTypeTsFile = project.createSourceFile(
        `${zenPath}/entities-type.ts`,
        '',
        {
            overwrite: true,
        },
    );

    entitiesTypeTsFile.addImportDeclarations([
        {
            moduleSpecifier: './entities',
            namedImports: [...ModelNames],
        },
    ]);

    entitiesTypeTsFile.addVariableStatement({
        declarationKind: VariableDeclarationKind.Const,
        declarations: [
            {
                name: 'Entities',
                initializer: `{${ModelNames.join(', \n')}} as const`,
            },
        ],
    });

    entitiesTypeTsFile.addTypeAlias({
        name: 'EntitiesType',
        type: 'typeof Entities',
        isExported: true,
    });
    entitiesTypeTsFile.addTypeAlias({
        name: 'NTTKey',
        type: 'keyof EntitiesType',
        isExported: true,
    });

    entitiesTypeTsFile.saveSync();
}

By this point, we have an example of generated code and a solid foundation for further generating controllers, test data, CRUD UIs and even basic visualizations for each of our models.

ts-morph is all it takes once we have a decent AST, like that of Prisma's DMMF. And we will be using a similar code-setup for generating most parts of a CRM that we would have manually (slowly) coded otherwise. Morphism is an abstraction over functions themselves. In our case, our basic morphisms are effects: code being written to a file. And the overall work is about composition of such morphisms.

Here's a basic mantra for this kind of code-zen:

Slide-y Thinking

Risav Karna — Thu, 25 Apr 2024 14:28:35 GMT

This is the live documentation of how we are creating the libre slidey-things library of slider components. We will be building a basic slider and iteratively adding features and polish to it. These are some basics to cover before we dive into keyframes, animations, pre-loading, caching, responsiveness, and browser-native optimizations.

Initial Conceptions

A slider may show at least one slide at once.
A slider can have a previous, central and next slide all visible at once.
Similarly, a slider can have many previous, one central and many next slides all visible at once (if they are very narrow in terms of width, assuming horizontal slides).
Sliding is defined as translating a slide from point A to point B.
Sliding and slides are not merely architectural/software constructs, they are also visual and human co-structures.

A Minimal Version

So far, we have established that a slide being shown may mean a translation to the center of view.

When implementing it in a browser natively without a build tool, we have to focus on HTML, CSS and JS first.

Here's how the HTML for such a slider may look like:

      <div class="slider">
        <div class="slide-l slidey">
          <img src="img/public/square/galaxy.png" alt="Andromeda">
        div>
        <div class="slide-c slidey">
          <img src="img/public/square/brain.png" alt="Pre-frontal Cortex">
        div>
        <div class="slide-r slidey">
          <img src="img/public/square/big.png" alt="Data Architecture">
        div>
      div>

We only have a div around each img element and a parent div classed as .slider. Presumably, we would need some CSS for

the left image (.slide-l)
the central image (.slide-c)
the right image (.slide-r)

And they all might share some common CSS config, which we could put under the .slidey class.

Various users of the sliders would want to use the slider differently and might have different tastes and design preferences.

We will keep our implementations open for extension and closed for modifications. Essentially, this necessitates immutable data structures and thinking of code / developmental work itself as a graphical data structure.

Let's grab a reference to the parent div first:

    const slides = document.querySelectorAll('.slider');

One way of thinking about these slides would be to think as if we are meant to unroll it as a film out of a film roll.

In JS, this could look like this if we think of pulling out each roll via cogs attached to each of the film's edges for pulling into view:

    function showSlide(index) {
        slides.forEach((slide) => {
            const slideWidth = slide.clientWidth;
            slide.style.transform = `translateX(-${index * slideWidth}px)`;
        });
    }

This is for showing a slide that is n index away from the original or zeroth position.

We extract the slideWidth of a given slide by iterating through each item inside the slides element we queried for earlier. Every image moves to the negative X direction i.e. the left of the screen.

If your nth image is moved to the left by n times the slideWidth, it would have hopped that width n number of times towards left.

Why does it make sense to initially do it this way?

If you are n steps further away, you need to move yourself n times to get to the original or zeroth position.

How big are each of those steps? Let's just say you move yourself just one unit to the left.

If we make this 'unit' a constant for each slide, it would not really cover the required distances.

If we are, for example, constraining all the images to the same height and allowing each image to be of any aspect ratio / width, we need to consider each image's journey / distances individually.

When thinking in natural numbers, we think in terms of unit increments and decrements also aside from n number of unit increments or decrements.

If we are able to have an action and an inverse operation for that action, e.g. both a forward and previous button / action associated with our slider then we would have something close to what is called a 'ring' in mathematics.

Let's try to have a simple pair of nextSlide and prevSlide functions that act as the inverses of each other. By this, we generally mean that doing one action and then the next will lead you back to your original position or state in general.

    let currentSlide = 0;

    function nextSlide() {
        currentSlide =(currentSlide + 1) % slides.length;
        showSlide(currentSlide);
    }

    function prevSlide() {
        currentSlide = (currentSlide - 1 + slides.length) % slides.length;
        showSlide(currentSlide);
    }

Apart from using let in JS, it is also a bit flaky because it has two places that try to modify the same value: the value of the currentSlide index. Such mutation oriented code needs to be dealt with with care because of the complexities that arise in reading and writing the same value from different optical vantage points. However, since we will try to keep it as simple as possible without it being useless, we will keep the index as a closure state within our function.

Holding the state of the current slide in memory requires us to think of how we want to place and manage this in memory. In JS, an idiomatic way to achieve this is by using closure inside a functional context. In mathematics, closure is a property that usually implies that operations do not change the type of outcome. Similarly, in programming, closure is a similar property that is bounded within a lexical or functional scope, like in the following example:

    function sliderContext(cssClass) {
      const slides = document.querySelectorAll(cssClass);
      let currentSlide = 0;

      function showSlide(index) {
        slides.forEach((slide) => {
          const slideWidth = slide.clientWidth;
          slide.style.transform = `translateX(-${index * slideWidth}px)`;
        });
      }

      function nextSlide() {
        currentSlide =(currentSlide + 1) % slides.length;
        showSlide(currentSlide);
      }

      function prevSlide() {
        currentSlide = (currentSlide - 1 + slides.length) % slides.length;
        showSlide(currentSlide);
      }

      return {
        nextSlide,
        prevSlide
      };
    }

Here we have created a function that takes an input and returns an object with a couple of functions as an output.

Depending on whether you have a functional or imperative upbringing, you might call such a thing a higher order function or a 'module pattern'.

In theindex.html file, with the above DOM elements for the sliders, one could add such functions as actions to dispatch when a user event like click happens.

Here's one simple way of initializing this:

        <script type="text/javascript" src="js/slidey/slider.js">script>
        <script>
            const slideContextA = sliderContext(".slidey");
        script>

And somewhere in the DOM, where you have your buttons, you could do this:

    <a class="prev" onclick="slideContextA.prevSlide()"><a>
    <a class="next" onclick="slideContextA.nextSlide()">>a>

We obtained these two functions to use in our HTML because the module or sliderContext function has returned that to us for use in the call site, which happens to be the index.html page itself in this case.

However, it is not always preferable to attach event listeners to the HTML itself like this, as this does not scale well with more complex use cases and large number of DOM nodes being involved.

HTML is meant to be a language for semantic markup, and usually is best used for that itself. We can attach event listeners to HTML DOM elements within the JS itself. The trade-off in that case is that the JS needs to know at least some identifiers to find the reference or know how the HTML is structurally as a DOM tree.

Either way, there is not going to be a true separation of concerns between semantic, visual, and causal markers in a language meant for a structural description of a document.

With user interface components like a slider, it is not always the case that a user interaction triggers an event. A slider could, for instance, automatically play in a loop forever or for a finite time span.

Here's how one could add options to auto the slides in a loop:

    const defaultOptions = {
      auto: false,
      infinite: false,
      time: 2000
    }

    function sliderContext(cssClass = ".slidey", options = defaultOptions) {
      // same code as before
      // ...
      // logic for looping:
      if (allOptions.auto) {
            const intervalRef = setInterval(nextSlide, allOptions.time);
            if (!allOptions.infinite) {
                setTimeout(clearInterval(intervalRef), allOptions.time * 2);
             }
      }

      return {
        nextSlide,
        prevSlide
      };

  }

Here we have decreed that the nextSlide function be called every 2000ms or any other finite duration. We have also said that if the slide is not supposed to be infinite, it should stop looping after a certain predefined amount of time.

A Minimal Requirement Analysis

We might need to have a lot more images in a slideshow. This also does not seem like it will be smooth without the appropriate CSS for the transitions and/or animations.

In a more complex and publicly usable slider, what else would be required of the slider HTML components and the JS/CSS backing them? Let's explore that in the next Slide-y Thinking session.

A Stack Attack

Risav Karna — Mon, 05 Apr 2021 12:11:48 GMT

Let's Find Ourselves Our Stack Inventory

We have now had a common work group, work space and work shop for a multi-disciplinary software/socioware team for about half a decade. For Project Ouroboros and C@ Ana, we mostly focused on contrasting and comparing the simple and dependent-type theories and finding the right use cases of type theory and categories in real-life systems running at home and in the wild. Here's a bunch of languages and their paradigms, which might also inspire your next codebase, stack or coding style - just as it did for us.

Universal Stack Languages: ABCs to XYZs

Agda

Agda is a dependently typed functional programming language. It has inductive families, which are like Haskell's GADTs. However, they can be indexed by values and not just types. It also has parameterized modules, mixfix operators, Unicode characters, and an interactive Emacs interface (the type checker can assist in the development of your code). Agda is also a proof assistant, it is an interactive system for writing and checking proofs. Agda is based on intuitionistic type theory, a foundational system for constructive mathematics developed by the Swedish logician Per Martin-Löf. It was originally developed aiming to support a wide range of type theories, but the current version supports dependent types, extensible records, coercive subtyping and universe polymorphism.

Bytecode (Graal VM/JVM/LLVM)

Bytecode is a highly optimized set of instructions designed to be executed by a software interpreter. The most common example of bytecode is Java bytecode, which is interpreted by the Java Virtual Machine (JVM). GraalVM is a high-performance runtime that provides significant improvements in application performance and efficiency which is ideal for cloud environments. It is designed for applications written in JavaScript, Python, Ruby, R, JVM-based languages like Java, Scala, Kotlin, and LLVM-based languages such as C and C++. LLVM on the other hand, is a library that is used to construct, optimize and produce intermediate and/or binary machine code. LLVM can be used as a compiler framework, where you provide the "front end" (parser and lexer) and the "back end" (code that converts LLVM's representation to actual machine code).

C/C++/Carbon

C and C++ are general-purpose programming languages which have been around for over 40 years. C++ is an extension of C, and it provides many additional features, such as classes and objects, which support the Object-Oriented Programming paradigm. This paradigm is useful for organizing large codebases and for creating reusable code. Carbon on the other hand, is a source-to-source programming language designed for writing maintainable and scalable code.

Datalog

Datalog is a declarative logic programming language that is syntactically a subset of Prolog. It is often used as a query language for deductive databases. Its applications in databases include integrity constraint checking, complex event processing, and logic-based B2B integration.

Eiffel

Eiffel is an object-oriented programming language emphasizing the design and construction of high-quality and reusable software. Its model of computation is based on Hoare logic and Meyer's design by contract, where assertions form a contract between subroutines and their callers.

Fortran

Fortran (formerly FORTRAN) is a general-purpose, compiled imperative programming language that is especially suited to numeric computation and scientific computing. It is one of the oldest programming languages still in use.

Gleam (Beam VM)

Gleam is a statically typed functional programming language for building scalable concurrent systems. It compiles to Erlang and runs on the BEAM virtual machine used by Erlang and Elixir.

HaXe

Haxe is a high-level cross-platform multi-paradigm programming language and compiler that can produce applications and source code, for many different computing platforms from one code-base. It is free and open-source software, distributed under the GNU General Public License (GPL) 2.0, and the standard library under the MIT License.

Idris 2

Idris 2 is a purely functional programming language with dependent types, optional lazy evaluation, and features inspired by Haskell and ML. Idris 2 is a complete reimplementation of Idris and is implemented in Idris itself.

Julia

Julia is a high-level, high-performance dynamic language for technical computing. It provides a sophisticated compiler, distributed parallel execution, numerical accuracy, and an extensive mathematical function library. Julia’s Base library, largely written in Julia itself, also integrates mature, best-of-breed open source C and Fortran libraries for linear algebra, random number generation, signal processing, and string processing.

sKotlin (ScalaJS + Kotlin UI monorepos)

Scala.js is a compiler that compiles Scala source code to equivalent Javascript code. That lets you write Scala code that you can run in a web browser. Kotlin provides a safer and more functional paradigm-based approach to using the JVM, Java, and Scala ecosystems.

Lean

Lean is a theorem prover and programming language designed to bridge the gap between interactive and automated theorem proving. It provides a dependent-type theory with a small number of basic axioms and inductive definitions.

Miranda

Miranda is a lazy, purely functional programming language, designed for teaching and research and for specialized industrial applications. It has a strong polymorphic type system and supports higher-order functions.

Nim

Nim is a statically typed, compiled systems programming language. It combines successful concepts from mature languages like Python, Ada and Modula. Its design focuses on efficiency, expressiveness, and elegance (in that order of priority).

OCaml

OCaml is a general-purpose programming language with an emphasis on expressiveness and safety. It is a member of the ML language family, which includes Standard ML and F#. INRIA, a French research institute, has developed and distributed it since 1996.

Prolog

Prolog is a logic programming language associated with artificial intelligence and computational linguistics. It has important role in artificial intelligence. Unlike many other programming languages, Prolog is intended primarily as a declarative programming language: the program logic is expressed in terms of relations, represented as facts and rules.

Q

Q# (pronounced as Q sharp) is a domain-specific programming language used for expressing quantum algorithms. It was initially released to the public by Microsoft as part of the Quantum Development Kit.

Racket (Racket + Rust Backend monorepos)

Racket is a general-purpose, multi-paradigm programming language based on the Scheme dialect of Lisp. It is designed to be a platform for programming language design and implementation. Racket is also used for scripting, computer science education, and research. Rust is a similarly optimized and powerful systems development language with a type-safe approach for accessing system internals and borrowing references politely.

Svelte 5

Svelte is a modern JavaScript compiler that allows you to write easy-to-understand JavaScript code, which is then compiled to highly efficient code that runs in the browser. You can write your code using techniques that make it easy to understand, and Svelte will do the work of translating it into highly efficient code that manipulates the DOM.

Typescript

TypeScript is a statically typed superset of JavaScript that compiles to plain JavaScript. It adds optional types, classes, and modules to JavaScript, and supports tools for large-scale JavaScript applications.

Unison

Unison is a new programming language currently under active development. It's a modern, statically-typed, purely functional language, similar to Haskell, but with the ability to describe entire distributed systems with a single program.

XSLT

XSLT is a language for transforming XML documents into other formats (like transforming XML into HTML). The original document is not changed; rather, a new document is created based on the content of an existing one. XSLT uses XPath to define parts of the source document that should match one or more predefined templates. When a match is found, XSLT transform that part of the source document into the result document.

YAML

YAML (a recursive acronym for "YAML Ain't Markup Language") is a human-readable data-serialization language. It is commonly used for configuration files and in applications where data is being stored or transmitted.

Zig

Zig is a general-purpose programming language and toolchain for maintaining robust, optimal, and reusable software. It is an imperative, statically-typed language designed for robustness, optimality, and maintainability.

Time - III [Draft]

Risav Karna — Sat, 09 Jan 2021 06:55:19 GMT

The Crisis in Infinite Quantum Futures

Time is also in a quantum crisis. In the previous post in this series, we have touched upon the symbolic and enumerative 'crisis' of time and the possibility that time may not exist as a feature of the symbolic and syntactic universe as it is but rather a feature of our modes of understanding of the symbolic universe. In this third part of the series, we will focus on two ways of understanding time and how these can be represented in mathematics or physics terms. Hopefully, this will build up the toolkit we require to easily dissect the more complex philosophical and technical arguments I would like to present in this third installation and the upcoming part of this series.

These two conceptions of time are also highly problematic. The classical deterministic view and the quantum probabilistic together have ugly mutual contradictions, transfinite regressions, and outdated metaphysical arguments. Furthermore, even mathematics and physics representing the various aspects of these 2 different possible universes do not claim to define time but rather just measure, calculate, or represent these conceptions of time. We will be visiting Non-Euclidean spaces and nullary spaces where time may even become a point or a line as opposed to an imaginary number or a real number in the previous post. I am getting ahead of myself here so we will start at the beginning of the philosophy of science's being and becoming in the west and then jump at least 2 millennia to find the oriental Vedic sciences again.

Quantum Physics and Quantum Field Theories

(This also assumes some sort of conservation, especially conservation of mass-energy for physical entities. This would be more fodder for this post on infinitude.)

Vedic Mathematics and Astrophysics

(This infinite eternal fire would be an interesting topic for this post when it becomes relevant for our discussions of infinite streams in programming and infinity's types in the Mathematics' (An)architecture series or for the upcoming theoretical quantum computer science topics.)

The Being and The Becoming

I would argue that both Heraclitean ‘Panta Rhei’ and Parmenidean ‘ex nihilo nihil fit’ are arguments, sustained or otherwise, for two very different albeit interlinked contexts.

Categorical Dual Theories of Time

Saying time flows is just as ridiculous as saying space flows. Our existence marches across time, not the other way round.

“Now he has departed this strange world a little ahead of me. [...] That signifies nothing. For us believing physicists, the distinction between past, present, and future is only a stubbornly persistent illusion.”

Time on 1, 2, 3, or more planes

The Higher Dimensions of Time and The n-Onions

MAP IV: Mathematics as a Language - I

Risav Karna — Thu, 29 Oct 2020 17:36:45 GMT

Is Math Invented or Discovered?

Whether mathematics is discovered or invented is an age-old question but it could also have been an innocent false dilemma all along. Could it be something in between or beyond?

An alternative approach to answering this could be that mathematics is a formal language that is constructed using formal structures and operations on those structures. More precisely, mathematics is a system for deciding whether mathematical statements about a mathematical structure are true.

So what could make a statement mathematical? Tautologically, any statement that follows mathematical rules could be called a mathematical statement.

Is that last sentence in itself an exhaustive rule? That is, are there other ways to write or construct a mathematical statement? Mathematics can sometimes be about something more than a system of rules or instructions. In fact, this simple rule-based 'kind' of mathematical statements is usually a small subset of mathematical statements that map directly to the kind of statements we write in imperative programming. As an example of this think of the following multiplication as a 'mathematical statement':

 1234
 x 56
______
 7404
61700
______
69104

We teach this kind of lifeless and tedious instruction set to our kids to teach them multiplication when they are starting to learn mathematics. And of course, most of them grow up to hate some parts or all of mathematics. Well there are some that do not mind these punishments. And that's how you have imperative programmers or drone-workers. However, these instruction sets in arithmetic come from the algebra of numbers, which is slightly more abstract than merely adding 1234 to itself 56 times as an arithmetician. Such algebras form mathematical structures as opposed to mere mathematical rules.

One of the Rings

Integers form a useful structure called a ring. And a ring is a set that just so happens to have two operations, + and · , defined on its elements and the second one, which is ·, happens to be distributive. Furthermore, it has both right and left distributivity. With this information, we know that:

1234 · 56 

= 1234 · (50 + 6) 
= (50 + 6) · 1234 

= 1234 · 50 + 1234 · 6
= 1234 · (5 · 10) + 1234 · 6

= (1 · 1000 + 2 · 100 + 3 · 10 + 4 · 1) · 5 · 10 
+ (1 · 1000 + 2 · 100 + 3 · 10 + 4 · 1) · 6

= (1 · 5 · 1000 + 2 · 5 · 100 + 3 · 5 · 10 + 4 · 5 · 1) · 10 
+ (1 · 6 · 1000 + 2 · 6 · 100 + 3 · 6 · 10 + 4 · 6 · 1)   

= 61700 + 7404
= 69104

In the first two pairs of steps, we demonstrate that integers have left and right distributivity under the second operation ·. In the subsequent two lines, we use associativity of this same operation i.e.

1234 · (5 · 10) 
= 1234 · 5 · 10
= (1234) · 5 · 10

Then we substitute 1234 with 1 · 1000 + 2 · 100 + 3 · 10 + 4 · 1. Can we take it for granted that such a substitution is allowed? If we are being serious about the grammar of our language, we cannot just assume that such a substitution is allowed.

Digression into Programming

We have mentioned, albeit not clarified, in previous posts in the MAP series, that propositions and types are isomorphic. So in a sneaky typed-programmer way, we can say that this substitution is allowed as long as + is a pure operation and, therefore, referentially transparent. Purity simply means that for the same two operands, the operator always returns the same value and nothing else - not even any hidden consequence or side-effect. Referential transparency for an expression like 30 + 4 means that the expression is replaceable with the value of the expression e.g. 34. How are we sure that merely 'purity' preserves this 'replaceability'?

Well, it is because we are guaranteed that along with the purity, we also have the rule that rings are closed under both operations i.e. + and · are always going to produce an integer result, in the case of the integer ring, or a real result, in case of the more general polynomial ring. So we know that the types of both operands and the results are the same. This is already good enough but insufficient support in our claim for allowing substitution. The more important question however is how are the semantics of + or ·operation defined?

Let's Play the Peano

For this, we have to build numbers themselves. If we are building up numbers from scratch it is easy to construct them from the Peano axioms (rule-based) or from Church numerals (function-based). Once you have integers and define how various operations are defined on integers, it is quite easy but not trivial to construct the reals.

A Peano number is either a 0 or a successor of another Peano number. In Haskell, we define Peano number values like this:

data Peano = Zero | Succ Peano

Obviously, Zero could correspond to the natural number 0. Do we have 1 in this system yet? It could not be the first rule, which only produces Zero and it can only be one of these two rules but not both. This is because the pipe | represents a disjoint union or an 'either-or' mode of selection in this case. As for the second rule, Zero is the only Peano number we know so far. So the second constructor Succ has only Zero as a candidate thus far. Note that Zero has type Peano, and Succ has type Peano -> Peano i.e. it takes a Peano number and returns a Peano number. We can alternatively say that Succ is closed under Peano numbers. In that sense, Peano numbers are instead of 0, 1, 2, 3, 4... simply Zero, Succ Zero, Succ (Succ Zero), Succ (Succ (Succ Zero)) and so on and so on.

Note that we have not defined any rule that allows for any Peano number x such that Succ x evaluates to Zero. This is because we do not have negative numbers (yet) and thus Zero is not the successor of any Peano number. We can still define ring-like operations on these numbers but without the inverse operations, we only have a semi-ring or just 'rig' - a ring without the 'negative' operations.

Take me to Church

Peano numbers define numbers in terms of either a Zero or repeated application of Succ initially on Zero and then on any subsequently formed Peano numbers. In the same way, Church numerals are defined in terms of how many applications of a function is needed to 'reach' or count to that number. This is shown in this handy Wikimedia image below:

We will examine the lambda expressions on the right in further detail in the later posts in this series. The important point to note is that a number is essentially the count of how many times a function had to be applied to a base number, which is again 0 in this case too. Note that we could have started the count at 1 instead of 0 like a heathen. It is more useful to not do something reproachable of that sort.

Do we have a formal semantics of + and · using this foundation for numbers? Turns out we are quite close. Even the unary Succ above can be thought of as a + operation with one of the operands 1. That is, Succ Zero is the same as 0 + 1 in our above school-level arithmetic that we started with. However, please note that it is not necessary to stick to a Succ as boring as that. We can also just as well define Succ a to be equivalent of a + x where x is an arbitrarily chosen but fixed Peano number. In fact, Succ a could just as well be a - x (which would generate negative numbers) or even 2^a (which would generate powers of base 2). Both formulations of Succ would lead to a set of 'numbers' of the same size - even though that size or cardinality is the same countable infinity. However, we have not formally defined such addition, subtraction, or exponentiation yet so using any of the examples in this paragraph to define Succ would be cheating.

In this case, it is actually easier to not cheat and simply write the definition. Let's start with the notion of what it means to have a number n in our system:

As seen in the last two images, 0 is simply the result of no applications of f. 1 is the result of one application of f. 2 is the result of f f. 3 is the result of f f f and so on and so on. We use this to motivate the intuition behind the notion of +:

If you have n applications of f lying around and I bring you m applications of f, and if we compose those together, we have m + n number of applications of f. According to our rule, m + n many applications of f results in the number m + n. Note that + is closed under addition. So we know that the result of the expression m + n is the same as the value m + n, which is still a Church numeral. So can we now define *? It is actually even easier.

If you have n applications of f lying around, just apply all of that m many times. That result is m * n.

Invented or Discovered?

So have we invented addition and multiplication for integer/polynomial rings or Church/Peano numbers? We probably cannot claim to invent that which has pre-existed our invention of it. So did we discover them instead? This is a slightly more acceptable proposition because we did 'run into' these operations. In the same sense that Columbus 'discovered' the Americas by getting there after so many other explorers and settlers, we did 'discover' addition and multiplication from our first principles.

A better approach to answering this question would be that we have neither invented nor discovered any mathematics. We have simply constructed a formal* language with which addition and multiplication are defined both intensionally and extensionally. We started with setting out basic rules like what it means to have a number and then we decided what we can do with the set of that numbers. We constrained ourselves to map from a number to a number instead of mapping a number to a fruit or a planet. We thus required a closed nature of operations. We then added further constraints about what the unary Succ or f can do. This unary operation allowed us to define what it means to have something after the base number Zero or 0.

f (f Zero) is 2 applications of f. Similarly, f (f (f Zero))) is 3 applications of f. We then thought of what it would mean to compose the first number with the second one. Such a composition gives us f (f (f (f (f Zero)))) or 5, which we defined as an addition. If we had just repeated the first number as many times as the value of the second number, we would have f (f (f (f (f (f Zero))))) or 6. The reason addition and multiplication behave this way is that we defined them to be exactly like this. Thus we have just been creating structures (e.g. the ring Z) and then defining what operations are allowed (e.g. +, · ). We can do something similar when playing a logical game like the zero-player game called Game of Life with the following production and termination rules for cells in an infinite grid of rectangular cells.

Any live cell with two or three live neighbours survives.
Any dead cell with three live neighbours becomes a live cell.
All other live cells die in the next generation. Similarly, all other dead cells stay dead.

Interestingly this allows for us to create a complete Turing machine on merely a cellular grid. This means that we have a computer with just 3 foundational rules (or first principles) and that anything that can be computed at all can be computed with just this eccentric grid (structure) and the introduction/elimination rules (operations) just like we did with the ring and its two essential and invertible operations. Is it better to say that the late John H Conway discovered the Game of Life or that he constructed it?

I bet it is the same answer for whether this great mathematician, or any other for that matter, discovered or constructed mathematics.

WW II and WW Covid: Diffs and Parallels

Risav Karna — Fri, 18 Sep 2020 04:10:57 GMT

We have a global war against the allied forces of COVID-19, Climate Change, and Jingoistic Conflicts. However, this is not the first time we have found ourselves in a global war and - if we survive this one - it is not going to be our last.

Here's an account of what some of the greatest scientists agreed to do in the name of finishing a war the last time we had to face a large scale existential threat:

https://youtu.be/7pSqk-XV2QM

Our hope is that this time around, there is a better solution than sacrificing numerous human lives to make a phallic symbol of death-threat. This was in-fact less civilized than a dog marking its territory with urination as that does not kill entire cities of innocents just to stop a fight that would kill even more.

Yet we like to talk of every large scale project solving a problem X and requiring our top intellectual minds as a 'Manhattan project for X'. It would be our collective failure as a civilization if the top minds we have even after more than half a century still are not any wiser than those in that original Manhattan project.

Reintroducing nep.work

Risav Karna — Fri, 11 Sep 2020 19:46:35 GMT

nep.work in a nutshell

We are a community that develops omnichannel platforms for sales and delivery of services through work contract exchanges and pre-purchases.

Or at least that is how I introduced nep.work on Jan 1, 2020. Note that, as developers of a general-purpose enterprise platform, we always mention services in a general sense and never in just the context of some specific type or range of services. However, by February, nep.work founders were starting to work more than "full-time" on our new-found devotion towards highly specific services providing COVID-19 combating solutions. The current beta of nep.work is thus essentially a COVID oriented rethinking of everything we wanted to do before.

We are still a community - a bigger, stronger one than before with many cross-disciplinary teams.

We still develop omnichannel platforms - but we now focus on the entire supply chain, not just on pre-purchases, sales, and delivery of the services.

Requirement re-engineering and business workflow studies

However, we are proud to be now specializing like this to create a software ecosystem that is COVID-aware and, in many cases, redesigned with COVID concerns at the front and center. Because of this, we have also had a stronger focus on human-with-human interactions around our systems too instead of just human-with-machine interactions via our software interfaces on any given screen or projection. Thus, COVID has also forced us to rethink nep.work's core project BOSS (Business Optimization & Security Services) where business workflows are the primary concern and human-machine interactions were the initial primary concern.

To this end, we have also worked with various Nepali organizations, big and small, to understand how classically accepted business-customer interactions can best be made digital without loss of monetary value or legal acceptance. We have also created robust integrations for project management, team management, and project-related bookkeeping for both technical and non-technical business projects.

We have already been able to digitally revitalize a few businesses dying of COVID recently and are hoping for a lot more such uplifting stories. Our depth and width of integrated services for Nepali businesses are unparalleled and going by our enterprise feature sets' growth rate, we will remain the ones breaking our own records and raising our own bars for quite a long time.

Payments: Simplified so much, it is mathematically impossible to further simplify

One big elephant in the room, that we wanted to actually just shoot and be done with, was the problem of money transfer as it kept coming up as a requirement in most problems our platform attempts to solve. We did not want to do anything directly related to fiat money but we also do not want anyone else to do participate in that communal madness. So, we just thought of the very last, minimal, and complete money send-receive solution we would need.

And we built exactly that. The least amount of unique information that uniquely identifies you in the merchant network (like that provided by the leading payment gateway in Nepal) is enough to send and receive money. The system still requires AI-assisted KYC/AML* related form fill-ups but there is no hiccup after that. You can literally just drag and drop some money into your friend's pockets. We were also thinking of strangers who just want to give back to the community by giving a little something to you in times of need. So, we came up with simplified UI with a drag-and-drop semantics or just a type-and-filter semantics for our maximally streamlined payment systems. Essentially you just need to select a friend from a dropdown and the amount is sent or you just drag and drop different amounts of money to different circles representing different users each. This is handy in a poker-tournament-like situation or when you need to send money as a group of users to another group of users for, say, a charity event.

We think it is a dangerous idea to build AML-detection ML/AI/Semantic algorithms alone. Such huge responsibilities are best handled by a group of cross-disciplinary experts.

Digital Contracts and Exchanges

Work contract exchanges and pre-purchases are still at the core of what we do. Notice that the exchange and trading of contractual documents are vastly different and more abstract than just creating or enforcing those contracts. Contracts are at the core of all assets and trades and thus also a first-class citizen of the nep.work platform.

Existing work contracts for each employee or business partner can get scanned and digitally signed. The nep.work-parsed contract now understands the concept of day-to-day business workflows and individual responsibilities distributed among the parties in the contract. All of these functional and behavioral descriptions of the workflow of businesses are analyzed by our system using the related contracts in the system. All the project management, team management, and customer relationship management capabilities are thus driven by the business logic derived from the contracts and agreements between all parties involved. Knowledge of the contracts also helps us in our additional plugin services for bookkeeping, customized ledgers, and product inventory management solutions. When your team members are working on any part of the contract, the system can now be aware of the current context e.g. patient searching a product category on the website or an employee looking up a return policy for a certain item. The former would receive intelligent assistance to help with the shopping - if determined to be needed. The latter could receive a complete takeover of the system in contextually looking up the specific business document needed.

So, you can think of us as an e-commerce platform for trading service agreements and insurances (e.g. for event organization and logistics) instead of just physical products.

As a semantically and contextually aware business assistant. As an operations research expert. An accounting assistant. A warehouse manager. Your trusty day-book and ledger maintenance staff at work.

And as a hospitality platform operational at every accessible venue instead of a particular location (e.g. hotel, stadium, government offices, hospital, website, app, etc).

And then there are also these little things that we said since the very first time that we are, and still are proud to be:

Omni-channel

A single channel, physical or digital, is not always enough for modern e-commerce and high-frequency transactions. Neither is a basic multichannel strategy. For every nep.work trading agreement related interactions between providers and consumers, we strive to bring together all available digital and physical real estate, payment gateways, and logistical solutions. As we will have a singular (but distributed) functional core, these various interaction points for the market will always be in synchrony and harmony with each other. That said we will still have service providers with brick and mortar shops and traditional online catalog-sales style webshops and apps.

Platform

Our sole product and exclusive service is the work exchange platform and its customer support program respectively. We have a foundational infrastructure for a contractualist service exchange, with or without money, backed by immutable, reliable and publicly verifiable structures such as blockchain and in-memory Merkle DAGs. The platform also has formal and socio-technical mechanisms to verify proof of work (PoW) for a given fulfilled contract. A verified PoW can be used for adding a new block of data to our blockchain and related persistence systems.

Sales and Delivery

Our services can be sold as contracts, insurances, and barter agreements. When money is involved, the transactions can be made using local payment channels or our on-delivery cash service. Payments in kind or barter may be included before, during and after the delivery as per the specific Trading Agreement (TAg). We also have logistics partners and will soon announce a unified access API and SDK for our TAg management, payments, and delivery.

Work Exchange

Our work exchange platform goes beyond a freelancer network for posting and grabbing project offers. After a service request or offer is posted, there is either an open or closed negotiation between the provider and consumer parties. Optionally, an auction of the type chosen by the author of the post may be created to bypass negotiations and find optimum pricing options without involving senile Walrasians.

Work exchange also enables us to remain self-organizing and self-preserving. We have learned from similar communities on the web and throughout history and created a public reputation-based trust system.

Providing such services in itself is also a part of our work exchange program. For every project, the project's team may have various developers, artists, sponsors, volunteers, reviewers, and even the audience itself due to our democratized innovation and feedback mechanisms. Membership in certain negative income tax service provider communities can help offset your monthly income in case it dips below a predefined baseline. Similar 'buffer financing' or insurance services also exist for the projects in the nep.work community. Finally, most communities and workgroups have a meta group that is responsible for critical reviews and future roadmaps of the group.

But we are also so much more.

And in many cases, words have not yet been invented for what we do. For instance, we have a new generation of blockchains that handle more than a million transactions per second whereas the fastest ever yet - Futurepia - is at ~300K TPS.

Unlike most of the blockchain community, we absolutely despise the decadence that cryptocurrencies have created instead of solving bigger problems.

Alike most of the community, we believe that the core ethos of verifiable and accountable systems backed by immutable sources of truth will bring some sense and order to our otherwise chaotic world filled with COVID, quakes, fires, floods, and a lot more.

prox-e

Risav Karna — Wed, 26 Aug 2020 11:07:40 GMT

prox-e provides simple REST APIs and dashboard to showcase CST's core functionalities & a simple dashboard to demonstrate what the system and the APIs above can do.

docs

Github pages driven by eleventy SSG.

sync.sh

The ./sync.sh script utilizes the sh scripts in sh folder to update all the current branches with the latest from develop.

tp

TP stands for Transport Protocols and Transport Protobufs and Top Priority. It has three important sub-projects.

channels

All pub-sub and stream channels are available under this single API and SDK.

nginx

All the nginx conf for all the projects will be made available here.

protocols

All the OpenAPI specs and spec drafts will be made available here.

hi

All sub-projects in the hi project group are of high priority. It has three distinct sub-projects.

couch-potato

This is currently just a collection of geojson blobs for CDN and redundant blob storage on edge networks.

cov-hub

This is the dashboard and the central activity hub for all things related to covid for everyone as a member of any given organization or as an individual.

prox-e

This is the additional Open API server for communicating with our system and acts as a proxy to our single source of truth - core db.

md

All sub-projects in the md project group are of medium priority. It has three distinct sub-projects.

indy

This is currently just a collection of geojson based id blobs for CDN and redundant blob storage on edge networks.

indy-pay

This is the dashboard and the central activity hub for all things related to payments for everyone as a member of any given organization or as an individual.

indy-vot

This is the additional Open API server for communicating with our system and acts as a proxy to our single source of consensus - core kdb.

lo

All sub-projects in the lo project group are of underground projects that serve as enablers for the other projects above. This group has three distinct sub-projects.

khetala

This is currently just a collection of geojson based id-metadata blobs for CDN and redundant blob storage on edge networks.

medigai

This is the dashboard and the central activity hub for all things related to medical ikigai for everyone as a member of any given organization or as an individual.

indycol

This is the additional Open API server for communicating with our system and acts as a proxy to our single source of collaboration - curedit.

Behavior Specs' Specs

Risav Karna — Sun, 19 Jul 2020 07:02:18 GMT

Introduction to Behavior Driven Development

Software development and project management is general traditionally focuses on a plan for success. This is not good enough for mission-critical and civilization scale software that also must plan for failure and for moving from failure or incompleteness to success or release regularly.

Initially, covidsim.team’s dashboard (user interface for data visualization and inputs) and related systems like our self-hosted cloud drive (collaborative editors and storage for Word, Excel, PowerPoint, Images) and supercomputer management software (custom VPS manager) were meant only for internal usage within the research team. And since I was the only one working on the system since 2015, I did not find it necessary to formalize my software development process with a development team in mind.

Now that covidsim.team is committed to testing the usage of some newly created forms by our public health team using this platform, we are going to need a proper development process for this national scale ambition. This BDD-based process is applicable for any quadrant in the Cynefin model.

Picture from: Harvard Business Review

These forms and visualizations of related data that we are about to roll out are quite trivial parts of the underlying system, technically speaking. However, these are the primary way for any user to interact with our COVID-specific system components currently and they effectively form a portal to the rest of my collaborative systems. Hence it is vital to us to define the behavioral specifications for each of the newly designed forms, data files or visualizations. Without a clear consensus on the new behavior, the new feature cannot - and therefore will not - be made. These specifications will be stored on cosys.work's Nepali servers only. We will require approvals and editing from a single point of contact from the customers and me or a software lead we appoint in the future. Updates to the documents should be notified via emails.

BDD Examples, Links and Resources

Example from: blogs.harvard.edu/markshead

A BDD feature file consists of one or more scenarios. These scenarios are just examples of how the application should behave from the standpoint of a user. For example, you might have a scenario that says:

Feature: User Login

Given I am on the login page

When I attempt to login with valid credentials

Then I am shown the application dashboard

And another scenario that says:

Given I am on the login page

When I attempt to login with invalid credentials

Then I am shown a login error message

These examples are written in English, but are still very structured. The Given portion tells the starting condition for the example, the When line tells what you actually do, and the Then tells what results are expected.

More formally speaking, the given clause is there to define the pre-conditions and context for a new feature. Similarly, the when clause is there to define the user action or a software event that may need a reaction or response from the system. Finally, the then clause is there to describe the reaction, response or output expected from the system for this scenario.

Adding feature description underneath the feature title is advisable but not usually necessary for trivial and well understood features like a normal login flow. You can also add freeform text descriptions under headers like Example, Scenario, Background, Scenario Outline and Rule followed by a colon (:) just like with Feature: above.

Another thing to note is that these Given-When-Then behavior specifications are not just documentational or prescriptive. They become a part of the source code of the software. The modules that have these specifications only perform what is specified in the Given-When-Then specs - nothing more, nothing less. These specifications and the software itself may change as frequently as the real-world requirements of the software change e.g. every second or week.

User Story -> Agreement -> Implementation

User's needs are told as user stories. Thus a user story is a story created about how a user might use the system. These are usually sent over to the development team by the customer organization or written by the project manager. A user story usually follows the format:

As a [WHO]

I want [WHAT]

So I can [WHY]

E.g.:

As an admin user

I want to be able to create and delete non-admin user accounts

So I can add new employees as users to the system whenever needed

Sometimes it is okay to just have 'As a user' as the first sentence. Since this is a common role and context, it is okay to bundle a bunch of user stories under a single 'As a user' block. After agreeing about such a story and the necessary features, a team makes a BDD spec for that user story. It will typically consist of a few Given-When-Then described features with logical connectives like 'and', 'or', 'not', 'with', 'but', 'once' or 'never'. Note that popular BDD tools like Gherkin only permit 'and', '*' and 'but' aside from 'given', 'when' and 'then'.

For example, our first user story could be about how someone from a municipality office logs in to her/his municipality’s system and what pages and data the user could expect to see and use. This story may have different scenarios e.g. whether or not the login credentials were valid, like in the earlier examples from Mark Shead.

Another user story we might need soon after launch is the story that imagines and describes how a user from EDCD/NPHL/NHRC/CCMC/MOHP could upload to the system. In this case, we may assert in the given clause that the user is already logged in. We could also create a user persona e.g. Biplav is a non-technical user at EDCD who is responsible for EDCD's side of data updates to the system every day.

So, this story could initially look like this:

Feature: CSV File Upload

Given Biplav is logged in to the dashboard

When Biplav clicks the CSV file uploader in the left sidebar

Then Biplay should be able to select, preview, review, upload or cancel the upload

These kinds of user stories may need further breakdown. This breakdown itself will also consist of more GWT sets. For example, this GWT set further explains what the system should do:

Feature: CSV File Upload

Scenario Outline: Invalid CSV File Review

Given Biplav has selected a CSV file to upload to the database using the file uploader and Biplav has not yet confirmed the upload

When The system informs him of any mistakes like "" mistake

Then Biplav should be able to edit the CSV headers and/or data before he cancels or confirms upload of the data.

Examples: Mistakes

| csv-invalid |

| invalid number of rows |

| a non-standard font or language is used |

| the file is encrypted or corrupted |

We can have many examples of inputs, outputs or options in general in the example section to clarify things.

A Given sentence can be directly followed by a Then sentence too. It is usually okay to put such statements as a rule e.g.:

Rule: Selecting an invalid CSV file should not disallow correction and eventual upload of the CSV file.

Note that we use this same structure every time because this has to be understood by software too and not just the humans.

Gherkin is a popular software interpretable GWT syntax. It is used to run behavior driven tests in the Cucumber testing framework. You can find the reference for the Gherkin syntax here.

Picture from: cucumber.io

How the system could behave is a crucial part of the development process. This is the discovery phase and is typically gathered via interactive sessions and workshops with some of the potential users of the system. The discovery phase allows for surfacing and formalizing organizational rules and a shared understanding.

An even more important part of development is the finer details of how the system should behave. This is done in the formulation phase. We create specifications, mocks, wireframes and break user stories into feasible epics or tasks in this phase.

Automation phase is about how the system actually behaves. Validation and demonstration of each automated process may lead to further discoveries and formulations.

The Couch & The Potato

Risav Karna — Sat, 18 Jul 2020 03:14:40 GMT

The Couch Potato

The Couch

When you are in the couch, you just need to be. The couch provides it all.

The Potato

The Potato always does something. The Potato does it all.

Socioware Specs: High Hopes

Risav Karna — Sat, 18 Jul 2020 01:24:27 GMT

a. Bottoms Up

The following article is a follow up article about covidsim.team's call for collaboration. Contributing to the software systems will be easier once we have a common specification and description of the systems in place.

Encumbered forever by desire and ambition

There's a hunger still unsatisfied

Our weary eyes still stray to the horizon

Though down this road we've been so many times

Pink Floyd, High Hopes

1. UX and Services

User interfaces and interaction patterns of this system are built orthogonally and in bulk. Because of this we are able to create new user interface elements using component factories and resolvers. We are able to do this on the fly instead of handcrafting them each time there is a new emergent use case. Therefore, most of this document is dedicated to the rest of the 6 layers in our octal system in which two UI/UX systems cover user inputs and visualizations.

The user experience is designed with a decentralized burden of responsibility. Every interface and API response type describes its own visualizations and pipelines. Hence simply streaming them through a UI component resolver is enough to get a new feature displayed to the user for use. This only defines our time to user. Our time to user interaction is however still dependent on how fast we fetch all the interaction endpoint services.

2. CouchDB and prox-e

The client is a light weight UI and windowing system for ambient computing. It also powers VR/AR interactions apart from Smart TV and Smart phone interfaces. The NoSQL database provides all the necessary data to the local system. All data processing APIs for CouchDB data is able under Potato API provided by a prox-e ubiquitous server instance.

2,50,00,000 (25 KK) pouch instances, 753 couch instances and 7 potato instances will be needed for an optimal run in our system launch for Nepal. We are possibly starting with Lalitpur Municipality in the central Kathmandu valley and in Karnali remote province.

3. Redis and pub-sub

More intensive applications aside from the dashboard will be needed for other spreading disasters and calamities. For example, for flood-detection we need to be monitoring weather patterns and running the stream against our climate knowledge base pipelines. Similarly, an earthquake response system will guide us out of any building and towards safety only if we provide our software system with the authentic and real time 3D API of our surrounding reality. Since every concept in the system has its own data streams and behavior sets for any given context, it is easier to just fan-out using pub-sub and close-in using topics and filters.

Each topic has associated filtering processes related to identity, auth, context and content. Each component has multiple topics of linked data. A higher order component handles many such components including lazy loading components as needed. Our dashboard is capable of providing a windowing system overlaid on a map or some other 3D surface because every user experience of the system is channeled through a collection of components which in turn are a collection of pub-sub topics.

4. State and Data

Our applications on their own and in integration have different system states. This data needs to be available and persisted. Without the ability to freeze and resume, we cannot really address all possible failure scenarios. Our systems use state and data management to travel back and forth in time and test itself in parallel even when it is being used.

Because fuzzers and testers of this system continually emulate all possible states and data interactions - using actions, effects and reducers, we are able to have high confidence in the system. And in the new parts of the system that the overall system creates for itself based on the results of those tests and fuzzers.

b. Reactive Components

1. Abstract Components

2. Auth & Context UI filters

3. UI Factories & Resolvers

4. Labels & Processes

c. NgRx Breeze

1. rx rsmq

2. Hibernate & Spark

3. Git-like Command Queues

4. Being & Performance

d. Top Down

1. Concept

2. Relation

3. Knowledge

4. Representation

Mastery isn't the problem, Slavery is

Risav Karna — Thu, 16 Jul 2020 09:00:04 GMT

Why Git repository providers needn't rename `master`

Mastery of a certain concept or skill is probably not our problem. Mastery over other people's human rights is. However, we delegate such mastery to structures of power all the time. This is because of an outdated social contract with unregulated late stage capitalism.

To actually do something beyond just a symbolic rename, GitHub, for instance, just needs to actually remove different forms of mastery because of organizational and managerial constraints.

For there to be any freedom to work but work unlike a slave, we need freedom to not work and still exist. This basic idea is connected not just to universal income but deeper constructs as well, e.g. negative income taxes and basic social equities.

We need to be trained before work

Learning to work needs to happen before work. This is especially so as to not let it mix up the organizational roles like 'interns' or another vague / undescriptive title.

With qualification, we need to have matching contractual offers. This is handled on an abstract and extensible + self-hosted way within nep.work core.

Removing `master`

Renaming master should be a local choice of the repository owners, not a globally imposed constraint from structures like Microsoft's GitHub.

This can be done with an easy script that could just be added to any CI/CD pipeline.

Suggestions from Papa

Risav Karna — Fri, 10 Jul 2020 22:33:36 GMT

Some great tips from John Papa and why you should (not) follow them:

RxJS Subscriptions

The first one is from a 2019 ng-conf talk where he suggests that one should unsubscribe from all our subscriptions reliably once we are done with them, typically in ngOnDestroy.

It would be unfortunate to need to install his proposed / friend's subsink library just so you can add all subscriptions to it and then unsubscribe from all of the using a single unsubscribe to the sink. RxJS Subscription allows you to do this natively. Here's a code snippet from the Firebase RxJS guide:

import { interval } from 'rxjs';

const observable1 = interval(400);
const observable2 = interval(300);

const subscription = observable1.subscribe(x => console.log('first: ' + x));
const childSubscription = observable2.subscribe(x => console.log('second: ' + x));

subscription.add(childSubscription);

setTimeout(() => {
  // Unsubscribes BOTH subscription and childSubscription
  subscription.unsubscribe();
}, 1000);

You can also abstract away this repeated unsubscribe in ngOnDestroy business as mentioned by Wojciech Trawiński here:


import { OnDestroy } from "@angular/core";
import { Subscription, Observable, PartialObserver } from "rxjs";

export abstract class SelfUnsubable implements OnDestroy {
  private subSink = new Subscription();

  ngOnDestroy() {
    this.subSink.unsubscribe();
  }

  protected subscribe(
    observable: Observable,
    observerOrNext?: PartialObserver | ((value: T) => void),
    error?: (error: any) => void,
    complete?: () => void
  ): Subscription {
    return this.subSink.add(
      observable.subscribe(observerOrNext as any, error, complete)
    );
  }

  protected unsubscribe(innerSub: Subscription) {
    this.subSink.remove(innerSub);
    innerSub.unsubscribe();
  }
}

Now you have a less error-prone way of managing subscriptions without any external library and just good ol' OOP.


import { Component, Input, OnInit } from "@angular/core";
import { interval, Subscription } from "rxjs";

import { SelfUnsubable } from "./selfUnsubable";

@Component({
  selector: "counter",
  template: `
    Count {{ count }}
  `
})
export class CounterComponent extends SelfUnsubable {
  count = 0;
  private manualSub: Subscription;

  ngOnInit() {
    // unsubscribed manually 
    this.manualSub = this.subscribe(interval(500), value => {
      this.count++;
      console.log(value);
    });

    // unsubscribed when component is destroyed
    this.subscribe(interval(1000), value => {
      this.count++;
      console.log(value);
    });

   setTimeout(this.unSubManually, 600);
  }

  unSubManually() {
    this.unsubscribe(this.manualSub);
  }
}

But wait, there's more. We can still do better.

Try your best not to subscribe in the first place!

Why is it that you are introducing imperative programming by subscribing to your reactive streams in the first place? The async pipe is there to handle exactly this problem. You can even use OnPush change detection strategy if you are just putting the stream directly into your components. This is much more effective than relying on zone.js based default change detection strategy. Here are two helpful talks on this topic:

https://youtu.be/-4cwkHNguXE

https://youtu.be/Z76QlSpYcck

NgRx Data

As NgRx developers are cautious to point out, you do not usually need NgRx. By this they usually mean that your app probably does not have use cases to justify the hundreds of lines of code bloat that NgRx requires for each of your state models.

A neat boilerplate code reduction can be achieved by @ngrx/data. If you are already using NgRX, it goes without saying that you should at least consider using this library to significantly reduce your store related code's size. However, if you are not already using NgRX, should you use it now that @ngrx/data is there to simplify things for you?

I personally favor using a full-fledged client-side immutable database instead of some immutable'store'. This has become easier than ever since PouchDB showed up and brought sanity to in-browser database-like solutions. Do you really need all the indirections, actions, effects, reducers, dispatchers, and selectors for each of your models? Most, if not all, of the time, you just need a basic reading of pure functional programming practices and an actual database with immutability or append-only behavior. Also note that actions, effects, reducers, dispatchers, and selectors can be thought of as general coding patterns instead of NgRX exclusive ideas.

So, is NgRx completely avoidable for even the most complex state management use cases? Your mileage may vary with the Couch-Pouch-like solution for state management. It even allows for syncing the current state to a remote server, possibly running async analytics, e.g., on user behavior, with very few additional lines of code. There is a lot more that you usually need to do with your application state than just put it in a 'store' manager. Keep it all simple by just using a database and immutable data structures.

You can use PouchDB even with databases with topic-wise event hooks like the Realtime and Firestore databases from Firebase or with web APIs with webhooks or push / pub-sub semantics. In any case, complex UI states can be easily stored and managed in client side in-memory stores or browser-based databases with or without state management solutions, even with boilerplate reducing solutions like NgRx Data, NgXS or Akita.

One particularly remarkable use case of NgRx is the possibility to create truly reactive updates to Angular without using zone.js. Here is an excellent talk from Mike Ryan regarding this exciting possibility.

https://youtu.be/rz-rcaGXhGk

Best practices and patterns for complicated software from big companies are not always the best for simpler projects. I will add more such (anti)-patterns here.

Anarchitecture & Skunkworks

Risav Karna — Fri, 03 Jul 2020 15:30:22 GMT

This article describes the road map for research & development in the area of enterprise collaboration architecture, unified research and sharing of intellectual and digital assets. We then describe how this enables nep.work to operate a Skunkworks-like cross-organizational framework for radical innovations not otherwise possible within the confines of organizational silos.

Introduction

Multiple pioneering players in industry are working on overlapping interests. While competition between individual companies is the key to driving innovation in a free market, it is not particularly optimal for the market as a whole because it involves significant duplication of similar efforts and investments. Redundancies within the systems of production especially in a resource constrained nation like ours can be reduced by our proposed model of forming large collaborations and fostering competitions between collaborating groups as opposed to that between individual companies. This way companies can also level the playing field when it comes to the more concerning competition with large foreign business entities.

Skunkworks is a corporate practice that originated 77 years ago as a radical approach for quick delivery of better products than the competition. In Lockheed Martin's eponymous case, the end product was a fighter jet expected to fly 200 miles per hour faster than the best in competition - high-speed German fighter jets, which were literal threats - but to be completed in just 180 days. This originally utilized an isolation-based approach to enable radically innovative work. In its current form, it still bypasses bureaucracy via selective isolation but it judiciously crosses departments and even organizations especially when given the right tools and protocols. Ideally companies should be continuously innovating and not needing a dedicated Skunkworks team. However, we need to initially resolve the problem that low capital companies can only do incremental innovations despite much needed large-scale breakthroughs for the overall market, owing to a less literal but devastating threat from multinationals and powerful foreign players.

Mass adoption of some of these enterprise technologies, both prospective or already developed, imply a tectonic shift in industry. Although these are not moonshot projects, these are still significantly challenging for most companies to carry out on their own. To this end, we have selectively consolidated parallel research and development efforts in multiple partnering companies to create a unified front that encompasses key areas of high interest to the business community. These projects will be open sourced under the stewardship of NAAMII's I & I committee, which will also include various pioneering business leaders, and will have representations from the overall business community via a community process with an open RFC driven decision making. The main focus is on collaboratively solving the industry's present problems with the most significant returns on investment or impact on business workflow optimization.

Concepts, Ontologies & Network Variants

Businesses and business processes have an inherent redundancy and re-actions involved. To do more than what we are doing now, we need to automate what we are doing now. To automate processes, we should start with annotating processes. We need to identify processes and connect them as needed. Each of the nodes in such a system is essentially model-able as a node in a social-political hyper-graph. Some of the nodes may be an individual. Since these nodes are admitted in a type agnostic way, any kind of node may exist. Some nodes may just be friendship's graph friends or some may be just representation of business workflow for writing, editing, approval, signing, sending, receiving, acknowledging and responding documents. These documents can form companies, organizations, projects, business deals and execution project's team playground.

What to Not Do

Since we are dealing with broad topics and scopes, we need to focus the efforts and scopes especially when getting started. In doing so, we also need to see what needs to be done first and what else can be done along with it. We propose starting simply with the beginning - entities and identities in a CoSys knowledge base (KB) or database (DB).

Where to Begin

We begin and end this document with the matrix. This was written around 2012-13 and has been the effective outline till the next decade.

Identity

An identity is a key with which the credential server returns the entity itself.

Entity

Any construct or structure in our DB or KB that has an identity is called an entity.

ID-IP

Identity that is e2e and p2p connected to identity's IP (intellectual property).

Codes

Code can be 1 dimensional in case of numbers or non-nested arrays.

Algebras

Codes have tuples of inputs which form algebraic structures and algebras.

Sciences

Code libraries of academic contexts and custom constants and constraints.

Environment

Code integrations form end to end systems in a given Environment.

PI-ID

Property IDs that are e2e and p2p connected to identity's IP and contexts.

Q Queries

Code query params can are always part of a unit dimensional array.

I Space

Codes' IDs are in a tuple space with custom algebraic structures and algebras.

J Resource

Codes live within distinct resources of academic contexts and constraints.

K Environment

Code integrations form end to end systems in a given environment.

Are there things that have no identities? What can we do about those that do? What should we not do? Simply put we do not do anything other than GET, PUT, POST and OPTIONS.

Thoughts on Covidiocracy

Risav Karna — Mon, 13 Apr 2020 21:37:12 GMT

Cover Illustration is from XKCD#2278

I would like to note a few concerning semantic and naming gymnastics pulled off by the media and WHO during this crisis. This list might become a lot longer in the future once I find the time to return back to it:

Social Distancing: We actually need to grow closer together than ever. What we need is 'physical distancing' from both the people and the things they touch. Very different things and very different kind of messaging.
COVID-19 virus: This name for the virus instead of the actual technical name SARS-CoV-2 (the virus that causes the COVID-19 disease) makes it sound as if this is a 2019+ problem. It completely hides the fact that this family of virus was known since 1930s and that human-infecting SARS related coronavirus have been known and are being warned about since the early 2000s.

Let's not have any doubt on whether this is indeed SARS 2.0. It is. And it is here because we have been ignoring all the expert warnings about the next re-emergence of SARS and even defunding antivirus researches for SARS related coronavirus.

Kung Flu/Chinese Virus: Wuhan Virus would have been consistent with past nomenclature but giving racist nicknames or attaching nationality as opposed to a region of origin wouldn't be fair.
Lockdown: A lockdown is a harsher word for a strict quarantine and conjures imprisonment related imagery due to semantic adjacency with 'lock up'. It's still more acceptable than 'curfew' though. In some places, stronger words might actually be necessary. However it brings resistance from the people in countries who are skeptical of their government's power grabs.
Infection/Death count: Although it is implicit in the name, these numbers only reflect the known infections or known deaths. Either 'reported' or 'confirmed' as qualifiers might be needed more often than not in public communications. Even the richest countries have not had enough tests done per capita to know the correct numbers.

To make things worse, countries are scrambling to save face and keep their actual numbers hidden. NYC recently had a very unusually high surge in 'heart attack' related deaths. China somehow has ridiculously low number of new cases per day.

This part of the rant was inspired by Bill Maher making some sense with his unpopular opinion in his New Rule monologue on the issue of naming the virus as per the academic tradition, among other things.

Speaking of power-grabs, here is a chilling Economist article about how the world will probably never be the same as countries move towards a new order.

"But history suggests that a return to pre-covid days is unlikely. Two lessons stand out. The first is that governmental control over the economy takes a large step up during periods of crisis—and in particular war. The second is that the forces encouraging governments to retain and expand economic control are stronger than the forces encouraging them to relinquish it, meaning that a “temporary” expansion of state power tends to become permanent."

If you are playing on the side of the virus, and many people inadvertently are, a change of world order might not even seem unexpected or even news since outright destruction and global scale genocide is not beyond the realms of possibilities.

In this regard, Ndemic's Plague Inc been my favorite strategy cum simulation game for quite a while. It has become depressingly prophetic, just like Black Mirror, but in a much more precise manner.

I have been infecting and killing the entire global population on and off for 5+ years now. In different scenarios and with different pathogens, it has mostly been the case that

starting with low severity of symptoms and low fatality rate makes most of the governments ignore the looming crisis till its quite late
emergence of new kinds of symptoms at a later point, e.g. diarrhea, means that you can start killing effectively with those sweet, free and inevitable mutations
the poor countries receive the most deaths and damages in any outbreak
densely populated countries and low income countries are the best places of interest for your early days as you can infect most of the population without the governments noticing
regular hand washing and closing down of ship/air ports are always included in scenarios with higher difficulty level
investing in research labs and collaboration around the world is bad news for your win.
better infect and kill scientists when they still have too low of a budget to protect themselves and work even in rich countries and especially those leading the global research efforts
if you are getting out of control in EU/US, you are close to victory
if you have reached Greenland and Madagascar, you have probably also infected almost every other country.
it is also a great idea to start from remote countries like these or reach there early on
bioaerosol and zoonosis are your best friends
olympics and other mass events/festivals not getting canceled is a good sign that the people have chosen money at the cost of deaths
the news cycle is highly depressing and morbidly funny at the same time

The key to our survival in this new covidiocratic world order would be to understand the signs and warnings we have had since many years, accept our past mistakes openly and move on together to fight the new ~~COVID-19~~ SARS virus.

Speaking of keys, I found myself quoting earlier to my friend the popular 3NF ideal, "The key, the whole key and nothing but the key. So help me Codd". It reminded me that as computer scientists we don't shy away from beta substituting God. Or as developers of worlds and as gamers in those worlds, we roleplay as God. But we also see similar grandiose delusions among many public leaders gaming our current world with their unjustified decisions and power grabs. Now is the absolute wrong time to have God-delusions or taking power hungry measures alike those from our political and high positioned covidiots.

The key is probably to be less covidiotic and more human.

(This needs to be normalized way beyond Codd's normal forms.)

MAP III - Types of Types II

Risav Karna — Mon, 13 Apr 2020 20:21:47 GMT

We need to invert the wasteful trend of learning something as timeless as higher maths only as a means to learn an ephemeral or ever-changing programming language or paradigm. Let's aim instead to explain and discuss foundational mathematical structures like types and categories through these programming languages as visual learning aids. This helps us become better developers in any language, including new ones for quantum computing or even natural language-based programming.

Previously on MAP - Types of Types, we looked at the type hierarchy of Scala and worked with a basic 'level 1' definition of types. Now we will level up and refine our previous working definition of types as value spaces with a set of allowed operations. At level 2, we will celebrate Function's tying of the knot with Data and how that adds a new dimension, literally as well, to how we have come to define and understand types. We will need to go back to discussing some basics like the very concept of functions and objects. With this slightly boring and unapologetically pedantic detour, we will demystify some commonly held oversimplifications or misconceptions and proceeds towards the more advanced and interesting concepts being on the same page. For instance we will see how functions and objects together create a fast track towards advanced abstract data types and algebraic structures or how classes and objects are simultaneously quite different and so much more than what we have been teaching our kids.

2. Function weds Data: Domain and Co-domain Spaces

Our last definition of types was mainly concerned with types of data. It was therefore sufficient to think of value spaces, basic syntactics and behavior, understood in the sense of a set of operations on elements of that space. In programming, behavior comes from patterns of actions, which itself comes from operational chunks of code, which in turn act only on either data or other codes.

It's no surprise that chunks of programming instructions and of data have lived together on the same memory space ever since von Neumann architecture was first adopted. This intimacy of data and code is lost on many programmers who do not often need to think of the nitty-gritty of the innards of their machines.

Functions and objects are two major constructs in programming where this intimacy-by-design is directly reflected at the level of program codes. Since they have already 'moved in' together, many languages including Scala makes this an official relationship, which I have cheekily phrased as 'function weds data'.

2.1 Object and Class

Just like every construct in Scala has a type, every instance of a type is an object. Scala does not have Java-style primitive types, which do not have objects.

Whether or not a language follows object oriented programming, an object in a programming language is a data abstraction representing value(s) in memory. An object has a named identifier and may have properties, methods with read/write access to the object's properties and relationships with other objects. Alike functions, objects have mathematical counterparts too. We will explore them in the next two posts in this series.

Templates

A class is famously known as a 'blueprint' or template for instantiating an object. The class defines what properties, methods and relations an object can have and/or from where it inherits them. A class name is also usually the name of the user-defined type of the objects made using the class's constructor, which also usually has the same name as the class.

This is only partially true in Scala but wholly correct in Scala's more popular cousin Java. In Scala, it is problematic to hold on to the idea of class as a sole blueprint for objects.

The Scala specs define objects, classes and traits (Java 8+ interface-like abstractions supporting states, mix-in type restrictions, reference to objects of implementing classes) in terms of templates, which are not actually available for programmers.

Templates, not classes, are the actual templates in Scala.

A template definition may be a class template definition or an object template definition or a trait template definition. A class definition or an object definition may be for a regular class of objects or a singleton object or for a case class or a case singleton object. These template definitions are what actually define the type signature of the resulting objects and the constructors or initial values therein define the initial states of those objects.

This conception of class in Scala is, as with functions, closer to a class in mathematics in the sense that a class definition represents a constrained set of instantiable objects while an object definition is just for one object.

Scala also throws away the concept of static classes that do not need instantiation in favor of singleton objects created via the object template definition. These static classes, values and methods are hard to reconcile with the object oriented paradigm. They do not support inheritance and dynamic dispatch due to polymorphism, which itself is possible due to inheritance. Lastly they are also tough to mock and unit test easily.

Bonus: Classless OOP

However a class is not the only mechanism for defining templates for object instantiation.

Here's an excerpt from MDN regarding how JavaScript has OOP, better than most other languages known for their OOP, without classes. The ES6 classes are actually just syntactic sugar to make it easier for those used to class based OOP.

A prototype-based language?

JavaScript is often described as a prototype-based language — to provide inheritance, objects can have a prototype object, which acts as a template object that it inherits methods and properties from. An object's prototype object may also have a prototype object, which it inherits methods and properties from, and so on.

This is often referred to as a prototype chain, and explains why different objects have properties and methods defined on other objects available to them. Well, to be exact, the properties and methods are defined on the prototype property on the Objects' constructor functions, not the object instances themselves.

2.2 Functions and Methods

A function is a chunk of code you can call by name. The function, its inputs and output can each be objects. In Scala, they always are.

Inputs:

Functions may take values as inputs, possibly operate on these values or create side-effects based on those values or the context, and likely return something in the end. Pure functions however are more well behaved and closer to the functions from mathematics thanks to idempotence, referential transparency, and separation from side effects.

Informally speaking, the space of all possible values that could be accepted as an input is called the domain of a function.

Spaces are just sets with added structures. In our context, spaces may have varying dimensionality and each element has its own previously defined level 1 type.

Outputs:

Every chunk of code in Scala that can be put on a single line returns a value.

These lines, which may be typed as blocks for readability, may be either functions or expressions that compose functions. In either case, an output per expression or per function is guaranteed. Even if an expression or function is handling side effects or running into an exception, the previously mentioned Unit return is guaranteed.

This style of programming with expressions is called expression oriented programming (EOP) and is a part of the functional programming paradigm.

Informally speaking, the space of all possible values that could be produced as an output is called the co-domain of a function.

2.3 Parameter Types and Return Types

Now that we are literally on the same page regarding the lingo and the concepts, let's take these concepts to meet some program codes.

Since every line is either a function or an expression, we can simply take note of what type of thing goes in and what comes out.

if/else-if/else (v1)

This is commonly needed and commonly avoidable expression. Let's break it down.

How could we write a (basic) version of the if-then expression? We know that it would

take a predicate as the first parameter i.e. either a Boolean or something that returns a Boolean
take a function to execute if the above value is true as the second parameter
the predicate and the function could use any number of parameters of Any type

// Naive if-then without advannced tytpes

def ifThen(someBool: Boolean)(func: () => Boolean): Any = {
  !someBool || func()
}

def unitaryFunc() = {
  false
}

ifThen(true) (unitaryFunc)
// res0: Any = false
// same as the invocation below

ifThen(1 == 2) {unitaryFunc}
// res1: Any = true

ifThen(42 == 42) {unitaryFunc}
// res2: Any = false

This is quite a useless if-then in its current form but it gives us a good outline of how if-then works and what we should improve over the course of this series.

As explained in 'Curly Braces and Parentheses' in Viva La Scala explainer collection, the curly braces you see in if statements and other control structures are nothing more than just simple parentheses substituted for readability and multi-line blocks.
The if-then structure or p -> q is logically equivalent to ¬p ∨ q

In terms of improvements, these would be highly necessary:

We will need to support more types of parameters and return values for the function being passed to the if-then function.
We will need to find a way to compose if-then to else-if and else constructs.

Function composition requires that the return type of the function passed as a parameter is the same as the parameter type of the function receiving that function as its parameter. How then do you compose functions that need to support incompatible return and parameter types? We will later explore some solutions to this, including monadic ones.

Here's an an example of how Java's Optional is converted to the monadic version in Scala:

def toScala[A](o: Optional[A]): Option[A] = if (o.isPresent) Some(o.get) else None

match-case

The above example is fairly simple use case for if-else because the Optional container has a nice and simple isPresent method make it easier to test against a method with Boolean return type. However here's how the reverse condition is achieved by Scala:

import java.util.{Optional, OptionalDouble, OptionalInt, OptionalLong}
import java.{lang => jl}

object OptionConverters {

  def toJavaOptionalInt(o: Option[jl.Integer]): OptionalInt = o match {
    case Some(a) => OptionalInt.of(a)
    case _ => OptionalInt.empty
  }
}

This time we are checking against the possible types. match-case is a better fit for checking against types, which in this case are either Some or anything else.

Note that the match construct is also an expression and the curly braces are once again super-powered parentheses which can take multiline values or functions or expressions as arguments.

The input to the match expression is o provided as the first operand in infix style and the case expressions provided together as the later parameter in the curly braces. The return type is determined by the return type of the expression on the right side of the matched case.

The match-case expression is also capable of checking against values and is especially better suited for cases with too many cases.

def getEnglishNames(x: Int): String = x match {
  case 0 => "Zero"
  case 1 => "One"
  case 2 => "Two"
  case 3 => "Three"
  case 4 => "Four"
  case 5 => "Five"
  case 6 => "Six"
  case 7 => "Seven"
  case 8 => "Eight"
  case 9 => "Nine"
  case _ => "Not enough fingers!!"
}

getEnglishNames(6)
// res0: String = Six

getEnglishNames(10)
// res1: String = Not enough fingers!!

How about mixing these two expressions? If you have a use case where you have to check for values while also checking for types, it's best to use both. Here's how Scala converts FiniteDuration to a Java duration:

  def toJava(duration: FiniteDuration): JDuration = {
    if (duration.length == 0) JDuration.ZERO
    else duration.unit match {
      case TimeUnit.NANOSECONDS => JDuration.ofNanos(duration.length)
      case TimeUnit.MICROSECONDS => JDuration.of(duration.length, ChronoUnit.MICROS)
      case TimeUnit.MILLISECONDS => JDuration.ofMillis(duration.length)
      case TimeUnit.SECONDS => JDuration.ofSeconds(duration.length)
      case TimeUnit.MINUTES => JDuration.ofMinutes(duration.length)
      case TimeUnit.HOURS => JDuration.ofHours(duration.length)
      case TimeUnit.DAYS => JDuration.ofDays(duration.length)
    }
  }

These are pre-existing types. Can we do this with custom types? As we discussed, custom types could come from custom objects or classes. You might have also noticed earlier that class and object template definitions may also have a case qualifier in front of it.

sealed abstract class Device
case class IPhoneX(uglyNotchSize: Long) extends Device
case class S10(holeCamOffset: Double) extends Device
case class NexS(popUpCam: Boolean) extends Device
case class Pixel(pixels: Int) extends Device {
  def sorry = "Old School"
}

def howBeautiful(device: Device): String = device match {
  case IPhoneX(_) => "Ugly"
  case S10(offset) if offset == 0.5 => "Symmetric but Meh"
  case S10(_) => "Meh"
  case NexS(false) => "Cool"
  case NexS(true) => "Clunky"
  case px: Pixel => px.sorry
}


howBeautiful(IPhoneX(10))
// res0: String = Ugly
howBeautiful(S10(0.5))
// res1: String = Symmetric but Meh
howBeautiful(S10(0.7))
// res2: String = Meh
howBeautiful(NexS(false))
// res3: String = Cool
howBeautiful(NexS(true))
// res4: String = Clunky
howBeautiful(Pixel(4))
// res5: String = Old School

The sealed keyword tells Scala that all possible subtypes or cases are present within the same file and not dispersed elsewhere in the code.

The if statement on the first S10 case is another way you could mix if with match. In this case, it is called a pattern guard.

Notice that the ordering of the cases matters because the fall through is similar to that found in switch statements in other languages.

Also note that all but the last case are still checking against values albeit of value objects. Only the last case is actually checking against a user defined type.

We now have a good groundwork to look into try-catch and try-success-failure. We will also be looking into for, for-yield and foreach expressions once we have collection types in our arsenal by the end of the next post. That is also where Scala/Mathematics concepts start to get real fun to play with.

Next on 'Types of Types'

To keep the posts from going beyond optimal lengths, we will explore each of the next 'levels' of understanding and defining types in their own posts. Here's a sneak peek.

3. Abstract Data Types: Types with Structure and Behavior

Here we will rediscover the good old data structures mostly used in programming through a more formal, functional and scala-ble lens 🧐. Sum and product types will also pave the way for the more involved algebraic data types in the later section.

3.1 Sum and Product Types

3.1.1 Union or Tagged Union
3.1.2 Tuple and Record

3.2 Set and List

3.2.1 Unordered and Non-duplicated
3.2.2 Ordered and Random Accessible

3.3 Maps

3.4 Stack and Queue

3.5 Graph and Tree

4. Algebraic Data Types and HKTs

Here we will (re-)discover universal algebra, ring-like, group-like, lattice-like, module-like, and algebra-like algebraic structures relevant to advanced functional and reactive programming. We will also demystify higher kinded types and similarly exotic higher order concepts.

5. A Gentle Introduction to Category Theory

Here we will use the Scala concepts and Type Theory topics thus far to go for a deep dive into Bartoz Milewski's excellent course on Category Theory .

Further resources:

2.1 Classes and Objects Specs in Scala: https://scala-lang.org/files/archive/spec/2.13/05-classes-and-objects.html
2.2 Scala source code: https://github.com/scala/scala/blob/2.13.x/src/library/scala/

1.1 D. L. Parnas, John E. Shore, David Weiss: Abstract types defined as classes of variables, https://dl.acm.org/doi/10.1145/942574.807133
1.2 Tony Hoare on his infamous billion dollar mistake: https://www.youtube.com/watch?v=kz7DfbOuvOM
1.3 Official Scala API Docs and resources: https://www.scala-lang.org/
1.4 L.G. Meredith, Pro Scala: Monadic Design Patterns for the Web: http://patryshev.com/books/GregoryMeredith.pdf

Viva la Scala

Risav Karna — Mon, 06 Apr 2020 02:09:41 GMT

Scala is a very scalable language. It actually is true to its portmanteau name. But scalability is a double-edged sword.

With great scalability, comes great extensibility. And confusion. Lots and lots of confusions.

We will have a look at a curated list of Scala's extensibility features. I have also noticed possible sources of confusions about Scala's syntax in my earlier post even though I was writing merely about the basics of Scala's type system.

Fortunately, Scala's syntax can be easily understood with an hour or less of on 'Tour of Scala' on the official site. An even shorter intro for people with Java, Kotlin or TypeScript experience can be found on just the [wikipedia page for Scala](https://en.wikipedia.org/wiki/Scala_(programming_language) of all places.

For further convenience, we will put together an ensemble of simple explainers of Scala's unique principles, features and syntax here. These are mainly going to be from the above links and the further reading links listed in the previous article.

Speaking of staying true to its name, this post itself will be a living document that we shall together keep updated. I hope we can form a one stop shop for picking up essential notes on Scala concepts and syntax details to get started. Explainers on control structures, types and advanced features will also be welcome. However we might need to put those ones elsewhere in the MAP series or the Play with Spring series if they are not already covered.

Viva la Scala!

Unified Type System

As we discussed before in MAP 3: Types of Types, every type in Scala has the top type Any as its supertype and the bottom type Nothing as its subtype.

In Scala, all types inherit from a top-level class Any, whose immediate children are AnyVal (value types, such as Int and Boolean) and AnyRef (reference types, as in Java). This means that [...] in Scala, boxing and unboxing is completely transparent to the user. Scala 2.10+ allows for new value types to be defined by the user.

Uniform Access Principle

This principle originally proposed by the computer scientist Bertrand Meyer, creator of Eiffel language, states that

"All services offered by a module should be available through a uniform notation, which does not betray whether they are implemented through storage or through computation".

In Scala it essentially translates to stipulating that calling an attribute, pre-computed property, method of an object etc all look the same syntactically. The semantics may defer as needed. For instance, even though values are evaluated only once, methods are evaluated each time they are invoked.

Function or method foo() can also be called as just foo; method thread.send(signo) can also be called as just thread send signo; and method foo.toString() can also be called as just foo toString

Dot Free and Point Free

As we saw in the above uniform access section, Scala supports a dot free notation for calling methods and members.

Any method can be used as an infix operator, e.g. "%d apples".format(num) and "%d apples" format num are equivalent.

'Dot free' notation is not to be confused with 'point free' notation or tacit programming style in which function definitions do not identify the arguments (or 'points' in topology terms). This style allows for concise syntax, focus on functions instead of data, easier function composition and currying, among other things. As of the time of writing, using point free style in Scala for anything but trivial pieces of code would not be advised.

//verbose 
List("dot", "free", "vs", "point", "free").foreach(s => println(s))

//dot free
List("dot", "free", "vs", "point", "free") foreach {println(_)}

//point free version:
List("dot", "free", "vs", "point", "free") foreach println

It is not easy for Scala to infer types when you use this parameter free style due to its local type inference system. If you are familiar with Haskell or SML with an essentially global Hindley-Milner type system, these are the kinds of type system super powers you will miss in Scala.

Infix and Symbolic Operators

In the expression +1, + is a prefix and also a unary operator. But when adding two Ints as in 1 + 2, + is an infix operator defined for Scala's integer type. + is a bona-fide method name in Scala defined separately for all Scala types where addition makes sense. This includes BigInt and BigDecimal types too unlike in Java where arithmetic operations for Big* types cannot use the arithmetic operators +, -, /, and *.

As we saw in the dot free and point free example, methods can be treated as infix operators.

In fact, arithmetic operators like + and << are treated just like any other methods, since function names are allowed to consist of sequences of arbitrary symbols (with a few exceptions made for things like parens, brackets and braces that must be handled specially); the only special treatment that such symbol-named methods undergo concerns the handling of precedence.

There are no postfix operators like ++ or -- in Scala. The value types in Scala are immutable by default so you cannot just add these to them without breaking the type system. If you happen to need the mutable var variables for a value type, you can just use += or -= like counter += 1. You can also use *= or /=.

Setters and Getters

This one is for allowing setters and getters.

Class body variables can be transparently implemented as separate getter and setter methods. For trait FooLike { var bar: Int }, an implementation may be

object Foo extends FooLike { 
  private var x = 0; 
  def bar = x; 
  def bar_=(value: Int) { x = value }} } }

The call site will still be able to use a concise foo.bar = 42.

Apply and Update

This one allows for simple creation of collections, case classes or complex objects without having to new up a new instance, usually an empty one and then populate it next. It is also handy for defining default apply and update behaviors that make sense for your class.

Methods apply and update have syntactic short forms. foo() — where foo is a value (singleton object or class instance) — is short for foo.apply(), and foo() = 42 is short for foo.update(42). Similarly, foo(42) is short for foo.apply(42), and foo(4) = 2 is short for foo.update(4, 2). This is used for collection classes and extends to many other cases, such as STM cells.

Curly Braces and Parentheses

The use of curly braces instead of [or after] parentheses is allowed in method calls. This allows pure library implementations of new control structures. [...] Vector.fill(4) { math.random } is the same as Vector.fill(4)(math.random). The curly braces variant allows the expression to span multiple lines. breakable { ... if (...) break() ... } looks as if breakable was a language defined keyword, but really is just a method taking a thunk argument.

Colon-oscopy

This one is for inline list creation and for colon-suffixed methods to be invoked in the order that looks intuitive:

Method names ending in colon : expect the argument on the left-hand-side and the receiver on the right-hand-side. For example, the 4 :: 2 :: Nil is the same as Nil.::(2).::(4), the first form corresponding visually to the result (a list with first element 4 and second element 2).

MAP - III : Types of Types

Risav Karna — Sat, 04 Apr 2020 15:30:50 GMT

We need to invert the wasteful trend of learning something as timeless as higher maths only as a means to learn an ephemeral or ever-changing programming language or paradigm. Let's aim instead to explain and discuss foundational mathematical structures like types and categories through these programming languages as visual learning aids. This helps us become better developers in any language, including new ones for quantum computing or even natural language-based programming.

When talking about types in computer science, we could be talking about two types of types. Types from type theory are closely related to the types from from type systems used in programming languages. Both of these are fun and playing with them illuminates how any language works. It may even help you obtain A3 or perhaps even L3-tier superpowers.

Previously in the Mathematics' (An)architecture and Processes (MAP) series, we have briefly touched upon Curry-Howard-Lambek isomorphism, which is a trifecta of independent realizations about, among other things, how propositions are types i.e. claims that are true or false, formal proofs, logical calculus' constraints and formulae are types too. With Scala's types, and Java 11+'s and TypeScript's later, we will see what it actually means to say something as abstract as 'propositions are types' and why understanding these esoteric and zen sayings can make you a better programmer - in any language, with any kind of type system.

For getting started with types, we will come up with our working definition of types with increasing levels of technical depth. We will also be iteratively refining our definition and intuitions regarding what types are. As for how types look in programming, here are four simple examples:


val obligatory = "Hello World! "  
// type inferred to be String based on its value

val goodbye : String = "Goodbye, cruel world..." 
// type explicitly defined to be a String

val emoji: Char = '☺'
// Unicode characters are present in Char type's value space

val heteroList: List[Any] = List(obligatory + goodbye, true, 42)
// String type allows addition operation 
// The composed type List[Any] allows use of any type in heteroList

This is certainly just a picture and certainly not the whole picture. However we will get closer to types through this kind of programmatic representation of types above.

This is not a pipe. This is just a representation of a pipe.

Almost every programming construct that you can use has a type. It follows that there are many types of types.

1. Common Data Types: Syntactic and Semantic bits

Let's start with the primitive ones. Types of values in data are the easiest to intuitively understand. Here at level 1, we will define type as a pre-defined value space for our data which possibly comes along with a set of allowed operations on the values in that value space.

At level 1, you may find, for instance, a type that does not have any possible operations or one that is simply an alias of another type. These are merely syntactic types. All types have this syntactic aspect in the sense that a type definition ensures the program compiler - and the programmer - that it adheres to the rules and constraints set by the programming language itself, especially regarding what is and isn't allowed to be done with a certain variable or construct.

For instance, if you find a type definition called Money in your program, it probably is a syntactic sugar representing some type of fixed or floating point numbering.

The semantic aspect of a type would be about what it means to have a certain type in the first place. For instance, type semantics could tell you what possible actions are possible on that type or what interactions are allowed with other types.

For a utility type like Money, the operations could be inherited arithmetic methods for addition or subtraction or convenience methods composed of other operations like methods for interest rate calculations.

Any and Nothing

All types in Scala are different derivations of the Any type. Informally, you may call it the mother of all types as it is the top type in Scala's type hierarchy.

Any of Scala is similar to the Object type defined by the class also named Object in Java. In TypeScript, there are two top types any and unknown. any just tells the TypeScript compiler that it should not even bother to do any type checks. So compile time type safety due to aforementioned type syntactics and semantics is effectively disabled for that variable or constant of any type. So related bugs may only be discovered if unexpected operations are performed on that value at runtime - during testing (hopefully) or in production (yikes). unknown type signifies that the actual type is yet unknown but can be narrowed down and pin-pointed based on its runtime values, usually using a language construct called type guards.

Any in Scala has general definitions of equals, hashCode and toString methods. This means that Any type in Scala allows for checking whether anything equals another and what its hashed or string representation is.

All puns intended. Always.

Type theory and category theory, from which we have type systems in programming, also have something called a bottom type. It is called so because it is at the bottom of the type hierarchy i.e. all types have the bottom type as their subtype. Scala's bottom type is called Nothing as there is nothing you can assign to a variable or construct of type Nothing. If a function throws an exception instead of returning any value, it has Nothing for a return type. We will find further uses of this in later posts, especially when we discuss covariance under parametric polymorphism.

AnyVal and AnyRef

In programming, you may be directly using or passing around a value from one memory location to another. These types of values are derived from AnyVal type in Scala, which itself is a descendant derived directly from the Any supertype.

You may also just be referring to where the value is located in your memory space instead of moving those values around. These variables have a reference type, which descends from the AnyRef supertype, which itself descends directly from the Any type.

When you look at the documentation for a type, for example Boolean, you might find methods that box or unbox a value. The scala.Boolean is a value type but you can wrap your boolean value inside a 'box' made of java.lang.Boolean, which is a reference type. You might need to do this boxing to access reference type's methods or, in this case, to ensure interoperability with Java clients of your library. Similarly, you can also 'unbox' a reference type into a value type.

Unit and Boolean

Just like Nothing allows nothing as its value, Unit allows only one and the same value, and Boolean allows only two.

Boolean type is probably the most well known and easily understood of all. It allows its value to be either true or false.

Unit type only allows one value, which is written as (). Therefore it holds no information and is very similar to, but not the same as, void used in C/C++/Java. It also should not be confused with aforementioned Nothing or with Null from the next section. We will return back to this type later when we talk about tuples and product types since Unit can be interpreted as a 0-tuple or a product of no types. We will also see how it is useful when we talk about abstract data types and algebraic data types.

Option and Null

Sometimes a reference is likely to point either to some value or none of it when the value of the instance of that type is requested for use by the program execution. In many languages, such references lead to a null pointer, which Tony Hoare, the pioneering computer scientist who introduced the null pointer concept, has famously called his billion dollar mistake.

The functional and type safe solution to this annoyance would be the the type Option or Optional (in Java, Scala, Kotlin, Swift, SML, OCaml, Rust) or Maybe (in Haskell, Agda, Idris). Depending on the implementation, this type may be monadic.

In Scala, Option is a monadic type and has two subtypes Some or None. We will discuss monads in the section on algebraic data types. For now, going by the above box or container metaphor, if you think that in your variable there maybe some value or optionally none at all then you can put it inside an Option box. To check if there are any valuables in the box, you can simply check the type of the content instead of various 'falsiness' checks or nullness checks on the value of the content itself. The type is is guaranteed to be Some if something is present or None otherwise. There also exists another similar trio called Try, Success and Failure in Scala. They essentially form a more concise syntax and are used when catching exceptions could be needed as opposed to just dealing with presence or absence of values.

We will explore examples of the usage of both of these trios when we later discuss pattern matching based on types and control flow structures like try-catch, match-case or for-yield.

**The Normie Types**

These are the common, necessary but boring types that will become interesting when we start composing them with other types and weaving mind-bending - and hopefully helpful - structures later.

I. Byte, Short and Int

These types have trivially defined value spaces. Available methods for each type will defer as relevant.

val u = 42  // defaults to Int type
val b: Byte = 1 // (-2^7 to 2^7-1] i.e. -128 to 127
val s: Short = 2 // (-2^15 to 2^15-1] i.e. -32,768 to 32,767
val i: Int = 3 // (-2^31 to 2^31-1] i.e. -2,147,483,648 to 2,147,483,647

II. Long, Float and Double

These are pretty much the same as the above numerical types except they allow for bigger ranges and, in case of Float and Double, fractional values with floating point precision.

val l: Long = 1 // (-2^63 to 2^63-1] 
val f: Float = 2.0 // 32-bit IEEE 754 single-precision float
// 1.40129846432481707e-45 to 3.40282346638528860e+38
val d: Double = 3.0 // 64-bit IEEE 754 double-precision float
// 4.94065645841246544e-324d to 1.79769313486231570e+308d

III. Char and String

A Char is a literal from the Unicode charset. A Char can be cast to its corresponding position in the Unicode charset as an Int, like you can in most languages.

scala> val c: Char = 'q'
c: Char = q

scala> val i: Int = c
i: Int = 113

scala> val emoji: Char = '☺'
emoji: Char = ☺

scala> emoji.toInt
res0: Int = 9786

A String is defined as a sequence of Chars. You assign a string using quotes:


val strQ = "Now I am become Death, the destroyer of worlds - J. Robert Oppenheimer"

val listToStr = List("If I'm not back in five minutes", "just wait longer").toString
// output: List(If I'm not back in five minutes, just wait longer)

Scala String relies on java.lang.String but is also augmented by many useful utility methods from the StringOps class.

It is not necessary to learn these methods or those of any Scala type by heart. You should just get more intimate with the docs and always check the docs before writing a routine or standard looking operation on your own. You can likely also discover them using your IDE's autocomplete or inline documentations.

IV. The Big and The Rich

There are StringOps equivalents for the other the other value types called RichBoolean, RichByte, RichChar, RichDouble, RichFloat, RichInt, RichLong and RichShort. These proxies provide additional methods to the above value types. Unlike the case in Java with its primitive and boxed reference types, this pattern allows for keeping the value types lightweight while still providing a rich API and therefore fewer use cases for boxing and unboxing due to performance vs convenience trade-offs.

Besides the Rich* ones, scala.math package provides BigDecimal and BigInt classes. These are for calculations with floating-point numbers of arbitrary precision starting with a default precision approximately that of IEEE 128-bit floating point numbers.

Next on 'Types of Types'

To keep the posts from going beyond optimal lengths, we will explore each of the next 'levels' of understanding and defining types in their own posts. Here's a sneak peek.

2. Function weds Data: Domain and Co-domain Spaces

Here we will introduce functions and methods together with objects and classes. We will see how OOP augments the power of functional programming and how best to avoid the many pitfalls of OOP.

2.1 Object and Class

2.2 Functions and Methods

2.3 Parameter Types and Return Types

3. Abstract Data Types: Types with Structure and Behavior

3.1 Sum and Product Types

3.1.1 Union or Tagged Union
3.1.2 Tuple and Record

3.2 Set and List

3.2.1 Unordered and Non-duplicated
3.2.2 Ordered and Random Accessible

3.3 Maps

3.4 Stack and Queue

3.5 Graph and Tree

4. Algebraic Data Types and HKTs

5. A Gentle Introduction to Category Theory

Here we will use the Scala concepts and Type Theory topics thus far to go for a deep dive into Bartoz Milewski's excellent course on Category Theory .

Beware that category theorists are sometimes insufferable and gave birth to memes like this one:

'A monad is just a monoid in the category of endofunctors, what's the problem?'

or this one below:

Further resources:

D. L. Parnas, John E. Shore, David Weiss: Abstract types defined as classes of variables, https://dl.acm.org/doi/10.1145/942574.807133
Tony Hoare on his infamous billion dollar mistake: https://www.youtube.com/watch?v=kz7DfbOuvOM
Official Scala API Docs and resources: https://www.scala-lang.org/
L.G. Meredith, Pro Scala: Monadic Design Patterns for the Web: http://patryshev.com/books/GregoryMeredith.pdf

Cover image stolen from: https://medium.com/@olxc/the-evolution-of-a-scala-programmer-1b7a709fb71f

anarchitekt

Mt. Kolmogorov: A Complex Climb

1 Kolmogorov Complexity & MDL

2 (Meta) Kolmogorov Complexity of our Apps

3 Schema and API

Mountains & Software: Similar Peaks

Builders & Generators

Prisma & The Tree

ts-morph & Morphisms

More Slide-y Thinking

A Minimal Requirement Analysis

Bounds, aka How To Get Rect

A Bit of Meta

A Short Keynote: Bored with Navigation Rodents

Getting our S#@% Together

The State of Slidey Things

A No Cap Recap

Slide-y Thinking

Initial Conceptions

A Minimal Version

A Minimal Requirement Analysis

A Stack Attack

Let's Find Ourselves Our Stack Inventory

Universal Stack Languages: ABCs to XYZs

Agda

Bytecode (Graal VM/JVM/LLVM)

C/C++/Carbon

Datalog

Eiffel

Fortran

Gleam (Beam VM)

HaXe

Idris 2

Julia

sKotlin (ScalaJS + Kotlin UI monorepos)

Lean

Miranda

Nim

OCaml

Prolog

Q

Racket (Racket + Rust Backend monorepos)

Svelte 5

Typescript

Unison

XSLT

YAML

Zig

Time - III [Draft]

The Crisis in Infinite Quantum Futures

Quantum Physics and Quantum Field Theories

Vedic Mathematics and Astrophysics

The Being and The Becoming

Categorical Dual Theories of Time

Time on 1, 2, 3, or more planes

The Higher Dimensions of Time and The n-Onions

MAP IV: Mathematics as a Language - I

Is Math Invented or Discovered?

One of the Rings

Digression into Programming

Let's Play the Peano

Take me to Church

Invented or Discovered?

WW II and WW Covid: Diffs and Parallels

Reintroducing nep.work

Requirement re-engineering and business workflow studies

Payments: Simplified so much, it is mathematically impossible to further simplify

Digital Contracts and Exchanges

Omni-channel

Platform

Sales and Delivery

Work Exchange

prox-e

docs

sync.sh

tp

channels

nginx

protocols

hi

The State of `Slidey` Things

Why Git repository providers needn't rename `master`

Removing `master`