Lecture 42 - Chapter 3: Transforming Databases

John Baez

To prepare yourself for the next phase of the course, here are some puzzles illustrating the ideas in Lecture 41.

Let's transform a database by composing it with a functor! We already did this in Lecture 41, and the answer should have made a lot of sense. But this time the answer will be strange... leading us to the next phase of the course.

Puzzle 122. Let \(\mathcal{C}\) be the free category on this graph:

Let's use this as a database schema. Draw an example of a database built using this schema. Mathematically it's a functor \(F: \mathcal{C} \to \mathbf{Set}\). But what does this mean in concrete terms? You can draw it as a table, like you would see in a spreadsheet.

Puzzle 123. Let \(\mathcal{D}\) be the free category on this graph:

Again this can be used as a database schema. What does a database built using this schema amount to, in concrete terms?

Puzzle 124. Show there exists a unique functor \(G : \mathcal{D} \to \mathcal{C}\) with

$$ G(\textrm{FriendsOf}) = G(\textrm{FriendsOf}^\prime) = \textrm{FriendsOf} .$$ What does this functor do, in concrete terms?

Puzzle 125. Take the database \(F: \mathcal{C} \to \mathbf{Set}\) that you constructed in Puzzle 122 and form the database \(F \circ G : \mathcal{D} \to \mathbf{Set}\). What do you get?

Don't be shocked if the answer to this last puzzle is sort of weird. You are taking a database about \(\mathrm{People}\) and their friends and turning it into a database about \(\mathrm{Germans}\) and \(\mathrm{Italians}\)... without having any information at hand about which people are Germans and which are Italians! This is bound to give something strange.

A more reasonable thing would be this: we start out with a database about \(\mathrm{Germans}\) and \(\mathrm{Italians}\), forget who is German and who is Italian, and get a database about \(\mathrm{People}\).

This means starting with a functor from \(\mathcal{D}\) to \(\mathbf{Set}\) and getting a functor from \(\mathcal{C}\) to \(\mathbf{Set}\). And this is a bit more tricky than what we just did, namely going the other way. We need to do something called a 'Kan extension'.

Kan extensions are great. In his book Categories for the Working Mathematician, Saunders Mac Lane wrote:

The notion of Kan extensions subsumes all the other fundamental concepts of category theory.

So, I'll have to explain Kan extensions!

But for this I'll need to explain two other concepts: 'natural transformations' and 'adjoint functors'. These are two of the most important concepts in category theory... and they're great fun.

To read other lectures go here.

Owen Biesel

Puzzle 124: I don't think there's just one functor from D to C. There is just one graph homomorphism, though. (Feel free to delete this paragraph after reading it.)

I'm excited about Kan extensions! I've come across them before but I've never been able to make a gut connection with them on the level of adjoints and limits, which they apparently generalize.

In the discussion after Lecture 38, we were looking at some slides from a talk by David Spivak about databases, and he mentioned the Grothendieck construction that converts an instance \(I:\mathcal{D}\to \mathbf{Set}\) to another category \(\mathrm{Gr}(I)\) equipped with a functor \(\mathrm{Gr}(I)\to \mathcal{D}\). It would be easy to compose this with the functor \(\mathcal{D}\to\mathcal{C}\) and get at least what he calls a "semi-instance" \(\mathrm{Gr}(I) \to \mathcal{C}\). Does this semi-instance correspond to the actual instance \(\mathcal{C}\to \mathbf{Set}\) we'll get with Kan extensions later?

Keith E. Peterson

Puzzle 122. Let C be the free category on this graph:

Let's use this as a database schema. Draw an example of a database built using this schema. Mathematically it's a functor F:C→Set. But what does this mean in concrete terms? You can draw it as a table, like you would see in a spreadsheet.

Giving an instance to the above free-category \(C\), just produces something that looks like a 2 column table,

\[ \begin{array}{c|c} \text{People} & \mathrm{FriendOf} \\ \hline Alice & Bob \\ Bob & Alice \\ \vdots & \vdots \\ Stan & Tyler \\ Tyler & Stan. \\ \vdots & \vdots \end{array} \]

Pierre Prado

Pz123 - Increasing dimensional space...i guess..related with the "strange" brains´s 11 dimensions. https://www.frontiersin.org/articles/10.3389/fncom.2017.00048/full Pz124-Dimensions reduction...related to the principal component analysis ...I guess

Keith E. Peterson

Puzzle 123. Let \(\mathcal{D}\) be the free category on this graph:

Again this can be used as a database schema. What does a database built using this schema amount to, in concrete terms?

If I did it correctly, it shoud be two tables,

\[ \begin{array}{c|c} \text{Germans} & \mathrm{FriendOf} \\ \hline Adele & Sara\\ Bertram & Antonio \\ \vdots & \vdots \\ Siegmund & Teresa \\ Tyler & Bruno, \\ \vdots & \vdots \end{array} \quad \begin{array}{c|c} \text{Italians} & \mathrm{FriendOf}' \\ \hline Antonio & Bertram \\ Bruno & Bertram \\ \vdots & \vdots \\ Sara & Adele \\ Teresa & Siegmund. \\ \vdots & \vdots \end{array}
\]

Keith E. Peterson

Puzzle 124. Show there is exactly one functor \(G : \mathcal{D} \to \mathcal{C}\). What does this functor do, in concrete terms? Hint: on objects it forgets the difference between Germans and Italians... but what does it do to morphisms?

As you stated, \(G\) maps \(\text{Italians}\) and \(\text{Germans}\) to the same place, since in \(\mathcal{C}\) there is only one place to go, \(\text{People}\), we must have,

\[ G(\text{Italians}) = G(\text{Germans}) = \text{People}, \]

and likewise, for the same reasoning, \[ G(\text{FriendOf}_\mathcal{D}) = G(\text{FriendOf}'_\mathcal{D}) = \text{FriendOf}_\mathcal{C}, \]

that follows the functorial constraints, \[ G(\text{FriendOf}_\mathcal{D} \circ \text{FriendOf}'_\mathcal{D}) \\ = G(\text{FriendOf}_\mathcal{D}) \circ G(\text{FriendOf}'_\mathcal{D}) \\ = \text{FriendOf}_\mathcal{C} \circ \text{FriendOf}_\mathcal{C} \]

and, \[ G(\text{FriendOf}'_\mathcal{D} \circ \text{FriendOf}_\mathcal{D}) \\ = G(\text{FriendOf}'_\mathcal{D}) \circ G(\text{FriendOf}_\mathcal{D}) \\ = \text{FriendOf}_\mathcal{C} \circ \text{FriendOf}_\mathcal{C}. \]

Anindya Bhattacharyya

Could we not send one or both FriendOf in D to the identity map in C?

Keith E. Peterson

You can, but everything will get deleted unless there are Germans who are Italians (and vise versa), and they are friends with themselves (and friends' with themselves).

Owen Biesel

Anindya wrote:

could we not send one or both FriendOf in D to the identity map in C?

Anindya, I agree, one or both of FriendOf in D could be sent to the identity map in C, or FriendOf^(-1), or FriendOf composed with itself five times, etc.

Keith wrote:

You can, but everything will get deleted unless there are Germans who are Italians (and vise versa), and they are friends with themselves (and friends' with themselves).

It sounds like you're thinking of what would happen if you took an instance of the D-schema and tried to interpret it as a C-schema using that functor \(D\to C\). We haven't made the concept precise yet of extending a functor \(D\to\mathbf{Set}\) to a functor \(C\to\mathbf{Set}\) using a functor \(D\to C\), so I don't know what would happen then. It sounds like you think we'd select the entries in the old database that make sense in the new database schema, and throw away the rest?

Keith E. Peterson

You may have a point.

So then \(G\) is more like a mapping between the underlying graphs of the categories than a mapping between databases themselves.

Though, \(G\) can be used to make a database, but only after we compose it with some \(F: \mathcal{C} \to \mathbf{Set}\) to get the database instance, \((F \circ G): \mathcal{D} \to \mathbf{Set}\).

Owen Biesel

Yes, that all sounds right to me. The funny thing, though, is that while you get valid database instances by composing the way you describe, they don't seem to be the "right" things to do. If you combine your answers to Puzzles 122 and 124 to get an answer to Puzzle 125, you get something like this:

\[ \begin{array}{c|c} \text{Germans} & \mathrm{FriendOf} \\ \hline Alice_0 & Bob_1 \\ Bob_0 & Alice_1 \\ \vdots & \vdots \\ Stan_0 & Tyler_1 \\ Tyler_0 & Stan_1. \\ \vdots & \vdots \end{array} \quad \begin{array}{c|c} \text{Italians} & \mathrm{FriendOf}' \\ \hline Alice_1 & Bob_0 \\ Bob_1 & Alice_0 \\ \vdots & \vdots \\ Stan_1 & Tyler_0 \\ Tyler_1 & Stan_0. \\ \vdots & \vdots \end{array} \]

Which seems to duplicate every person into both a German and an Italian version, but make all the German versions' friends the Italian versions and vice versa. Weird!

MatthewDoty

Here is how I would write the answer to Puzzle 123 in SQL:

CREATE TABLE Germans (
  German_name VARCHAR(256) PRIMARY KEY
);

CREATE TABLE Italians (
  Italian_name VARCHAR(255) PRIMARY KEY
);

CREATE TABLE German_Italian_friend_of_join (
  German_name VARCHAR(255) REFERENCES Germans(German_name) NOT NULL,
  friend_of VARCHAR(255) REFERENCES Italians(Italian_name) NOT NULL,
  Primary Key (German_name, friend_of)
);

CREATE TABLE Italian_German_friend_of_prime_join (
  Italian_name VARCHAR(255) REFERENCES Italians(Italian_name) NOT NULL,
  friend_of_prime VARCHAR(255) REFERENCES Germans(German_name) NOT NULL,
  Primary Key (Italian_name, friend_of_prime)
);

Is Ryan Wisnesky around? I am extremely curious what SQL the FQL system should compile to in this case.

Keith E. Peterson

Owen Biesel , that doesn't seem right actually.

You seem to be taking a scheme, which is a functor \(F: \mathcal{C} \to \mathbf{Set}\), and then applying it backwards along a functor \(G\) mapping between the categories \(\mathcal{D} \to \mathcal{C}\).

If intuition serves, \(G\) should mapping both tables into the single table. That is to say, \(G\) is an inclusion of sorts.

Therefor, the instance \((F \circ G): \mathcal{D} \to \mathbf{Set}\) should be one big table,

\[ \begin{array}{c|c} \text{People} & \mathrm{FriendOf} \\ \hline Alice & Bob \\ Bob & Alice \\ \vdots & \vdots \\ Stan & Tyler \\ Tyler & Stan. \\ \vdots & \vdots \\ Adele & Sara\\ Bertram & Antonio \\ \vdots & \vdots \\ Siegmund & Teresa \\ \vdots & \vdots \\ Antonio & Bertram \\ Bruno & Bertram \\ \vdots & \vdots \\ Sara & Adele \\ Teresa & Siegmund. \\ \vdots & \vdots \end{array} \]

Edit: Note that \(Tyler\) doesn't show up twice and that his Italian friend gets mapped to someone else as a friend.

Ryan Wisnesky

The connection between FQL and SQL (or rather, SPCU+keygen relational algebra) is described in 'Relational Foundations for Functorial Data Migration': http://arxiv.org/abs/1212.5303. Given a directed multi graph, the simplest strategy is to have one reflexive two column table for each node and one two column table representing a total function for each edge. FQL emits this SQL:

CREATE TABLE Germans(c0 VARCHAR(255) PRIMARY KEY, c1 VARCHAR(255)); CREATE TABLE Italians(c0 VARCHAR(255) PRIMARY KEY, c1 VARCHAR(255)); CREATE TABLE friendOf(c0 VARCHAR(255) PRIMARY KEY, c1 VARCHAR(255)); CREATE TABLE friendOf2(c0 VARCHAR(255) PRIMARY KEY, c1 VARCHAR(255)); But Matthew is correct that you can (and probably should) add foreign key indicators as well (these are just tricky to do in SQL when there are cycles). If instead of SQL we permit ourselves first-order logic (or rather, so-called 'embedded dependencies'), we can completely axiomatize the instances we want (also taken from FQL):

forall v0 v1, Germans(v0, v1) -> v0 = v1 forall v2 v3, Italians(v2, v3) -> v2 = v3 forall v4 v5, friendOf(v4, v5) -> Italians(v5, v5) /\ Germans(v4, v4) forall v4 v5 v6, friendOf(v4, v6) /\ friendOf(v4, v5) -> v5 = v6 forall v6, Germans(v6, v6) -> exists v7, friendOf(v6, v7) forall v8 v9, friendOf2(v8, v9) -> Italians(v8, v8) /\ Germans(v9, v9) forall v8 v9 v10, friendOf2(v8, v9) /\ friendOf2(v8, v10) -> v9 = v10 forall v10, Italians(v10, v10) -> exists v11, friendOf2(v10, v11) Of course, you could instead 'pre join' all the binary tables together (often done in papers to save space), or use unary tables for the nodes, and other techniques.

MatthewDoty

What dialect of SQL do you target exactly?

In MySQL / SQL Server / Oracle / MS Access / PostGresQL / H2 you can write:

CREATE TABLE Germans(c0 VARCHAR(255) PRIMARY KEY, c1 VARCHAR(255));
ALTER TABLE Germans ADD CONSTRAINT CHK_GermansRefl CHECK (c0 = c1);

CREATE TABLE Italians(c0 VARCHAR(255) PRIMARY KEY, c1 VARCHAR(255));
ALTER TABLE Italians ADD CONSTRAINT CHK_ItaliansRefl CHECK (c0 = c1);

CREATE TABLE friendOf(c0 VARCHAR(255) PRIMARY KEY, c1 VARCHAR(255));
CREATE TABLE friendOf2(c0 VARCHAR(255) PRIMARY KEY, c1 VARCHAR(255));

This would cover the axioms:

$$ \begin{align} \forall v_0, v_1.\; \textsf{Germans}(v_0,v_1) & \to v_0 = v_1 \\ \forall v_2, v_3.\; \textsf{Italians}(v_2,v_3) & \to v_2 = v_3 \end{align} $$ You might want to have an optimization to elide all these duplicate columns. I suspect it can add significant overhead.

(As I look at your schema generation code, I am seeing that this is easier said than done.)

Moreover, it might be a good idea to introduce NOT NULL constraints everywhere to match the FOL semantics you propose.

Ryan Wisnesky

Building an 'industrial strength' implementation of AQL on SQL is an interesting challenge, which may possibly be the next step for the AQL project. It's tricky not only because of optimization issues like you identify in your post, but because SQL doesn't perfectly match AQL's semantics (e.g., SQL uses three valued logic). If you'd like to get involved, please let me know!

MatthewDoty

If you'd like to get involved, please let me know!

Right now I want to focus on getting up to speed with Category Theory.

Do you program Haskell? A couple months ago we had a discussion about compiling finally tagless languages into the categories presented in Conal Elliot's Compiling to Categories (2017).

I don't know how to do this with AQL, but if you want to guide me I would be happy to learn.

Ryan Wisnesky

Both the category of schemas (category of categories), and for each schema S, the category of S-databases, are cartesian closed. That means that schemas and S-databases are some of the alternative semantics you can give to simply typed lambda calculi types and terms! The details are described here: https://categoricaldata.net/fql/haskell.pdf

MatthewDoty

I've read that presentation you've linked.

The likeness of the simply typed lambda calculus to Haskell is a bit superficial, don't you think? The simply typed lambda calculus is strongly normalizing, and Haskell is not. If you haven't read the Conal Elliot paper, that is actually good news for you. This is because CCCs with fixed points are still a research topic for Conal's formalism AFAIK.

There are more differences in the type systems. I am sure you know Haskell compiles to System FC. In terms of Barendregt's Lambda cube this means that Haskell is more akin to \(\lambda 2\). As far as I can tell your semantics are like \(\lambda \to\) and are therefore simpler.

Compiling to Conal Elliot's categories seems like an excellent idea for FQL, since the two are quite similar and appear to have the same expressive power. You could then be agnostic regarding implementation just like Conal Elliot is in his paper.

I would strongly prefer to not write out ADTs for the AST of FQL along with a parser. A much more direct way to get started is to use a finally tagless DSL á la Oleg Kiselyov. If you go to that thread I previously linked, I work out with Miguel how to compile one to Conal Elliot's categories.

But I am afraid I need your help in writing the finally tagless DSL, because I can't figure out how to start from your PDF.

Ryan Wisnesky

Hi Matthew, you're right that the connection is between AQL and STLC, not AQL and Haskell. My presentation was about giving a semantics to the STLC using AQL; going the other way, to give a semantics to AQL using STLC/Haskell, is trickier. Let's restrict ourselves just to interpreting AQL schemas as STLC types and AQL mappings as STLC terms (the same story repeats at the level of instances). It's easy to interpret products, sums, etc of AQL schemas as products, sums, etc of STLC types. There is a problem, however, in interpreting finitely presented categories as STLC types - in particular, we need to extend the STLC with an infinite family of type constants, one for each finitely presented category (see http://categoricaldata.net/fql/fql_def.pdf), because you can't build all finitely presented categories using 0, 1, +, *, ^ alone. One alternative is to use a dependent type theory, where each finitely presented category [type constant] can be represented as a dependent type (the approach taken here: https://arxiv.org/pdf/1401.7694.pdf ). Indeed, I tried to prototype AQL in Haskell but it never really worked out due to the need for dependent types (note that http://okmij.org/ftp/tagless-final/course/lecture.pdf is about representing the STLC in Haskell, not about representing the STLC + an infinite family of constants or the STLC + dependent types in Haskell). Of course, you can still do a 'deep embedding' of AQL into Haskell/Java/etc, which is how AQL is actually implemented.

Interestingly, we found that in practice, it's actually pretty rare for people to want to use 0, 1, +, *, ^ to describe their schemas - instead, people just seem to write down specific finitely presented categories and go from there. For that reason, the only operation on schemas provided by newer versions of AQL is the colimit operation, which is very dependently typed (i.e., a colimit schema [type] is defined in terms of schema mappings [terms]).

MatthewDoty

... in particular, we need to extend the STLC with an infinite family of type constants, one for each finitely presented category (see http://categoricaldata.net/fql/fql_def.pdf), because you can't build all finitely presented categories using 0, 1, +, *, ^ alone.

I don't really understand. Can't you do this with type families using -XTypeFamilies or is there some limitation I am not understanding? You might be able to get away with just a GADT.

...note that http://okmij.org/ftp/tagless-final/course/lecture.pdf is about representing the STLC in Haskell...

This isn't exactly right. If you read that paper carefully you see Oleg presents the Linear Lambda Calculus rather than the Simply Typed Lambda Calculus.

That paper talks about other final tagless DSLs too. My favorite DSL like this is Servant. Mestanogullari has a paper on this from 2015 if you haven't encountered it.

From what I can tell, Servant the gold standard for type driven development web development. I can talk more about this in the Categories for the Working Hacker discussion group if you are interested.

Of course, you can still do a 'deep embedding' of AQL into Haskell/Java/etc, which is how AQL is actually implemented.

I am reluctant to embrace this.

Java in particular is a huge pain. Manipulating AST typically involves what the Gang of Four calls the "Visitor pattern". This is really hard to extend, debug and understand. Alex Miller has a nice discussion of this problem over on the IBM developerWorks blog (2011).

You would presumably also have to rewrite type inference and unification.

I assume you guys are doing generative testing in lieu of real formal verification. Is there a good Java library for that?

Interestingly, we found that in practice, it's actually pretty rare for people to want to use 0, 1, +, *, ^ to describe their schemas - instead, people just seem to write down specific finitely presented categories and go from there.

That matches my intuition about people.

For that reason, the only operation on schemas provided by newer versions of AQL is the colimit operation, which is very dependently typed (i.e., a colimit schema [type] is defined in terms of schema mappings [terms]).

Why do schema mappings need to be terms?

Couldn't they be types like how Servant has routes as types?

(We should probably take this discussion over to the Applied Category Theory Discussion Groups section).

John Baez

Owen wrote:

Puzzle 124: I don't think there's just one functor from \(\mathcal{D}\) to \(\mathcal{C}\). There is just one graph homomorphism, though.

You're right!

I'll fix the puzzle by indicating the functor I'm actually interested in: the one that maps the morphisms \(\textrm{FriendOf}\) and \(\textrm{FriendOf}^\prime\) in \(\mathcal{D}\) to the morphism \(\textrm{FriendOf}\) in \(\mathcal{C}\).

(Feel free to delete this paragraph after reading it.)

I'll leave it in! I don't want to convey the false impression that I'm infallible. I do, however, want to fix up my "lectures" so that students don't get confused by mistakes.

Anindya wrote:

Could we not send one or both FriendOf in \(\mathcal{D}\) to the identity map in \(\mathcal{C}\)?

Yes, that's one of infinitely many other functors from \(\mathcal{D}\) to \(\mathcal{C}\) I hadn't noticed when first stating Puzzle 124.

John Baez

Okay, let me discuss some answers to the puzzles!

Puzzle 122. Let C be the free category on this graph:

Let's use this as a database schema. Draw an example of a database built using this schema. Mathematically it's a functor \(F:\mathcal{C} \to \textrm{Set}\). But what does this mean in concrete terms? You can draw it as a table, like you would see in a spreadsheet.

Keith wrote:

\[ \begin{array}{c|c} \text{People} & \mathrm{FriendOf} \\ \hline Alice & Bob \\ Bob & Alice \\ \vdots & \vdots \\ Stan & Tyler \\ Tyler & Stan \\ \vdots & \vdots \end{array} \]

Yes! That's right. However, you've done an example with symmetries that aren't required. We could have Alice be a friend of Bob even though Bob is not a friend of Alice, etc.

Just like lots of people "friend" Tom Cruise on Facebook even though he doesn't friend them back!

Puzzle 123. Let \(\mathcal{D}\) be the free category on this graph:

Again this can be used as a database schema. What does a database built using this schema amount to, in concrete terms?

Keith wrote:

If I did it correctly, it should be two tables:

\[ \begin{array}{c|c} \text{Germans} & \mathrm{FriendOf} \\ \hline Adele & Sara\\ Bertram & Antonio \\ \vdots & \vdots \\ Siegmund & Teresa \\ Tyler & Bruno \\ \vdots & \vdots \end{array} \quad \begin{array}{c|c} \text{Italians} & \mathrm{FriendOf}^\prime \\ \hline Antonio & Bertram \\ Bruno & Bertram \\ \vdots & \vdots \\ Sara & Adele \\ Teresa & Siegmund \\ \vdots & \vdots \end{array}
\]

Yes, that's right! Nice names, too!

Owen wrote to Keith:

If you combine your answers to Puzzles 122 and 124 to get an answer to Puzzle 125, you get something like this:

\[ \begin{array}{c|c} \text{Germans} & \mathrm{FriendOf} \\ \hline Alice_0 & Bob_1 \\ Bob_0 & Alice_1 \\ \vdots & \vdots \\ Stan_0 & Tyler_1 \\ Tyler_0 & Stan_1 \\ \vdots & \vdots \end{array} \quad \begin{array}{c|c} \text{Italians} & \mathrm{FriendOf}' \\ \hline Alice_1 & Bob_0 \\ Bob_1 & Alice_0 \\ \vdots & \vdots \\ Stan_1 & Tyler_0 \\ Tyler_1 & Stan_0 \\ \vdots & \vdots \end{array} \]

Which seems to duplicate every person into both a German and an Italian version, but make all the German versions' friends the Italian versions and vice versa. Weird!

Right! This is the correct answer to Puzzle 125.

You say the answer is weird... and you're right! When I posed the puzzle, I said:

Don't be shocked if the answer to this last puzzle is sort of weird. You are taking a database about \(\textrm{People}\) and their friends and turning it into a database about \(\textrm{Germans}\) and \(\textrm{Italians}\)... without having any information at hand about which people are Germans and which are Italians! This is bound to give something strange.

Since you're not saying whether Alice is German or Italian, and you're forcing the math to make a decision about this, without flipping a coin to break the symmetry, all the math can do is posit both a German and an Italian version of Alice! And similarly with Bob, and all the rest.

Next, you're demanding that Germans have only Italian friends and Italians have only German friends, and you're saying Alice is a friend of Bob. So, all the math can do is guess that the German Alice has the Italian Bob as a friend, and the Italian Alice has the German Bob as a friend.

It's wacky, but it's about the best the poor math can do, given what you're demanding of it.

John Baez

Owen wrote:

I'm excited about Kan extensions! I've come across them before but I've never been able to make a gut connection with them on the level of adjoints and limits, which they apparently generalize.

They're a lot less complicated than most people manage to make them seem. They're a special case of adjoint functors.

Here's an ultra-terse explanation which may only make sense to me; I'll expand it later in this class. We've seen here that for any categories \(\mathcal{C}\) and \(\mathcal{D}\), composing with \(G : \mathcal{D} \to \mathcal{C}\) turns functors \(\mathcal{C} \to \mathbf{Set}\) into functors \(\mathcal{D} \to \mathbf{Set}\). We'd like to 'reverse' this process. Here's how. This process is actually a functor - so to 'reverse' it, we take its left or right adjoint! Its left adjoint is called 'left Kan extension', and its right adjoint is called 'right Kan extension'.

In a bit more detail:

There's a category of functors \(\mathcal{C} \to \mathbf{Set}\), called \(\mathbf{Set}^\mathcal{C}\) . There's a category of functors \(\mathcal{D} \to \mathbf{Set}\), called \(\mathbf{Set}^\mathcal{D}\) . Composing with \(G : \mathcal{D} \to \mathcal{C}\) gives a functor from the first of these categories to the second:

$$ \textrm{composition with } G : \mathbf{Set}^\mathcal{C} \to \mathbf{Set}^\mathcal{D} $$ This functor has left and right adjoints, which are called left Kan extension and right Kan extension.

Christopher Upshaw

I actually understood that! Thanks for shrinking it down!

Owen Biesel

Me too! I'll try to absorb the mantra "A left (or right) Kan extension (along \(F\)) is a left (or right) adjoint to precomposition (by \(F\))."

John Baez

Let me say a bit more about this puzzle:

Puzzle 122. Let \(\mathcal{C}\) be the free category on this graph:

Let's use this as a database schema. Draw an example of a database built using this schema. Mathematically it's a functor \(F:\mathcal{C} \to \textrm{Set}\). But what does this mean in concrete terms? You can draw it as a table, like you would see in a spreadsheet.

Keith gave this answer:

\[ \begin{array}{c|c} \text{People} & \mathrm{FriendOf} \\ \hline Alice & Bob \\ Bob & Alice \\ \vdots & \vdots \\ Stan & Tyler \\ Tyler & Stan \\ \vdots & \vdots \end{array} \]

This table has a symmetry that's not required of every functor \(F : \mathcal{C} \to \mathbf{Set}\) Another perfectly fine answer would have been this:

\[ \begin{array}{c|c} \text{People} & \mathrm{FriendOf} \\ \hline Alice & Bob \\ Bob & Bob \\ \vdots & \vdots \\ Stan & Bob \\ Tyler & Alice \\ \vdots & \vdots \end{array} \]

Puzzle. Can we change the category \(\mathcal{C}\) to a category \(\mathcal{C}^\prime\) so that functors \(F : \mathcal{C}^\prime \to \mathbf{Set}\) are just databases of the sort Keith drew, with the kind of symmetry that his table has?

John Baez

Owen wrote:

In the discussion after Lecture 38, we were looking at some slides from a talk by David Spivak about databases, and he mentioned the Grothendieck construction that converts an instance \(I:\mathcal{D}\to \mathbf{Set}\) to another category \(\mathrm{Gr}(I)\) equipped with a functor \(\mathrm{Gr}(I)\to \mathcal{D}\). It would be easy to compose this with the functor \(\mathcal{D}\to\mathcal{C}\) and get at least what he calls a "semi-instance" \(\mathrm{Gr}(I) \to \mathcal{C}\). Does this semi-instance correspond to the actual instance \(\mathcal{C}\to \mathbf{Set}\) we'll get with Kan extensions later?

I don't think so. A functor \(\mathrm{Gr}(I) \to \mathcal{C}\) is pretty darn different from a functor \(\mathcal{C}\to \mathbf{Set}\) .

An object of \(\mathrm{Gr}(I)\) is an object \(d\) of \(\mathcal{D}\) together with an element of the set \(I(d)\). So, a functor \(\mathrm{Gr}(I) \to \mathcal{C}\) takes an object \(d\) of \(\mathcal{D}\) together with an element of \(I(d)\) and spits out an object in \(\mathcal{C}\). On the other hand, a functor \(\mathcal{C}\to \mathbf{Set}\) takes an object of \(\mathcal{C}\) and spits out a set. These are pretty darn different things to do!

Owen Biesel

John wrote:

An object of \(\mathrm{Gr}(I)\) is an object \(d\) of \(\mathcal{D}\) together with an element of the set \(I(d)\). So, a functor \(\mathrm{Gr}(I) \to \mathcal{C}\) takes an object \(d\) of \(\mathcal{D}\) together with an element of \(I(d)\) and spits out an object in \(\mathcal{C}\). On the other hand, a functor \(\mathcal{C}\to \mathbf{Set}\) takes an object of \(\mathcal{C}\) and spits out a set. These are pretty darn different things to do!

That's true, but there's a way to at least make the types match up: We can ask if the following square commutes (for a suitable choice of left or right Kan extension):

$$\begin{array}{ccccc} & & \text{Grothendieck construction} & & \\ \text{Left/right} & \text{Functors }\mathcal{D}\to\mathbf{Set} & \to & \text{Functors to }\mathcal{D} & \\ \text{Kan extension along } & \downarrow & & \downarrow & \text{Compose with }\\ \mathcal{D}\to\mathcal{C} & \text{Functors }\mathcal{C}\to\mathbf{Set} & \to & \text{Functors to }\mathcal{C} & \mathcal{D}\to\mathcal{C} \\ & & \text{Grothendieck construction} & & \end{array}$$ It's bizarre to me that (certain types of) functors to \(\mathcal{C}\) correspond to functors \(\mathcal{C}\to\mathbf{Set}\). The former can be easily pushed forward along functors \(\mathcal{C}\to\mathcal{C}'\); the latter can be easily pulled back along functors \(\mathcal{C}'\to\mathcal{C}\). Usually I expect at most one of these to be easy!

Ryan Wisnesky

Hi Matthew, I'm happy to move the discussion to another thread - if you start one, let me know.

Just to answer your question, the STLC forms a cartesian closed category, where the objects are types and the morphisms are terms. Similarly, the category of categories forms a cartesian closed category, where the objects are categories and the morphisms are functors. Similarly, AQL schemas form a cartesian closed category, where the objects are schemas (finitely presented categories) and the morphisms are schema mappings (morphisms of finitely presented categories). That is why it is natural to translate STLC types <-> categories and STLC terms <-> functors.

Under the above analogy, a concrete example of a finitely presented category that I'm not sure how to represent as a Haskell type/type family/etc but I know how to represent as a Coq type, is the presentation with two objects, A and B, two generating arrows f : A -> B and g : B -> A, and the equation f.g = id.

MatthewDoty

Hey Ryan,

I started the discussion here: https://forum.azimuthproject.org/discussion/2242/fql-discussion

John Baez

I use this sign when I'm talking about stuff that we're not ready for, given what the course has covered so far. If students see a sign like this, it means "Don't worry that you don't understand this stuff: you're not really supposed to. You can just skip it."

Owen wrote:

It's bizarre to me that (certain types of) functors to \(\mathcal{C}\) correspond to functors \(\mathcal{C}\to\mathbf{Set}\).

That bizarreness is exactly why the Grothendieck construction is important. However, it's just a souped-up version of a construction you've probably seen, and putting it in context makes it seem less strange.

Here's the basic idea: there are two ways to think of a subset \(S\) of a set \(X\). One the 'inclusion' function

$$ i_S: S \to X $$ namely the 1-1 function sending any element of \(S\) to itself viewed as an element of \(X\). The other is the 'characteristic function' or 'indicator function'

$$ \chi_S : X \to \lbrace 0,1 \rbrace $$ with \(\chi_S(x) = 1\) if \(x \in S\) and \(\chi_S(x) = 0 \) otherwise.

So, we can think of subsets of \(X\) either as special maps into \(X\), or maps out of \(X\)... into a special thing!

Both viewpoints are important. The first viewpoint lets us 'push forward' a subset of \(X\) along a map \(X \to Y\) and get a subset of \(Y\). The second lets us 'pull back' a subset of \(X\) along a map \(Y \to X\) and get a subset of \(Y\). The first method is called the 'image' (or in this course 'direct image'), while the second is called the 'inverse image'). We discussed both, and their relationship, in Chapter 1.

This idea has lots of generalizations. For example, if we replace \(X\) with a category \(\mathcal{X}\) and replace \( \lbrace 0,1 \rbrace \) with the category \(\mathbf{Set}\), we get the following fact: a specially nice kind of functor called a 'discrete fibration' \(i_S : \mathcal{S} \to \mathcal{X}\) is the same as a functor \(\chi_S : \mathcal{X} \to \mathbf{Set}\).

The Grothendieck construction goes even further: now we we replace \(X\) with a category \(\mathcal{X}\) and replace \( \lbrace 0,1 \rbrace \) with the category \(\mathbf{Cat}\)! Now a specially nice kind of functor called 'fibration' \(i_S : \mathcal{S} \to \mathcal{X}\) is the same as a functor \(\chi_S : \mathcal{X} \to \mathbf{Cat}\).

It goes on, and I've written a whole big fat essay about this idea. For example, it's very important in topology. But in every case, the map I'm calling \(i_S\) needs to be 'nice' in some way; it can't be arbitrary.

Despite all this stuff, I'm still not optimistic about your original question.

Christopher Upshaw

What's the name of the essay?

Keith E. Peterson

John Baez's constraints for subsets can also be given by the diagram, \[ \begin{matrix} T & \overset{i} \hookrightarrow & S\\ u\downarrow & & \downarrow \chi \\ \lbrace \texttt{true} \rbrace & \overset{\ulcorner \texttt{true} \urcorner}\hookrightarrow & \lbrace \texttt{true}, \texttt{false} \rbrace \end{matrix} \] with the constraint, \[ (\chi \circ i)(t)= \left (\ulcorner \texttt{true} \urcorner \circ u \right )(t). \]

Puzzle KEP: Viewing this diagram as a category, \(\mathcal{C}\), produce an instance \(F : \mathcal{C} \to \mathbf{Set}\). Do instances of the set \(F(T)\) always correspond to a subset of an instance of the set \(F(S)\)? Why or why not?

John Baez

Christopher wrote:

What's the name of the essay?

It's Lectures on \(n\)-categories and cohomology. For understanding fibrations and such, you want the part starting in Section 1.2 where I say

Connected covering spaces of \(B\) are classified by subgroups \(H \subseteq \pi_1(B) \).

up to the part in Section 1.5 where I say

Fibrations \(E \to B\) where \(E\) and \(B\) are \(n\)-groupoids are classified by weak \((n + 1)\)-functors \(B \to n\textrm{Gpd}\).

Both these mottos, and many more, are fancied-up versions of the very basic one I forgot to say:

Subsets \(E \subseteq B\) of a set \(B\) are classified by functions \(B \to \lbrace 0,1\rbrace \).

I then go on to flesh out the philosophy connecting this idea to cohomology. In the process I do explain why truth values \( \lbrace 0,1\rbrace\) are \(n\)-groupoids with \(n = -1\)! That's in Section 2, "The power of negative thinking".

Keith E. Peterson

So then, \(\lbrace 0,1 \rbrace \) is classified by the point (\((-2)\)-groupoid)?

MatthewDoty

\[ \begin{matrix} T & \overset{i} \hookrightarrow & S\\ u\downarrow & & \downarrow \chi \\ \lbrace \texttt{true} \rbrace & \overset{\ulcorner \texttt{true} \urcorner}\hookrightarrow & \lbrace \texttt{true}, \texttt{false} \rbrace \end{matrix} \] with the constraint, \[ (\chi \circ i)(t)= \left (\ulcorner \texttt{true} \urcorner \circ u \right )(t). \]

Puzzle KEP: Viewing this diagram as a category, \(\mathcal{C}\), produce an instance \(F : \mathcal{C} \to \mathbf{Set}\). Do instances of the set \(F(T)\) always correspond to a subset of an instance of the set \(F(S)\)? Why or why not?

I'm not sure what you're getting at.

Can you say the answer Keith?

Christopher Upshaw

Well,he is sort of mixing things up, because the top half is a random category, and the bottom half is part of Set.

Keith E. Peterson

Matthew Doty wrote:

I'm not sure what you're getting at.

Can you say the answer Keith?

I'll give it until next morning to allow other people to give it a try.

Christopher Upshaw wrote:

Well,he is sort of mixing things up, because the top half is a random category, and the bottom half is part of Set.

No, the top half are sets.

Christopher Upshaw

IIUC, the -2 groupoid is {true}.

Keith E. Peterson

Yes.

And it has only one morphism, which is also invertible, \(id_{\texttt{true}}\).

Dan Oneata

John asked in comment 26:

Puzzle. Can we change the category \(\mathcal{C}\) to a category \(\mathcal{C}^\prime\) so that functors \(F : \mathcal{C}^\prime \to \mathbf{Set}\) are just databases of the sort Keith drew, with the kind of symmetry that his table has?

I think we need to add the further constraint that \( \textrm{FriendOf} \circ \textrm{FriendOf} = 1_{\textrm{People}} \).

John Baez

Dan - right! Then we're back to a category we know and love: the group \(\mathbb{Z}/2\), thought of as a category with one object and 2 morphisms.

Before you added this constraint our category was secretly the monoid \((\mathbb{N}, +, 0) \).

Robert Figura

I'd be glad to partake in a discussion about John's (and Michael Shulman's) essay "Lectures on n-categories and cohomology". I picked it up when John first mentioned it in a lecture here, and I can't let go since!

On a different note, I'm still puzzling about how to categorify database editing. I might be pushing the database metaphor too far. Or not, as it seems I might be able to combine what I know about monads into something which then describes the editing primitives abstractly. I suspect I might find answers in the papers Ryan Wisnesky suggested, which I'll check very soon.

MatthewDoty

Hey Keith:

Matthew Doty wrote:

I'm not sure what you're getting at. Can you say the answer Keith?

I'll give it until next morning to allow other people to give it a try.

Bump?

Keith E. Peterson

Oh sure, I was getting carried away with programming pictures of relations to make this very proof...

If we look at random functions into \(\lbrace \text{true} \rbrace\), we see that everything maps to the same place,

therefor \( u(t) = \lbrace \text{true} \rbrace, \text{ for all, } t \text{ in } T\).

We can then easily fill in the \(u\) part in the table, \[ \begin{array}{c|ccc} T & u & i & \chi \circ i = \ulcorner \text{true}\urcorner \circ u \\ \hline a & \lbrace \text{true} \rbrace & ? & ? \\ b & \lbrace \text{true} \rbrace & ? & ? \\ c & \lbrace \text{true} \rbrace & ? & ? \\ d & \lbrace \text{true} \rbrace & ? & ? \\ e & \lbrace \text{true} \rbrace & ? & ? \\ \vdots & \vdots & \vdots & \vdots \\ \end{array} \]

Next, note that the function \( \ulcorner \text{true}\urcorner \) picks out \(\lbrace \text{true} \rbrace \) in \(\lbrace \text{false, true} \rbrace \),

so \((\ulcorner \text{true}\urcorner \circ u)(t) = \lbrace \text{true} \rbrace, \text{ for all, } t \text{ in } T\), and we can easily fill out the table again,

\[ \begin{array}{c|ccc} T & u & i & \chi \circ i = \ulcorner \text{true}\urcorner \circ u \\ \hline a & \lbrace \text{true} \rbrace & ? & \lbrace \text{true} \rbrace \\ b & \lbrace \text{true} \rbrace & ? & \lbrace \text{true} \rbrace \\ c & \lbrace \text{true} \rbrace & ? & \lbrace \text{true} \rbrace \\ d & \lbrace \text{true} \rbrace & ? & \lbrace \text{true} \rbrace \\ e & \lbrace \text{true} \rbrace & ? & \lbrace \text{true} \rbrace \\ \vdots & \vdots & \vdots & \vdots \\ \end{array} \]

Now, we turn our attention to \(S\).

Given that \(\chi\) can only send elements of \(S\) to either \(\lbrace \text{false} \rbrace \) or \(\lbrace \text{true} \rbrace \), we see that randomly given functions look like,

that is to say, \(\chi\) partitions \(S\) into two sets, this gives tables of the form,

\begin{array}{c|c} S & \chi \\ \hline s & \lbrace \text{true} \rbrace \\ t & \lbrace \text{false} \rbrace \\ u & \lbrace \text{true} \rbrace \\ v & \lbrace \text{false} \rbrace \\ w & \lbrace \text{true} \rbrace \\ x & \lbrace \text{false} \rbrace \\ y & \lbrace \text{false} \rbrace \\ z & \lbrace \text{false} \rbrace \\ \vdots & \vdots \\ \end{array}

Now, if we have a given \(\chi\), we can create an arbitary injection \(i\), by mapping \(t \text{ in } T\) to some \(s \text{ in } S\) that maps to \(\lbrace \text{true} \rbrace\), because we have the constraint \(\chi \circ i = \ulcorner \text{true}\urcorner \circ u\).

Since I haven't produced a program that can compose functions (yet), this leaves an excellent,

Exercise: graphically find an injection \(i\),

such that \(\chi \circ i = \ulcorner \text{true}\urcorner \circ u\).

Likewise, we can get a \(\chi\) from a random injection \(i\).

Exercise: graphically find a surjection \(\chi\),

such that \(\chi \circ i = \ulcorner \text{true}\urcorner \circ u\).

Sophie Libkind

I've got a wondering about how stuff, structure, and properties fits into databases.

In the example of transforming databases from Lecture 41, the functor that ignores departmental data is full and faithful but not essentially surjective. So it "forgets at most properties". I can't figure out how to think of departmental data as a property but maybe I am trying to take the analogy too far?

In the example from this lecture, I think that \(F: \mathcal D \to \mathcal C\) is full and essentially surjective so it forgets at most stuff. So maybe it makes sense to think of nationality as "stuff" (or information) that a person has.

Here is another example. Consider let \(\mathcal E\) be the free graph on

And let \(H: \mathcal{E} \to \mathcal{C}\) map the acquaintance morphism to the morphism \(\textrm{FriendOf} \circ \textrm{FriendOf}\). \(H\) is essentially surjective and faithful so it forgets at most structure.

Jesus Lopez

John! You've just given the first account of Kan Extensions that I've managed to understand and make mine, thank you a bunch!

John Baez

Jesus - great! It took me a long time to understand them too... because most explanations make them sound more "new" and "different" than they really are.

John Baez

Sophie wrote:

In the example of transforming databases from Lecture 41, the functor that ignores departmental data is full and faithful but not essentially surjective. So it "forgets at most properties". I can't figure out how to think of departmental data as a property but maybe I am trying to take the analogy too far?

You can never take this analogy - or more precisely, this concept of 'stuff, structure and properties' - too far. But I'm a bit confused.

Let me remember what we were talking about. We created a category \(\mathcal{C}\) by taking the free category on this graph:

and a category \(\mathcal{D}\) by taking the free category on this graph:

We got an obvious 'inclusion' functor \(G : \mathcal{D} \to \mathcal{C}\).

It sounds like you're asking whether \(G\) is full, faithful and/or essentially surjective, and asking what this means in terms of stuff, structure and properties. That's fun to think about.

But here's another fun thing to think about! "Composing with \(G\)" turns databases built on the schema \(\mathcal{C}\) into databases built on the schema \(\mathcal{D}\). In other words, we have a functor

$$ \textrm{ composing with } G \; : \mathbf{Set}^\mathcal{C} \to \mathbf{Set}^\mathcal{D} $$ We can also ask if this functor is full, faithful and/or essentially surjective, and what this means in terms of stuff, structure and properties.

I think I was confused, at first, about what you were thinking about. Now I think it's really the first, since \(G\) is indeed full, faithful but not essentially surjective.

I have more to say, but not now, since it's Brandon Coya's graduation and I have to go put a hood on him!

John Baez

Okay, so continuing on... let \(\mathcal{D}\) be the free category on this graph:

and let \(\mathcal{C}\) be the free category on this graph:

We got an obvious 'inclusion' functor \(G : \mathcal{D} \to \mathcal{C}\). Now, let's try to think of this as a 'forgetful' functor. There's no technical definition of a forgetful functor, so this is just a matter of attitude. We need to imagine the objects of \(\mathcal{C}\) and \(\mathcal{D}\) as 'mathematical objects' of some kind. Their names fight against this way of thinking! That's why this problem is tough. But try.

The functor \(G\) is full, faithful, but not essentially surjective. This basically says that \(\mathcal{C}\) has more objects than \(\mathcal{D}\), but if you have two objects of \(\mathcal{C}\) that came from objects in \(\mathcal{D}\), the morphisms between them in \(\mathcal{D}\) are the same as the morphisms between them in \(\mathcal{C}\)

A great mathematical example of this sort of situation is where\( \mathcal{D} \) is the category of abelian groups and \(\mathcal{C}\) is the category of groups. Here the forgetful functor \(G : \mathcal{D} \to \mathcal{C}\) forgets the property of being abelian.

In our example, therefore, \(G\) is forgetting the property of being "not \(\textbf{Department}\)". That's a property that all objects of \(\mathcal{D}\) have, but not all objects of \(\mathcal{C}\).

I know this sounds weird. But it's just a psychological thing. If we regard the objects of \(\mathcal{D}\) as mathematical gadgets of some sort, and the objects of \(\mathcal{C}\) as mathematical gadgets with some extra property, then it's not so bad.