Interest aggregation ←

This post might contain off-putting things, for two reasons: Firstly, it is investigating methods of user data aggregation, which can be an offending topic in itself and secondly, it investigates the interests of porn website users. For educational purposes only.

The data has been politely scraped over a couple of days from a free website that hosts images/videos, blogs and public user profiles. For this post i just looked at the "interests", which are zero or more words or groups of words, that each user can specify, to be found more easily on the profile search page, or to simply state the obvious.

The data¹You can download the file interest-graph.json it contains some 2D-maps of interests and all the edges between interests that have been used together. The format: { # all "interests" sorted by most-used first "vertices": ["anal", "bdsm", ...], # all connections of "interests" [index1, index2, count] "edges": [[0,23,3367], [0,5,3278], ...], # number of times each "interest" was used "vertex_counts": [22339, 10737, ...], # list of 2D-maps "vertex_positions": [ { "name": "pca1000-tsne2", # list of xy positions "data": [[1.1901,-0.9366], [-0.5495,6.2871], ...] }, ... } } The file has been rendered with scripts/build_word_relations.py. You can have a look but it requires the scraped profiles data, of course. published in this article does not contain any user data. Instead it is a point-cloud of interests, gathered by looking at patterns in about 116K profiles. Now dive into the realms of this little porn community with the Interest Similarity Search²If you are worried that what you click and type will be transmitted to 'other parties', well.. I did not put anything like that into the javascript (main.js interest-graph.js ). That does not mean that your browser can be trusted to keep your web behaviour private. That's not how things are nowadays. and read the technical details below.³Haha, sorry. I'm over-using those newly implemented foot notes that work without javascript (thanks to Tyler Vigen for bringing it to my attention). However, this post is not so much a numerical log with endless technical terms and abbreviations like others in this blog. I assumed a more general audience and describe things in more detail. In fact this article is more suited for my data blog defgsus.github.io/blog/ but i just realized that i never want to use the Jekyll static site renderer ever again. I tried it back then because it was the 'github' thing to do but today i rather develop something myself. Which i did right here! So let's go on with this article and don't think about migrating the data blog at some point in distant time... And yes! I tried to implement recursive integrated js-free foot notes but it's a bit complicated..

Interest Similarity Search ←

Data source ←

I do not usually scrape user profiles but this data is genuinely interesting. The particular website attracts and promotes all kinds of kinkiness and perversions as long as it looks like legal content. It's a safe place for people to freely admit they require small penis humiliation, cum on food, cock milking or financial domination. And i wondered if i could produce a map of sexual interests.

Those interest strings are case-sensitive in the search page, i guess because in porn language, ANAL means something different than just anal. I lower-cased all strings but otherwise left them as they are.⁴I also left the encoding errors as they are because i found it hard to fix. For german phrases like Ärsche or Füße, it seems like utf8 encoding was latin1-decoded. Reversing that error via encode("latin1").decode("utf8") works for stuff that fits into latin1 but there is also a bit of chinese and smileys and other non-latin1 things so i just left everything as it is. girl is not girls and mother/son is not the same as mother son. The site only lists the first 5 pages for any search result and then repeats, which, i think, is a fair thing to do.⁵It's actually quite funny how it just repeats the same page. That's how you quick-fix things permanently in the shady areas of the web. I got about 147K profiles at all but included only the ones with at least 5 listed interests. That's still 116K profiles which create a graph of 137K interests and 3.2M interconnections - links between one interest and another.⁶Because the search only shows 5 pages, i searched for all interest strings that i previously found, sorted by most-famous and by last-logged-in and then repeated that for newly found interests. It's possible i got a quite skewed representation of user interests that way. Who knows? Certainly only got a fraction of the user profiles that list the top-ranking interests like anal, bdsm, bondage, mature, aso..

Graph/network representation ←

It turned out that, as graph representation, this data is quite unusable because of the overwhelming number of interconnections. The 30 most often listed interests already have like 600 connections between each other. I tried a few graph/network programs and they would stop reacting within acceptable amounts of time when loading networks with a thousand interests. Not to mention that the layouting algorithms, just by looking at interconnectedness - even weighted, do not have much chance to find a visual arrangement that really helps to understand the interest topology. anal is used 23 thousand times and is connected 230 thousands times with almost every other major interest. To visualize a network, you actually need to exclude anal, bdsm and a few others. I also filtered out edges by some criteria, just to be able to look at the rest of the network more conveniently.

This network plot shows 375 nodes and many connections have been removed. It's an interesting starting point but not good for displaying all words and all connections. And, for example, the words 'public' and 'outdoors' are at very different places in the map. One can see that they are related because they have a thick connection drawn in between but i'd rather have them close together. The layout algorithm did what it could.

Note that there is nothing much of the users left in this representation, except the sum of how often each interest has been proclaimed together. Sorting the interests-browser above by the number of edges with your query-interest calculates the relation between one interest and another by how often these terms have been used together, but it's kind of boring. The top interests are spread everywhere.

Large graphs are a heavy topic. There exist neat and well-researched algorithms to extract essential information from graphs, like community detection. But all of them have a terrible run-time for large graphs (e.g. running for days). Sensible pruning or reduction of graphs is the only means i am aware of to run any of the sophisticated graph algorithms in a short amount of time.⁷For example, check A Comprehensive Survey on Graph Reduction: Sparsification, Coarsening, and Condensation (arXiv:2402.03358)

Latent representation ←

Instead of comparing interests by their number of connections we can measure similarity in some 'representational space', which might be called latent space, embedding space, latent features or feature vectors. There are many methods to compress complex data into latent features, including many kinds of neural networks. These methods typically create a vector (a list) of numbers of fixed size that can be compared with classic numeric means. A recommendation system can then suggest similar interests by calculating a single number from two feature vectors, e.g., the euclidean distance.

As an example, i added the feature vectors of some text embedding models to the similarity search. They take in a piece of text and output a fixed-sized vector that allows comparing the texts in a numerical way. It was done using the Sentence Transformers library which supports an incalculable collection of models.⁸huggingface is the machine learning model hub with literally millions of models which makes it's model search page close to useless. You can search by tag or by full text but not both. Huh? They print the model sizes in the search results but you can't restrict your search to, e.g. small models or sort by size. I don't get it. I picked the following models:

The embeddings produced by these models have nothing (or only a microscopic fraction) to do with this dataset. They are trained on huge text corpora to facilitate the search for similar text. For example, the ibm granite model judges love very close to 'romance' and 'romantic', the general MongoDB model ties love to 'relationship' and 'couple'. The MongoDB IR model puts love close to 'innocent', 'beautiful' and 'showing off' (?), while the Zip-1 model more or less fails the love. The embeddings created by these models are a mix of syntactic and semantic features of the input text. E.g., they judge 'winter' and 'winter sports' to be similar but also 'january' or 'snow'.

Here is an example for three different words and how the embeddings of the granite model look like:

'love' and 'romance' are pretty similar. We don't see much red in the plot because it's mostly behind the green. There is a lot of blue visible, though, because the 'monster cocks' embedding is very different from the other two. Please note that an embedding by itself carries no meaning. We don't know why there is a peak at index 304 and we don't need to care. The embeddings are meaningful only in relation to one another.

If you wonder what the -tsne2 part in the search method dropdown means then jump to the visual representation part. In short, instead of publishing a lot of long embedding vectors i compressed them to 2-dimensional vectors to limit the download bandwidth.⁹This is just a static website without a backend. The similarity search runs in your browser so it needs to download all the vectors beforehand. If there was a backend, this site could post the search term to the backend and retrieve the results. In that case, the backend can crunch through any amount of data, provided the server has enough resources to respond in a quick manner.

That was a little digression on latent embeddings in general. Now, how do we create numeric vectors that represent this particular porn interests dataset?

Representation learning with PCA ←

We can try to build a model of conglomerated user interest groups from the shared interests. First build a table like this:

Then, for a start, fit a Principal Component Analysis (PCA) to the data. It will squeeze the vector of a 100K+ users into something smaller, say 1000, while preserving a high explainability of the variations in the data. I used the sklearn IncrementalPCA which can be trained in batches which avoids having to put together a potentially gigantic table in memory at once.

I'm actually not sure if it's okay to call PCA a method of representation learning because nowadays, researchers almost always mean some form of neural network training while in comparison, the PCA is an ancient machine learning technique that simply fits a linear model by error-regression. However, it's output is mathematically understandable, it represents the most prominent features in the data and it is not a black-box like most neural networks.

The components of the PCA, after fitting, each represent, in decreasing order of variance explanation, an aspect or trade of a modelled user interest group. That compresses the above table's size to number-of-interests * 1000, which is an acceptable amount of data to process further and still contains (mathematically proven) a high amount of the stuff one is interested in.

To limit the size of the json data in this article, i only included interests that are proclaimed at least 30 times, which are about 3,000. Click here to view them all. This also limits the interest-interest connections to mere 866K.¹⁰These connections are the largest part of the json data. They are included merely to show that the interest similarity search above does something different than counting the number of connections between words.

The interests compressed into numbers by the fitted PCA look like this:

To conceptually grasp the meaning of this compressed representation, first look at a plot of these numbers for the top-3 interests:

As mentioned above, the components are sorted by variance explanation. The first component explains the most variation in the data. So the mean amplitude of these numbers decreases from left to right. Every further component explains a piece of variation that is not explained by the components before. If we would calculate as many components as there are users, the PCA representation would not be a compression and the user interests could be lossless-ly reproduced from the representation.

Here is a zoomed-in view as bar plot:

We do not yet know what the components actually represent but we can see that each interest is mapped to a unique mixture of these components. For example, the amplitudes of components 10 and 11 are positive for 'bondage' and negative for 'bdsm' and then it flips at component 12. They are mutually exclusive, although bdsm and bondage are quite related interests. These components seem to explain different streams of interests within the modelled user group. 'anal' just has a small amplitude in these components so it's not about that. Probably both sub-group users like 'anal', like 23 thousand others.

To represent interests and arrive at these numbers, the PCA is learning a number-of-users-sized vectors for each component during training. Those are amplitudes of relations between each component and each user:

To calculate the amplitude of, e.g., component 10 from above for the interest 'friends', we create a vector of zeros or ones for each user, putting a one wherever the user has listed 'friends' as interest and then calculate the dot product of this vector and the component vector, which yields a single number: the amplitude of that component for the specified interest.¹¹There is also some mean-shifting and everything is done in big matrix operations but that's not relevant for understanding the concept.

As long as we have the internal PCA vectors available, we can map back and forth between interests and users. As an example, we can look at which users are weighted strongest (positively and negatively) by component 10.

The bold words are shared between these 4 positive and negative weighted users, all the other words are not shared. The non-linked words are not in the dataset (used less than 30 times) and have therefore not been seen by the PCA. Looking at the interests, it kind of gives a hint of what this component 10 is about. Or does it? For comparison, here is the same for component 0, the most important one:

I would argue that component 0 has a lot to do with the usage frequency of the interests and, in extension, because of the source material, how 'generically porn' the interest is. user82882 has more facebook-style of interests and, as painful as it is to read user18683's interests, i think they are likely not judged by the PCA as belonging to the 'generic porn interests' group. From what has been put in front of the PCA, it has no idea what the words are. It's only looking at distributions.

Similarly to the weights-per-user we can look at the interests which have the highest or lowest number for a particular component in their representation.

Haha, singing and gardening! So, yes, component 0 is highly related to the number of times an interest appears in the dataset. Semantically, analverkehr belongs to the 'typical group of interests` but the PCA is grouping it together with all the other german words. Not because it knows german but from the distributions in the dataset.

The component 10 interests ranking:

This component's amplitude is not proportional to the word frequency. Blondes and teens are completely opposite here. As well as bondage and bdsm, as we have seen before. Note that 'teen' and 'teens' are also in the opposite sides of this component. Some deep psychological archetype might divide users that identify themselves as interested in teen from users interested in teens. The terms have not been used together a single time in this dataset.

Now, whatever exact explanation behind each component's meaning might exist, the mixture of a 1000 components should give us a quite diverse map of interest territories.

In the next plot, the interests, sorted by frequency, are put on the x-axis, with the most-used on the left. The y-axis shows the PCA component amplitudes for a couple of components.

Component 0 (blue) indeed is tied to the interest frequency. I imagine it like this: PCA starts to build user groups and realizes that the most variation in all the data can be explained by how typical or atypical the interests of each user are, in terms of the usage frequency. Then it goes out and tries to find the explanation for all the remaining data variation that is not explained by component 0. Interestingly, in the plot we see that each component's amplitudes are strongest (positive or negative) at around the X-th most frequent interest. Component 50 (red) has the largest amplitudes at around the 50 most frequent interest and this tendency holds for all the other components.

This is not entirely optimal in my opinion. I'd like to calculate interest similarity with less dependence on the actual usage frequency. Eventually, i'm calculating the euclidean distance between those PCA features and a large component amplitude is not similar to a small one. I tried a couple of things to somehow normalize the PCA features¹²For example, in the initial table, instead of placing a 1.0 into each cell, i placed the number 1.0 / (a + interest-frequency) (with a between 0 and 1000) and then calculated the PCA on that data. Or i used the original PCA but divided each interest's PCA feature by the mean of it's absolute values. Some normalization methods actually worked when not comparing the normalized features by euclidean distance but instead by their Pearson correlation coefficient but i can not actually explain why it works for the normalized features but not for original features, although the correlation normalizes everything anyway.. but results got much more messy. The similarity search might consider quite unrelated interests as close to each other.

In any case, the PCA is a very powerful and efficient method to get started. Above you can select different PCA component ranges in the method choice-box. For example, pca1000 uses the whole 1000 components to compare the interests, pca100 only uses the first 100 and pca[10:1000] will use all components except the first 10. I included them just for experimentation.

Comparison with Factor Analysis ←

More common in psychology is the Factor Analysis (FA). It tries to explain the variation in the data in terms of a number of unobserved variables that are learned. As initial step, one needs to define the number of components that are taken into account. That's a bit different to PCA, where the first 100 components are the same regardless if we fit a PCA with 100, 200 or a 1000 components. The Factor Analysis gives different results depending on the number of components that are fitted. I tried a few numbers and found that 25 works quite well for this dataset. For example it groups most of the different mother son spellings nicely together, although it does not actually see the words.

Below are two plots like above but for the Factor Analysis with 10 and 25 components each:

It generally shows the same phenomenon that largest amplitudes are around the X-th most frequent interests.¹³It's not so visible in this plot but i checked it with more components as well. The component 0 is certainly tied to the frequency of the interests as with the PCA. The first 4 components also behave the same for FA10 and FA25 but the following components each describe something different in the data.

There is some open research to determine the correct number of FA factors analytically but i'm not diving into that. I just tried a few numbers and subjectively chose one. You can try the FA with 5, 15 or 25 factors and judge it yourselves.

Finally i merged the (subjectively) best working representations into the magic-mix method. It concatenates, for each interest, the PCA1000 features, the FA25 features and the granite sentence embedding into one vector (1000 + 25 + 384). The sentence embedding is multiplied by 0.3 so it has less influence on the overall interest similarity. However, since the PCA features have a lower amplitude for lesser used interest, as shown above, the sentence similarity has a stronger influence for these interests. I'm telling you because this is an article. A real-world recommendation system usually does not reveal what it is doing behind the curtains.

A little privacy notice.¹⁴For educational purposes i'm creating a map of interests, aggregated over the individual user profiles. If we transpose the initial table at the top and put the interests into the columns, we create a map of users, aggregated over their shared interests. That is what's done every day. To target you with ads that supposedly do not waste your time (that's what the ad people are told), to calculate an individual price for your purchase that archives a maximum transfer of money to the shareholders (that's what the shareholders are told), to target fickle media consumers before the next election (that's what Cambridge Analytica was bragging about some years ago) or to algorithmically lower the wages of workers to the minimum possible. https://pluralistic.net/tag/algorithmic-wage-discrimination/

Below is a comparison for the top-50 interests. It shows the closest interests in terms of edge count (how often used together) and in terms of distance of feature-vectors. Note that we got rid of the 'anal' popping up everywhere without removing it from the graph or similar destructive measures. (The number in brackets is the edge count between top-interest and the closest interest.)

Visual representation ←

Okay, so how do we browse this modelled territory of interests except via endless tables of sorted words? The latent vector of any algorithm has a specific size, in our case 1000. That requires 1000-dimensional vision and thinking capability! Fortunately, other 3-dimensional beings have developed nice tools for further compressing an N-dimensional vector to a more comprehensible 2d or 3d version. Compressing our 100K users vector into two dimensions sounds like a very lossy process for sure, but to make far-too-complex data understandable, a plot in two or three dimensions can be very informative. It's not that a 1000-dimensional problem can not be looked at in 2d, it just takes many different angles to do so.

One of the most famous algorithms is tSNE (wiki, python). It's there for the exact purpose of mapping any feature space into two or three dimensions and make it look good. You don't have to think about anything¹⁵except you have millions of items... then you have to think about what to do with your time until tSNE finishes, not even remember what tSNE stands for. Just put your vectors in and get 2d positions out. If there are clusters in the data, tSNE will make them visible. If your data is uniformly distributed, you will see it in the plot. The thing about a point in the map is not if it's east or north, but how far away it is from other points of interest.

Looks nice, doesn't it? Even without some interactivity like showing the word when hovering a point, one can see some specifics of the whole dataset. It has some homogeneous core, a couple of clusters and a few outliers.

In the Interest Similarity Search above, the similarity is calculated by calculating the euclidean distance of the resulting 2d positions. The 2d map is also displayed in a (pretty ugly) plot. You can zoom in and out (with the mouse-wheel or the zoom buttons) and explore the different interest territories. For the selected interest, all connections to other used-together interests are drawn via lines.

And that's about it. At the moment, i'm quite tired of the whole topic because i spent about two weeks on writing and data-crunching. Hope you enjoy and feel free to drop a comment on github or send me a mail cy5iZXJrZUBuZXR6a29sY2hvc2UuZGU=.