Preloader

Technology

PCB is brought to you by Fable 5

第一号 · Issue 001 · Hardware

This PCB is brought to you by Fable 5

An experiment to design a cute PCB (without touching any tools) in plain English

The board I will be journaling about.

I have been wanting to design a simple PCB for the last couple of years now. The thought of converting a design idea into a physical board and programming it to do things is fascinating to me. Long story short, I procrastinated until I tried to vibe-generate a dead simple PCB with Claude Opus 4.8 and it was terrible ! It had no idea about the orientation of the components, it did not do any proper routing. I was disappointed and accepted the fact that these tools were not there yet.

Then arrived the Fable 5. At first I was not that hopeful. One Thursday evening, around 10 hours before my weekly reset for Claude, I decided to give it another try; this time with Fable 5.

I had two rules:

  1. No manual edits or verification of the board.
  2. Every problem I face before manufacturing will be solved by Fable.

This meant I was going to trust Fable with my wallet. I decided to describe the board I want it to generate, and I was not going to be involved in the design phase. I was the end customer.

This is the prompt I gave it:

That was it, a short description of a RPI 2350 based development board which can drive an E-ink display. After working autonomously for a couple of hours it came up with the design shown in the video below.

<!– YOUTUBE: when the video is up, replace this

Everything else in this figure — the frame, the caption — stays. –>

The board as it freshly came out of Kicad ⃰.
Board
31.8 × 37.32 mm, 4 layer
MCU
RP2350A
Display
1.54″ E-ink, 200×200
Flash
8 MB QSPI
Cost
26€ per board

A closer look to some errors

The design process was not error free of course. When I showed the initial design (Claude was still working on it) of the PCB to my colleague, the first thing he wanted me to check (after recovering from the pain of seeing them tracks) was the DRC (Design Rule Checking) errors. I did not know what it was, and when we looked at it, there were indeed 65 DRC errors. Following the rule, I only mentioned the errors to Fable and did nothing else.

Fable was amazing at component selection, except for the two components I highlighted above. The big one on the top left is the SPI flash which stores the firmware and other data that you want to save. The SPI flash memory Claude chose was W25Q128JVS. It comes with the SOIC-8 wide package, but the pads designed for the memory was for a SOP-8 package, meaning the chip is too big for the pads. The bottom left component on the other hand is the transistor that switches the boost converter for the E-ink driver circuitry. As you can see it also chose the wrong package, as it is too small for the pads. I did not realise these until I uploaded the required files to JLCPCB. There I could see the issues, and I discussed it with Claude. For the W25Q128JVS it insisted that there was a SOP-8 package but I could not find it in LCSC’s library. Eventually we settled at the P25Q64SH chip.

But wait a minute, how did it even route ?

The design had 65 footprints, 54 nets and 118 unconnected lines. It is not a complex PCB by any means :P. Fable decided to use the Freerouting open-source project. The tool worked for 2 minutes, and after seventeen passes it plateaued at sixty nine connections, leaving 49 disconnected, and it could not finish the job. The remaining connections were hand-routed by Claude.

Freerouting doing its job

After I ordered the board my colleague mentioned to me the KiCadRoutingTools open-source project. It ran for 1.25 seconds and it could route all the connections with no problem. I will try this tool out for my upcoming hardware projects.

Ordering with JLCPCB

I had never ordered anything from a PCB manufacturer before. It seemed complex and I was reluctant to take the first step. Upon Claude’s compilation of the project, I asked it to prepare the required files for JLCPCB, and tell me what to select on their GUI. Man am I satisfied with JLCPCB. It was so straight forward, easy to interact with and there was no bloat. I uploaded the files, some components Claude selected were not available, we did a back and forth and voila we were done. For five fully assembled boards I paid 130 Euros, ordered the E-ink displays from a local shop and now it was the waiting game.

Some cool animations while we are waiting for the PCBs

The four layers, pulled apart.

Assembly of our board

The boards have arrived !!!

I received the PCBs and the first thing I wanted to do was to plug it in to my laptop. I have broken multiple USB modules for my Framework earlier, hence I thought checking for a short between 3v3 and ground was a no-brainer, although my colleague was suggesting me to just yeet it since this was a fully vibe-generated board. There was no short and I just plugged it in. There it was, the board was recognized and it was ready to be used.

What do I do with it ?

I already wrote some proof-of-concept apps, and they worked perfectly fine. I can read on that beautiful 1.54 inch display 😀 Stopwatch is quite handy if i need some focusing, and the album is my favourite feature since even after powering the board off, the images stay on the display thanks to E-ink.

Hands on with the finished board.

How do I feel all about this ?

Great and meh. I love how I was able to just describe the board I want in plain English, send the files overseas and then receive a fully functional board without knowing any proper PCB design knowledge. The possibilities are limitless here, and I will certainly continue doing this in the future.

That said, I was not feeling much of an accomplishment, rightfully so. I used to enjoy the learning and the struggle that came with it. Although we are in the best era to learn about something, the fact that you can make things without knowing anything about a subject puts you in an uncomfortable spot.

The future I wish to have

Yes it does suck that the joy we had while building has been sucked out of us and now we are told to enjoy building from a higher abstraction level. I am trying to adjust to this, especially at work. At work I can’t just YOLO stuff so I meticulously review all the time. It is tiring but knowing the fact that my input still matters, is rewarding. For my hobby projects tho, I will continue to YOLO it and build stuff fast without necessarily knowing about the details.

I hope one day JLCPCB or PCBWAY will have a chat-box where I can dump all my ideas and some of my illustrations, and two days later they will ship me the board. I want them to remove the middle man, and make PCB generation so much simpler and safer.

Sneak Peek

I already started working on my next project. Using Fable 5.1 with KiCAD and KiCADRoutingTools I am building an NVIDIA Jetson Orin Nano based tablet. The same rules I mentioned earlier will apply and I will let you know about the results(if I get to order it :P) .

NVIDIA Jetson Orin Nano based Tablet

Thank you for reading my journal, sharing is the most fun part of tinkering and building. I appreciate that you are part of this fun journey 🙂



Source: Hacker News

‘Really helpful’: the AI bootcamps aimed at addressing UK youth unemployment

Two teenage boys sit looking at laptop screens as a young woman in a blue TechFirst T-shirt stands over them. They are all smiling. The wall behind them is deep blue.

The course has ‘changed my opinion’ on AI, says Remy Simms, right, who is interested in applying for the content creator apprenticeship. Photograph: Christopher Thomond/The Guardian

The course has ‘changed my opinion’ on AI, says Remy Simms, right, who is interested in applying for the content creator apprenticeship. Photograph: Christopher Thomond/The Guardian

‘Really helpful’: the AI bootcamps aimed at addressing UK youth unemployment

Pilot project in Preston comes with apprenticeship offer for Neets at end of three-week course

In a youth centre opposite Preston bus station, the UK government is trying to address two of the greatest challenges facing the national economy: AI and youth unemployment.

The hope is that one will solve – rather than cause – the other. In a room at the recently opened Vault, a community hub in the Lancashire city, an “AI bootcamp” is under way – part of a trial in the north-west to give 16- to 24-year-olds AI training.

The bootcamp tutor is standing in front of a video screen and guiding the five attenders sitting around a long table on laptops through a task designed to emphasise the importance of glitch-free data in AI models.

“Does that make sense?” he asks. The room nods.

The three-week pilot boot camps are open to 70 young people who are classified as Neet or at risk of becoming so. Photograph: Christopher Thomond/The Guardian

Everyone attending these sessions is either a 16- to 24-year-old not in education, employment or training – Neet, for short – or likely to join that bracket.

Youth unemployment is a problem in the UK. Nearly a million young people are Neet – just over one in 10 people in that age group. The advent of AI, either a threat or boon to the job market depending on your perspective, is now another factor to take into consideration when tackling the youth joblessness crisis.

Khudeija Rafique, who is attending the course, sees potential in the new tech. “It would be really helpful with whatever job I would like to go into,” the 18-year-old from Burnley says.

These taxpayer-funded bootcamps are a pilot scheme, open to 70 people who will undergo a three-week programme specially designed for Neets to learn how to build AI tools, understand how businesses use AI, and learn to use the technology responsibly. The retailer JD Sports, food company Heinz and IT firm Agilysys are among the partners offering AI apprenticeships specifically designed for the attenders to join at the end of the course.

There are two types of apprenticeship available: one for aspiring content creators to become AI marketing specialists, another to join IT helpdesks that need AI expertise. If a boot-camp trainee gets an apprenticeship, they will be going to an organisation that wants these skills and has an AI-focused role to fill.

‘This is an opportunity to bridge a gap, to improve productivity and bring AI into organisations,’ says Lauren Monks. Photograph: Christopher Thomond/The Guardian

“This is an opportunity to bridge a gap, to improve productivity and bring AI into organisations,” says Lauren Monks, an executive at IN4 Group, the company contracted to run the programme.

Amid general concerns about the availability of entry-level jobs and whether AI is stifling that market, being on top of the technology can be an advantage if you’re seeking your first job.

This is common advice from graduate recruiters, but experts say it can apply to Neets as well, depending on what sector of the economy they are joining.

Remy Simms, 17, is interested in applying for the content creator apprenticeship and says the Preston course has “changed my opinion” on AI. “It shows that you can work with AI and you don’t have to work against it,” he says. “I’ve got a better understanding of it and I know more about how it works.”

Remy Simms, 17, says the Preston course has given him a ‘better understanding’ of AI. Photograph: Christopher Thomond/The Guardian

Alan Milburn, a former Labour cabinet minister, warned in a recent report that the country was “at risk of a lost generation” without government action on the Neet crisis. Milburn’s prescription goes a lot further than AI bootcamps – he calls for reform of the welfare system, for instance – and his report points out that work programmes by successive governments have failed to answer the problem.

Concerns about AI’s impact on employment have focused on the graduate job market so far, not on school leavers and non-graduates, amid expectations that it will be able to do the “grunt work” associated with junior employees in areas such as banking, consulting and law.

skip past newsletter promotion


The impact on non-graduate work was unclear, according to Dr Bouke Klein Teeselink, who is researching AI and the future of work at King’s College London.

“We don’t have really good insights into how AI affects the non-graduate market compared to the graduate market,” he says, but suggests this scheme could help answer the puzzle of getting more young people into the labour market.

The key test of the bootcamps and ensuing apprenticeships, Teeselink says, would be the thoroughness of the AI training and whether these newcomers made a real difference to their employers’ productivity. “I hope the policy will be rolled out in a way that allows the government to answer both those questions,” he says.

Hamid Muhammad-Asif, 16, says he would be happy with a content creator or IT helpdesk apprenticeship. Photograph: Christopher Thomond/The Guardian

So will the bootcamps make a difference to Neet numbers? Experts say there is no single cause of the crisis and any comprehensive solution must be multi-faceted.

Chris Goulden, the deputy chief executive of the Youth Futures Organisation, a non-profit that researches how to help young people find employment, says he does not believe AI is a driver of Neet numbers. The country still needs to increase the amount of apprenticeships it offers, and guide children who are not on a pathway to university into vocations, he says.

“Our sense is that AI is not driving the increase in Neets since the [Covid] pandemic,” he says. “It’s more to do with mental health issues and a general slowdown in the economy. But that’s not to be complacent, because AI is doing new things every day and it’s going to affect lots of jobs.”

Goulden says if the “first rung of the ladder” is being taken away by AI, it is logical to give young people a foothold in the technology. “Teaching young people to use AI is part of how we prepare them for the future,” he says.

In Preston, the bootcamp attenders are keen to take the next step.

Asked whether he wants the content creator or the IT helpdesk apprenticeship, 16-year-old Hamid Muhammad-Asif, from Blackburn, says: “I’m happy with both.”


Source: Technology

The Relation Between Mathematics and Physics by Paul Dirac

The physicist, in his study of natural phenomena, has two methods of
making progress: (1) the method of experiment and observation, and (2)
the method of mathematical reasoning. The former is just the collection
of selected data; the latter enables one to infer results about
experiments that have not been performed. There is no logical reason
why the second method should be possible at all, but one has found in
practice that it does work and meets with reasonable success. This must
be ascribed to some mathematical quality in Nature, a quality which the
casual observer of Nature would not suspect, but which nevertheless
plays an important role in Nature's scheme.

One might describe the mathematical quality in Nature by saying that the
universe is so constituted that mathematics is a useful took in its
description. However, recent advances in physical science show that
this statement of the case is too trivial. The connection between
mathematics and the description of the universe goes far deeper than
this, and one can get an appreciation of it only from a thorough
examination of the various facts that make it up. The main aim of my
talk to you will be to give you such an appreciation. I propose to deal
with how the physicist's views on this subject have been gradually
modified by the succession of recent developments in physics, and then I
would like to make a little speculation about the future.

Let us take as our starting-point that scheme of physical science which
was generally accepted in the last century – the mechanistic scheme.
This considers the whole universe to be a dynamical system (of course an
extremely complicated dynamical system), subject to laws of motion which
are essentially of the Newtonian type. The role of mathematics in this
scheme is to represent the laws of motion by equations, and to obtain
solutions of the equations referring to observed conditions.

The dominating idea in this application of mathematics to physics is
that the equations representing the laws of motion should be of a simple
form. The whole success of the scheme is due to the fact that equations
of simple form do seem to work. The physicist is thus provided with a
principle of simplicity, which he can use as an instrument of research.
If he obtains, from some rough experiments, data which fit in roughly
with certain simple equations, he infers that if he performed the
experiments more accurately he would obtain data fitting in more
accurately with the equations. The method is much restricted, however,
since the principle of simplicity applies only to fundamental laws of
motion, not to natural phenomena in general. For example, rough
experiments about the relation between the pressure and volume of a gas
at a fixed temperature give results fitting in with a law of inverse
proportionality, but it would be wrong to infer that more accurate
experiments would confirm this law with greater accuracy, as one is here
dealing with a phenomenon which is not connected in any very direct way
with the fundamental laws of motion.

The discovery of the theory of relativity made it necessary to modify
the principle of simplicity. Presumably one of the fundamental laws of
motion is the law of gravitation which, according to Newton, is
represented by a very simple equation, but, according to Einstein,
needs the development of an elaborate technique before its equation can
even be written down. It is true that, from the standpoint of higher
mathematics, one can give reasons in favour of the view that Einstein's
law of gravitation is actually simpler than Newton's, but this involves
assigning a rather subtle meaning to simplicity, which largely spoils
the practical value of the principle of simplicity as an instrument of
research into the foundations of physics.

What makes the theory of relativity so acceptable to physicists in spite
of its going against the principle of simplicity is its great
mathematical beauty. This is a quality which cannot be defined, any
more than beauty in art can be defined, but which people who study
mathematics usually have no difficulty in appreciating. The theory of
relativity introduced mathematical beauty to an unprecedented extent
into the description of Nature. The restricted theory changed our ideas
of space and time in a way that may be summarised by stating that the
group of transformations to which the space-time continuum is subject
must be changed from the Galilean group to the Lorentz group. The
latter group is a much more beautiful thing than the former – in fact,
the former would be called mathematically a degenerate special case of
the latter. The general theory of relativity involved another step of a
rather similar character, although the increase in beauty this time is
usually considered to be not quite so great as with the restricted
theory, which results in the general theory being not quite so firmly
believed in as the restricted theory.

We now see that we have to change the principle of simplicity into a
principle of mathematical beauty. The research worker, in his efforts
to express the fundamental laws of Nature in mathematical form, should
strive mainly for mathematical beauty. He should still take simplicity
into consideration in a subordinate way to beauty. (For example
Einstein, in choosing a law of gravitation, took the simplest one
compatible with his space-time continuum, and was successful.). It
often happens that the requirements of simplicity and of beauty are the
same, but where they clash the latter must take precedence.

Let us pass on to the second revolution in physical thought of the
present century – the quantum theory. This is a theory of atomic
phenomena based on a mechanics of an essentially different type from
Newton's. The difference may be expressed concisely, but in a rather
abstract way, by saying that dynamical variables in quantum mechanics
are subject to an algebra in which the commutative axiom of
multiplication does not hold. Apart from this, there is an extremely
close formal analogy between quantum mechanics and the old mechanics.
In fact, it is remarkable how adaptable the old mechanics is to the
generalization of non-commutative algebra. All the elegant features of
the old mechanics can be carried over to the new mechanics, where they
reappear with an enhanced beauty.

Quantum mechanics requires the introduction into physical theory of a
vast new domain of pure mathematics – the whole domain connected with
non-commutative multiplication. This, coming on top of the introduction
of new geometries by the theory of relativity, indicates a trend which
we may expect to continue. We may expect that in the future further big
domains of pure mathematics will have to be brought in to deal with the
advances in fundamental physics.

Pure mathematics and physics are becoming ever more closely connected,
though their methods remain different. One may describe the situation
by saying that the mathematician plays a game in which he himself
invents the rules while the physicist plays a game in which the rules
are provided by Nature, but as time goes on it becomes increasingly
evident that the rules which the mathematician finds interesting are the
same as those which Nature has chosen. It is difficult to predict what
the result of all this will be. Possibly, the two subjects will
ultimately unify, every branch of pure mathematics then having its
physical application, its importance in physics being proportional to
its interest in mathematics. At present we are, of course, very far
from this stage, even with regard to some of the most elementary
questions. For example, only four-dimensional space is of importance in
physics, while spaces with other numbers of dimensions are of about
equal interest in mathematics.

It may well be, however, that this discrepancy is due to the
incompleteness of present-day knowledge, and that future developments
will show four-dimensional space to be of far greater mathematical
interest than all the others.

The trend of mathematics and physics towards unification provides the
physicist with a powerful new method of research into the foundations of
his subject, a method which has not yet been applied successfully, but
which I feel confident will prove its value in the future. The method
is to begin by choosing that branch of mathematics which one thinks will
form the basis of the new theory. One should be influenced very much in
this choice by considerations of mathematical beauty. It would probably
be a good thing also to give a preference to those branches of
mathematics that have an interesting group of transformations underlying
them, since transformations play an important role in modern physical
theory, both relativity and quantum theory seeming to show that
transformations are of more fundamental importance than equations.
Having decided on the branch of mathematics, one should proceed to
develop it along suitable lines, at the same time looking for that way
in which it appears to lend itself naturally to physical interpretation.

This method was used by Jordan in an attempt to get an improved quantum
theory on the basis of an algebra with non-associative multiplication.
The attempt was not successful, as one would rather expect, if one
considers that non-associative algebra is not a specially beautiful
branch of mathematics, and is not connected with an interesting
transformation theory. I would suggest, as a more hopeful-looking idea
for getting an improved quantum theory, that one take as basis the
theory of functions of a complex variable. This branch of mathematics
is of exceptional beauty, and further, the group of transformations in
the complex plane, is the same as the Lorentz group governing the
space-time of restricted relativity. One is thus led to suspect the
existence of some deep-lying connection between the theory of functions
of a complex variable and the space-time of restricted relativity, the
working out of which will be a difficult task for the future.

Let us now discuss the extent of the mathematical quality in Nature.
According to the mechanistic scheme of physics or to its relativistic
modification, one needs for the complete description of the universe not
merely a complete system of equations of motion, but also a complete set
of initial conditions, and it is only to the former of these that
mathematical theories apply. The latter are considered to be not
amenable to theoretical treatment and to be determinable only from
observation.

The enormous complexity of the universe is ascribed to an enormous
complexity in the initial conditions, which removes them beyond the
range of mathematical discussion.

I find this position very unsatisfactory philosophically, as it goes
against all ideas of the unity of Nature. Anyhow, if it is only to a
part of the description of the universe that mathematical theory
applies, this part ought certainly to be sharply distinguished from the
remainder. But in fact there does not seem to be any natural place in
which to draw the line. Are such things as the properties of the
elementary particles of physics, their masses and the numerical
coefficients occurring in their laws of force, subject to mathematical
theory? According to the narrow mechanistic view, they should be
counted as initial conditions and outside mathematical theory. However,
since the elementary particles all belong to one or other of a number of
definite types, the members of one type being all exactly similar, they
must be governed by mathematical law to some extent, and most physicists
now consider it to be quite a large extent. For example, Eddington has
been building up a theory to account for the masses. But even if one
supposed all the properties of the elementary particles to be
determinable by theory, one would still not know where to draw the line,
as one would be faced by the next question – Are the relative abundances
of the various chemical elements determinable by theory? One would pass
gradually from atomic to astronomic questions.

This unsatisfactory situation gets changed for the worse by the new
quantum mechanics. In spite of the great analogy between quantum
mechanics and the older mechanics with regard to their mathematical
formalisms, they differ drastically with regard to the nature of their
physical consequences. According to the older mechanics, the result of
any observation is determinate and can be calculated theoretically from
given initial conditions; but with quantum mechanics there is usually an
indeterminacy in the result of an observation, connected with the
possibility of occurrence of a quantum jump, and the most that can be
calculated theoretically is the probability of any particular result
being obtained. The question, which particular result will be obtained
in some particular case, lies outside the theory. This must not be
attributed to an incompleteness of the theory, but is essential for the
application of a formalism of the kind used by quantum mechanics.

Thus according to quantum mechanics we need, for a complete description
of the universe, not only the laws of motion and the initial conditions,
but also information about which quantum jump occurs in each case when a
quantum jump does occur. The latter information must be included,
together with the initial conditions, in that part of the description of
the universe outside mathematical theory.

The increase thus arising in the non-mathematical part of the
description of the universe provides a philosophical objection to
quantum mechanics, and is, I believe, the underlying reason why some
physicists still find it difficult to accept this mechanics. Quantum
mechanics should not be abandoned, however, firstly, because of its very
widespread and detailed agreement with experiment, and secondly, because
the indeterminacy it introduces into the results of observations is of a
kind which is philosophically satisfying, being readily ascribable to an
inescapable crudeness in the means of observation available for
small-scale experiments. The objection does show, all the same, that
the foundations of physics are still far from their final form.

We come now to the third great development of physical science of the
present century – the new cosmology. This will probably turn out to be
philosophically even more revolutionary than relativity or the quantum
theory, although at present one can hardly realize its full
implications. The starting-point is the observed red-shift in the
spectra of distance heavenly bodies, indicating that they are receding
from us with velocities proportional to their distances.* The
velocities of the more distant ones are so enormous that it is evident
we have here a fact of the utmost importance, not a temporary or local
condition, but something fundamental for our picture of the universe.

If we go backwards into the past we come to a time, about 2 x 109 years
ago, when all the matter in the universe was concentrated in a very
small volume. It seems as though something like an explosion then took
place, the fragments of which we now observe still scattering outwards.
This picture has been elaborated by LemaÏtre, who considers the universe
to have started as a single very heavy atom, which underwent violent
radioactive disintegrations and so broke up into the present collection
of astronomical bodies, at the same time giving off the cosmic rays.

With this kind of cosmological picture one is led to suppose that there
was a beginning of time, and that it is meaningless to inquire into what
happened before then. One can get a rough idea of the geometrical
relationships this involves by imagining the present to be the surface
of a sphere, going into the past to be going in towards the centre of
the sphere, and going into the future to be going outwards. There is
then no limit to how far one may go into the future, but there is a
limit to how far one can go into the past, corresponding to when one has
reached the centre of the sphere. The beginning of time provides a
natural origin from which to measure the time of any event. The result
is usually called the epoch of that event. Thus the present epoch is 2
x 109 years.

Let us now return to dynamical questions. With the new cosmology the
universe must have been started off in some very simple way. What,
then, becomes of the initial conditions required by dynamical theory?
Plainly there cannot be any, or they must be trivial. We are left in a
situation which would be untenable with the old mechanics. If the
universe were simply the motion which follows from a given scheme of
equations of motion with trivial initial conditions, it could not
contain the complexity we observe. Quantum mechanics provides an escape
from the difficulty. It enables us to ascribe the complexity to the
quantum jumps, lying outside the scheme of equations of motion. The
quantum jumps now form the uncalculable part of natural phenomena, to
replace the initial conditions of the old mechanistic view.

One further point in connection with the new cosmology is worthy of
note. At the beginning of time the laws of Nature were probably very
different from what they are now. Thus we should consider the laws of
Nature as continually changing with the epoch, instead of as holding
uniformly throughout space-time. This idea was first put forward by
Milne, who worked it out on the assumptions that the universe at a given
epoch is roughly everywhere uniform and spherically symmetrical. I find
these assumptions not very satisfying, because the local departures from
uniformity are so great and are of such essential importance for our
world of life that it seems unlikely there should be a principle of
uniformity overlying them. Further, as we already have the laws of
Nature depending on the epoch, we should expect them also to depend on
position in space, in order to preserve the beautiful idea of the theory
of relativity there is fundamental similarity between space and time.
This goes more drastically against Milne's assumptions than a mere lack
of uniformity in the distribution of matter.

We have followed through the main course of the development of the
relation between mathematics and physics up to the present time, and
have reached a stage where it becomes interesting to indulge in
speculations about the future. There has always been an unsatisfactory
feature in the relation, namely, the limitation in the extent to which
mathematical theory applies to a description of the physical universe.
The part to which it does not apply has suffered an increase with the
arrival of quantum mechanics and a decrease with the arrival of the new
cosmology, but has always remained.

This feature is so unsatisfactory that I think it safe to predict it
will disappear in the future, in spite of the startling changes in our
ordinary ideas to which we should then be led. It would mean the
existence of a scheme in which the whole of the description of the
universe has its mathematical counterpart, and we must suppose that a
person with a complete knowledge of mathematics could deduce, not only
astronomical data, but also all the historical events that take place in
the world, even the most trivial ones. Of course, it must be beyond
human power actually to make these deductions, since life as we know it
would be impossible if one could calculate future events, but the
methods of making them would have to be well defined. The scheme could
not be subject to the principle of simplicity since it would have to be
extremely complicated, but it may well be subject to the principle of
mathematical beauty.

I would like to put forward a suggestion as to how such a scheme might
be realized. If we express the present epoch, 2 x 109 years, in terms
of a unit of time defined by the atomic constants, we get a number of
the order 1039, which characterizes the present in an absolute sense.
Might it not be that all present events correspond to properties of this
large number, and, more generally, that the whole history of the
universe corresponds to properties of the whole sequence of natural
numbers? At first sight it would seem that the universe is far too
complex for such a correspondence to be possible. But I think this
objection cannot be maintained, since a number of the order 1039 is
excessively complicated, just because it is so enormous. We have a
brief way of writing it down, but this should not blind us to the fact
that it must have excessivly complicated properties.

There is thus a possibility that the ancient dream of philosophers to
connect all Nature with the properties of whole numbers will some day be
realized. To do so physics will have to develop a long way to establish
the details of how the correspondence is to be made. One hint for this
development seems pretty obvious, namely, the study of whole numbers in
modern mathematics is inextricably bound up with the theory of functions
of a complex variable, which theory we have already seen has a good
chance of forming the basis of the physics of the future. The working
out of this idea would lead to a connection between atomic theory and
cosmology.

* The recession velocities are not strictly proved, since one may
postulate some other cause for the spectral red-shift. However, the new
cause would presumably be equally drastic in its effect on cosmological
theory and would still need the introduction of a parameter of the order
2 x 109 years for its mathematical discussion, so it would probably not
disturb the essential ideas of the argument in the text.


Source: Hacker News

Japan's book scene is moving from bookstores to libraries


Source: Hacker News

Reversing Factorio's RNG

Reversing Factorio’s RNG

Published at August 22, 2026

#Factorio#RNG#reverse-engineering

Broken in Factorio 2.1 – Version 2.0 only!

Factorio 2.1 changes the way the RNG is used.

This breaks my in-game implementations.
The theoretical aspects of how the RNG works still apply, as they still use the same RNG.
For more info see section 6.1. Factorio 2.1.

Introduction

With the release of the Space-Age DLC in Factorio several new mechanics were introduced.
One major mechanic was the new concept of different items and building qualities.
By default, items are created with common quality.
If quality modules are used in the crafting machine we gain a small chance to obtain items of higher quality.

What does that entail? In short: Items and buildings gain improved stats, such as faster crafting speeds, modules providing stronger buffs, power poles having an increased range and inserters swinging faster.
Thats pretty neat – hence we are interested in obtaining the highest possible quality on our items and buildings.
To source a large number of such items the devs essentially said that this randomness boils down to “basically statistics”,1 i.e. if the volume high quality items one obtains is sufficiently large, then the observed distribution of qualities will be close to the expected distribution.

But is it the only way to scale?
Thinking about it, one might ponder:

How is it possible for a deterministic game like Factorio to have a random mechanic?

The short answer is: It isn’t random.

Instead – as is common in computing – the simulation makes use of a pseudo-random number generator (PRNG).
A PRNG is a deterministic algorithm which produces a sequence of numbers which for all intents and purposes appears to be random.
In particular this means properties like it following a well defined distribution of outputs, which contains no discernable patterns.
Normally in computer science one can get away with treating the PRNG as just a black box function which can yield random numbers, without concerning oneself with how it actually works.
Yet by taking a look under the hood we can do something funny.

The Funny: What happens if we know the exact algorithm and its internal state?

Then we could just run the same algorithm on the state and obtain the same outputs, which will also be seen by the game internally.
Its necessarily always the same outputs, as otherwise the chosen algorithm would not be deterministic.
As such we can run the same computations simultaneously to the game and predict the future outputs of the PRNG,
allowing us to predict the future “random” events which will occur in the game, such as which crafts will observe an increase in quality.

In the following sections I’ll build up to that point, starting from what RNG the game uses, how it is breakable and how it can be abused ingame.
The entire background should be understandable if you have a rudimentary understanding of linear algebra. That should be the only prerequisite.

Starting from Nothing

Soooo, how does one figure out what PRNG algorithm Factorio uses?
Afterall, there are several different implementations out there they could have chosen from.

To figure this out, my first step was a rudimentary internet research.
As the Factorio community is quite large and filled with many technically inclined individuals, surely someone must have asked this question before.
After digging around a bit I found a post on the Factorio forums asking basically
the same thing I wanted to know, though 11 years have passed since then.
In that thread, we also find the following answer by Cube, a former developer at Wube. They wrote:

Cube – Tue Sep 30, 2014 – forums.factorio.com: Topic 5995

[…] We chose taus88 mainly because it is the fastest from boost’s generators.
I was thinking of removing one of the three LFSRs (that should make it about
40% (?) faster), but there is no point, since the ran[d]om numbers are not a
bottleneck for us.

This already gives us a lead on where to look next: the Boost.Random library.
There we find the following (abbreviated) implementation for the taus88 generator:

typedef xor_combine_engine<
  xor_combine_engine<
    linear_feedback_shift_engine<uint32_t, 32, 31, 13, 12>, 0,
    linear_feedback_shift_engine<uint32_t, 32, 29, 2, 4>, 0>, 0,
  linear_feedback_shift_engine<uint32_t, 32, 28, 3, 17>, 0> taus88;

template<class UIntType, int w, int k, int q, int s>
class linear_feedback_shift_engine {
  // w = word size (e.g. 32 for 32 bit uint)
  // k = number of bits in the LFSR
  // q = feedback tap position
  // s = number of steps to do at once
  // wordmask() = 0b11...111; mask of w low bits set
  result_type operator()() {
    const UIntType b = (((value << q) ^ value) & wordmask()) >> (k-s);
    const UIntType mask = (wordmask() << (w-k)) & wordmask();
    value = ((value & mask) << s) ^ b;
    return value;
  }
}

This means that taus88 consists of 3 linear_feedback_shift_engine whose results are XORed together.
Note that this linear feedback shift engine is more commonly referred to as a linear feedback shift register (LFSR).

Though when starting the project that post was already 8 years old.
Hence, I wanted to cross-check the information given on the forum against the most
accurate source available: The game binary.

While the game itself is closed source, the developers graciously ship a .pdb file containing the debug symbols alongside the game binary.
This means we can generate a well-annotated decompilation of the binary to inspect the code and figure out what is going on under the hood.
To this end, I used the open source decompilation tool Ghidra, switching to Binary Ninja later on in the project.
Regardless of which tool one uses, one can rather quickly find the RandomGenerator class in the game’s code, where getInt() is implemented as follows:

uint RandomGenerator::getInt(RandomGenerator *this) {
  uint a = this->seed1;
  uint b = this->seed2;
  uint c = this->seed3;

  a = (a << 12 ^ a >> 6) & 0x1fff ^ a >> 19 ^ a << 12;
  b = (b << 4 ^ b >> 23) & 0x7f ^ b >> 25 ^ b << 4;
  c = (c << 17 ^ c >> 8) & 0x1fffff ^ c >> 11 ^ c << 17;

  this->seed1 = a;
  this->seed2 = b;
  this->seed3 = c;

  return a ^ b ^ c;
}

The first thing which stands out is that almost none of the constants used in the original taus88 definition remain.
This can be attributed to the compiler performing optimizations such as constant folding to reduce the number of required operations.
Yet the fact that we store 3 seeds for our RNG state is a first strong indicator that it is indeed the same generator.
Likewise, the states are updated independently of one another, with the final result being the XOR of all three states.

To rid myself of all remaining doubt about whether the two implementations are equivalent, I rewrote both
variants in Python so they can be run using sympy, a Python library for symbolic manipulation.
Advancing both variants by a single step confirms that all bits of the corresponding registers update
in the exact same fashion. I used sympy here because doing these equivalence checks by
hand (3⋅32=963 cdot 32 = 96) would have been quite tedious.
The corresponding code can be found here.

Ok, with all that established, we are certain that the RNG used in Factorio is indeed the taus88 generator,
which itself is a combination of 3 LFSRs.
This is a very interesting result, as LFSRs are known to be quite weak PRNGs –
in the literature one even finds the statement that they are trivially breakable.2

To understand what makes them ”weak” and how we can exploit this weakness to predict the future RNG calls,
we first need to look at the underlying mathematics of LFSRs, which is the subject of the next section.

LFSR Maths Review

To begin, we need to understand what the linear feedback shift register (LFSR) actually models.
First, consider a simple register.
It describes a collection of bits, aggregated into a single value xx<!–>:

–>x=xn−1…x2x1x0x = x_{n-1} dots x_2 x_1 x_0<!–>

Each individual bit –>xix_i<!–> can be seen as a binary variable with –>xi∈{0,1}=F2x_i in {0, 1} = mathbb{F}_2.
While it is common to consider this register value xx<!–> to represent a number in the range –>[0,2n − 1]⊆Z[0, 2^n~-~1] subseteq mathbb{Z},
it is more useful in our case to instead consider the register to actually describe a vector of individual bits, i.e. x∈F2nx in mathbb{F}_2^n.
Additionally, we consider two operations which operate on each individual bit:

  1. xi⊕yix_i oplus y_i: The XOR operation takes 2 bits and returns 1 if the bits differ and 0 if they are equal.
    Note that this is equivalent to addition modulo 2.
  2. xi⋅yix_i cdot y_i: The AND operation takes 2 bits and returns 1 if both bits are 1, otherwise it returns 0.
    This is equivalent to multiplication modulo 2.

Now extend that notion of a register into a shift register.
To shift, move all bits in the register downwards by taking each higher bit and shifting it 1 position down,
dropping the lowest bit as the output.
Here we follow the convention that the most significant bit (highest bit) is the leftmost bit xn−1x_{n-1}<!–> and the least significant bit is the rightmost bit –>x0x_0.

You can interact with this kind of register below.
Tap the individual bits to toggle them, and use the controls to start, stop and step the registers.

1000011

]–>

Well… This is boring! We converge pretty quickly to the same value of 0 regardless of the initial state.
This does not seem random at all!
To keep our shift register from always just discarding all information we will add another component,
namely some feedback.
The first idea is to just loop the discarded lowest bit back into the highest bit,
as it previously had no preceding bit from which it could obtain (new) information,
while we discard information in the lowest bit.
Doing this we have basically implemented a bit roll operation:

0000011

]–>

Hmmm… At least we no longer always arrive at an empty register.
But the sequence when the last bit lights up is very predictable.
This is due to the fact that the information that we observe repeats every nn steps
– each bit remains unmodified after all!
Thanks to the low cycle length, one can again quickly spot the pattern produced by our current feedback shift register.
To combat this we insert some linear feedback, by adding the feedback not
only to the first bit, but also some intermediate bits.
Note that addition here means XOR, as we are working with individual bits:

0000011

]–>

Unlike the last steps it might not be immediately obvious why this step is named the way it is.
It comes from the fact that the XOR ⊕oplus operation
we use to combine the feedback into the inner bits causes our bit states to be a
linear combination of the previous bit states. This linearity is also the reason
why we can reconstruct the internal RNG state from just observations alone,
and why standalone LFSRs are cryptographically weak PRNGs.

Note that in all the cases above only the last bit was considered an output.
However, in practice it is more common to output the entire register state as the result of the RNG call.
This then allows one to reinterpret the number as a proper integer in [0,2n−1][0, 2^n-1] producing random looking numbers.

We now know how an LFSR gets constructed.
In particular, given the state x(t)x^{(t)}<!–> at time –>tt<!–> we now know how to:

  1. Generate the next state –>x(t+1)x^{(t+1)}.
  2. Generate corresponding output bits, either one at a time, or as a full integer value.
Regarding the Cycle length of LFSRs

The cycle length of an LFSR is the number of steps it takes until the state repeats.
For an LFSR with nn<!–> bits, the maximum cycle length is –>2n−12^n – 1<!–> (every state except the all-zero state).

LFSR mechanics are actually also modeled by polynomials over –>F2mathbb{F}_2.
A single step then corresponds to multiplying the current state S(X)=∑i=0n−1xiXiS(X) = sum_{i=0}^{n-1} x_i X^i<!–> with –>XX<!–> (this shifts the bits up one index).

Finally, by doing this modulo a feedback polynomial –>P(X)P(X)<!–> of degree –>nn<!–> that is primitive (and therefore irreducible), we can ensure that the cycle length is maximal, i.e. –>2n−12^n – 1<!–>. –>P(X)P(X) then specifies to the spaces where the feedback is inserted, i.e. the bits which are XORed with the feedback bit.
In other words, not all feedback configurations are equally good, and the choice of feedback taps is crucial to ensure a long cycle length.

LFSRs are linear

Let us take another look at the linearity claim from above.
To do that rather than considering arbitrary instantiations of xx,
i.e. states where all bits were set to either 0 or 1,
we can now consider a symbolic representation of the LFSR.
We still start at an arbitrary point in time t=0t=0, at which we label each individual bit
with an additional symbolic variable xix_i.
Then we track how the bits evolve over time as we apply the LFSR transition rules.
Each symbol is colored either on or off depending on the state at which the LFSR was started.
As before, you can toggle individual bits by clicking on them – though only while the bit labels are in the initial state.

0001

]–>

We see that every bit is always just a combination of the initial bit states.
Note that whenever we observe xi⊕xix_i oplus x_i<!–> the –>xix_i cancels out, allowing us to remove it from the equation again.
While stepping, each instantiated bit will always stay equal to the parity of “on” bits in the symbolic combination depending on the state when we assigned the labels.
Additionally we notice that a bit is either just the preceding bit state, or it is a combination of the preceding and the feedback state.

Now what has this got to do with linearity?

First of all, note that the set of bits F2={0,1}mathbb{F}_2 = {0, 1}<!–> joined with the operations –>⊕oplus<!–> and –>⋅cdot form a structure known as a field.
This field is commonly known as the Galois field GF(2) = Z/2Z = F2text{GF}(2)~=~mathbb{Z}/2mathbb{Z}~=~mathbb{F}_2 or the modular arithmetic mod 2.
A field is just a math term for a set of values joined with some operations which satisfy a certain set of properties (see below).

Field properties of (F2,⊕,⋅)(mathbb{F}_2, oplus, cdot)<!–>

The properties which need to be satisfied to declare –>(F2,⊕,⋅)(mathbb{F}_2, oplus, cdot)<!–> a field are as follows:

  • Associativity (both –>⊕oplus<!–> and –>⋅cdot<!–>): –>a⊕(b⊕c)=(a⊕b)⊕ca oplus (b oplus c) = (a oplus b) oplus c<!–> and –>a⋅(b⋅c)=(a⋅b)⋅ca cdot (b cdot c) = (a cdot b) cdot c<!–>.
  • Commutativity (both –>⊕oplus<!–> and –>⋅cdot<!–>): –>a⊕b=b⊕aa oplus b = b oplus a<!–> and –>a⋅b=b⋅aa cdot b = b cdot a<!–>.
  • Identity: For –>⊕oplus<!–> this is –>00<!–>: –>0⊕a=a0 oplus a = a<!–> and for –>⋅cdot<!–> this is –>11<!–>: –>1⋅a=a1 cdot a = a<!–>.
  • Additive Inverse: For any –>aa<!–> we have a –>−a-a<!–> such that –>a+(−a)=0a + (-a) = 0<!–>. Note that here we have –>a=−aa = -a<!–>.
  • Multiplicative Inverse: For any –>a≠0a neq 0<!–> we have –>a−1a^{-1}<!–> such that –>a⋅a−1=1a cdot a^{-1} = 1<!–> is trivial as the only other element is 1.
  • Distributivity: –>a⋅(b⊕c)=(a⊕b)⋅(a⊕c)a cdot (b oplus c) = (a oplus b) cdot (a oplus c)<!–>.

Note that these properties can easily be checked for –>F2mathbb{F}_2 with truth tables,
at most 8 rows are necessary.

Why do we care about this?
Because having a field structure is a prerequisite for vector spaces.
In particular here, we consider the vector space spanning all vectors of length nn<!–> over the field –>GF(2)text{GF}(2)<!–>, which is denoted as –>GF(2)ntext{GF}(2)^n.
The operators are now applied pointwise to each coordinate of the vector using the operators from original the field GF(2)text{GF}(2).
And wherever we have a vector space, we can talk about linear combinations of vectors.

Personally, after getting an introduction into linear algebra and vector spaces within that abstract framework,
I subsequently only ever saw them applied to either Rnmathbb{R}^n<!–> or, if spicy, to –>Cnmathbb{C}^n.
However, the original definition of a vector space is kept very generic on purpose!
It allows any structure which satisfies the necessary properties to be manipulated in the same way,
enabling us to apply well-known algorithms that you may have only seen applied to systems described by Rm×nmathbb{R}^{m times n}<!–> matrices to arbitrary matrices, regardless of the underlying field –>FF<!–>.3 And as luck would have it, the previously defined operations XOR and AND on the bits span a field!

Taking another look at the symbolic example, we can reformulate each individual bits transition as a linear combination of previous bit states:

–>x3(t+1)=x0(t)x2(t+1)=x3(t)⊕x0(t)x1(t+1)=x2(t)x0(t+1)=x1(t)begin{align*} x_3^{(t+1)} &= x_0^{(t)} \ x_2^{(t+1)} &= x_3^{(t)} oplus x_0^{(t)} \ x_1^{(t+1)} &= x_2^{(t)} \ x_0^{(t+1)} &= x_1^{(t)} end{align*}

Since we can write the entire state as a vector of these bits, and each transition is linear itself,
this means we can write the transition between the current state s(t)∈GF(2)ns^{(t)} in text{GF}(2)^n<!–> to the next state –>s(t+1)∈GF(2)ns^{(t+1)} in text{GF}(2)^n<!–> as a matrix product:

–>s(t+1)=Ts(t)s^{(t+1)} = T s^{(t)}<!–>

where –>T∈GF(2)n×nT in text{GF}(2)^{n times n}.
Note that like the bits in the state vector, each individual entry in the matrix is a value in GF(2)text{GF}(2),
i.e. it’s either 0 or 1.
In particular this allows us to visualize this matrix as a bitmap, where a bright entry means a 1 and a dark pixel represents a 0.
For the 6 bit wide toy LFSRs we saw previously, this looks as follows:

This mathematical notation allows us to start rewriting some operations in a more compact way.
The most notable of them is the ability to advance the LFSR by multiple steps at once in compact notation:

s(t+k)=Ts(t+k−1)=T2s(t+k−2)=⋯=Tks(t)s^{(t+k)} = T s^{(t+k-1)} = T^2 s^{(t+k-2)} = dots = T^k s^{(t)}

Now that we have seen a bunch of theory, we can actually apply it to the above code snippets.
Focusing on a single LFSR component we have:

a = (a << 12 ^ a >> 6) & 0x1fff ^ a >> 19 ^ a << 12;

This can be written out for each individual bit and evaluated.
In turn we obtain a system of 32 equations, as each LFSR is defined over uint32_t words, which have 32 bits.
Note that some of the bits are actually redundant due to the construction of the LFSRs in the taus88 library: kk<!–> the LFSR size is always chosen smaller than –>ww<!–> the word size.

[–>

a31′=((a19⊕0)⋅0) ⊕0 ⊕a19=a19a30′=((a18⊕0)⋅0) ⊕0 ⊕a18=a18a29′=((a17⊕0)⋅0) ⊕0 ⊕a17=a17a28′=((a16⊕0)⋅0) ⊕0 ⊕a16=a16a27′=((a15⊕0)⋅0) ⊕0 ⊕a15=a15a26′=((a14⊕0)⋅0) ⊕0 ⊕a14=a14a25′=((a13⊕a31)⋅0) ⊕0 ⊕a13=a13a24′=((a12⊕a30)⋅0) ⊕0 ⊕a12=a12a23′=((a11⊕a29)⋅0) ⊕0 ⊕a11=a11a22′=((a10⊕a28)⋅0) ⊕0 ⊕a10=a10a21′=((a9⊕a27)⋅0) ⊕0 ⊕a9=a9a20′=((a8⊕a26)⋅0) ⊕0 ⊕a8=a8a19′=((a7⊕a25)⋅0) ⊕0 ⊕a7=a7a18′=((a6⊕a24)⋅0) ⊕0 ⊕a6=a6a17′=((a5⊕a23)⋅0) ⊕0 ⊕a5=a5a16′=((a4⊕a22)⋅0) ⊕0 ⊕a4=a4a15′=((a3⊕a21)⋅0) ⊕0 ⊕a3=a3a14′=((a2⊕a20)⋅0) ⊕0 ⊕a2=a2a13′=((a1⊕a19)⋅0) ⊕0 ⊕a1=a1a12′=((a0⊕a18)⋅1) ⊕0 ⊕a0=a18a11′=((0⊕a17)⋅1) ⊕a31 ⊕0=a17 ⊕a31a10′=((0⊕a16)⋅1) ⊕a30 ⊕0=a16 ⊕a30a9′=((0⊕a15)⋅1) ⊕a29 ⊕0=a15 ⊕a29a8′=((0⊕a14)⋅1) ⊕a28 ⊕0=a14 ⊕a28a7′=((0⊕a13)⋅1) ⊕a27 ⊕0=a13 ⊕a27a6′=((0⊕a12)⋅1) ⊕a26 ⊕0=a12 ⊕a26a5′=((0⊕a11)⋅1) ⊕a25 ⊕0=a11 ⊕a25a4′=((0⊕a10)⋅1) ⊕a24 ⊕0=a10 ⊕a24a3′=((0⊕a9)⋅1) ⊕a23 ⊕0=a9 ⊕a23a2′=((0⊕a8)⋅1) ⊕a22 ⊕0=a8 ⊕a22a1′=((0⊕a7)⋅1) ⊕a21 ⊕0=a7 ⊕a21a0′=((0⊕a6)⋅1) ⊕a20 ⊕0=a6 ⊕a20begin{array}{rrrrrrrrc} a_{31}’ = big((& a_{19} &oplus& 0 ) cdot 0big) oplus& 0 & oplus& a_{19} =& & a_{19} \ a_{30}’ = big((& a_{18} &oplus& 0 ) cdot 0big) oplus& 0 & oplus& a_{18} =& & a_{18} \ a_{29}’ = big((& a_{17} &oplus& 0 ) cdot 0big) oplus& 0 & oplus& a_{17} =& & a_{17} \ a_{28}’ = big((& a_{16} &oplus& 0 ) cdot 0big) oplus& 0 & oplus& a_{16} =& & a_{16} \ a_{27}’ = big((& a_{15} &oplus& 0 ) cdot 0big) oplus& 0 & oplus& a_{15} =& & a_{15} \ a_{26}’ = big((& a_{14} &oplus& 0 ) cdot 0big) oplus& 0 & oplus& a_{14} =& & a_{14} \ a_{25}’ = big((& a_{13} &oplus& a_{31}) cdot 0big) oplus& 0 & oplus& a_{13} =& & a_{13} \ a_{24}’ = big((& a_{12} &oplus& a_{30}) cdot 0big) oplus& 0 & oplus& a_{12} =& & a_{12} \ a_{23}’ = big((& a_{11} &oplus& a_{29}) cdot 0big) oplus& 0 & oplus& a_{11} =& & a_{11} \ a_{22}’ = big((& a_{10} &oplus& a_{28}) cdot 0big) oplus& 0 & oplus& a_{10} =& & a_{10} \ a_{21}’ = big((& a_{9} &oplus& a_{27}) cdot 0big) oplus& 0 & oplus& a_{9} =& & a_{9} \ a_{20}’ = big((& a_{8} &oplus& a_{26}) cdot 0big) oplus& 0 & oplus& a_{8} =& & a_{8} \ a_{19}’ = big((& a_{7} &oplus& a_{25}) cdot 0big) oplus& 0 & oplus& a_{7} =& & a_{7} \ a_{18}’ = big((& a_{6} &oplus& a_{24}) cdot 0big) oplus& 0 & oplus& a_{6} =& & a_{6} \ a_{17}’ = big((& a_{5} &oplus& a_{23}) cdot 0big) oplus& 0 & oplus& a_{5} =& & a_{5} \ a_{16}’ = big((& a_{4} &oplus& a_{22}) cdot 0big) oplus& 0 & oplus& a_{4} =& & a_{4} \ a_{15}’ = big((& a_{3} &oplus& a_{21}) cdot 0big) oplus& 0 & oplus& a_{3} =& & a_{3} \ a_{14}’ = big((& a_{2} &oplus& a_{20}) cdot 0big) oplus& 0 & oplus& a_{2} =& & a_{2} \ a_{13}’ = big((& a_{1} &oplus& a_{19}) cdot 0big) oplus& 0 & oplus& a_{1} =& & a_{1} \ a_{12}’ = big((& a_{0} &oplus& a_{18}) cdot 1big) oplus& 0 & oplus& a_{0} =& & a_{18} \ a_{11}’ = big((& 0 &oplus& a_{17}) cdot 1big) oplus& a_{31} & oplus& 0 =& a_{17} oplus & a_{31} \ a_{10}’ = big((& 0 &oplus& a_{16}) cdot 1big) oplus& a_{30} & oplus& 0 =& a_{16} oplus & a_{30} \ a_{9}’ = big((& 0 &oplus& a_{15}) cdot 1big) oplus& a_{29} & oplus& 0 =& a_{15} oplus & a_{29} \ a_{8}’ = big((& 0 &oplus& a_{14}) cdot 1big) oplus& a_{28} & oplus& 0 =& a_{14} oplus & a_{28} \ a_{7}’ = big((& 0 &oplus& a_{13}) cdot 1big) oplus& a_{27} & oplus& 0 =& a_{13} oplus & a_{27} \ a_{6}’ = big((& 0 &oplus& a_{12}) cdot 1big) oplus& a_{26} & oplus& 0 =& a_{12} oplus & a_{26} \ a_{5}’ = big((& 0 &oplus& a_{11}) cdot 1big) oplus& a_{25} & oplus& 0 =& a_{11} oplus & a_{25} \ a_{4}’ = big((& 0 &oplus& a_{10}) cdot 1big) oplus& a_{24} & oplus& 0 =& a_{10} oplus & a_{24} \ a_{3}’ = big((& 0 &oplus& a_{9} ) cdot 1big) oplus& a_{23} & oplus& 0 =& a_{9} oplus & a_{23} \ a_{2}’ = big((& 0 &oplus& a_{8} ) cdot 1big) oplus& a_{22} & oplus& 0 =& a_{8} oplus & a_{22} \ a_{1}’ = big((& 0 &oplus& a_{7} ) cdot 1big) oplus& a_{21} & oplus& 0 =& a_{7} oplus & a_{21} \ a_{0}’ = big((& 0 &oplus& a_{6} ) cdot 1big) oplus& a_{20} & oplus& 0 =& a_{6} oplus & a_{20} end{array}

<!–>

Mapping the toy example process to the actual LFSRs which occur in taus88 we obtain the following 3 transition matrices –>T1,T2,T3T_1, T_2, T_3 which map to of one of the three generators respectively.
We again can visualize these matrices as bitmaps, where a bright pixel corresponds to a 1 and a dark pixel corresponds to a 0.
In them we also nicely see the independence from the lowest w−kw-k<!–> bits, as they appear as empty columns in the transition matrix.

#s6–><!–>
Transition matrices [!–><!–>T1,T2,T3T_1, T_2, T_3–> for the 3 LFSRs.

Note that unlike the previously discussed LFSRs these change more than just the feedback bits directly.
This is what the parameter ss does in the linear_feedback_shift_engine constructor, which essentially is the number of steps each individual LFSR is advanced in a single step.

Inverting an LFSR

Going forward quickly is already nice.
Going backwards though, that is where the real shenanigans occur.
Since should we then somehow observe enough outputs of the RNG, we could then infer the full state just from the observed data, which then allows us to run the LFSR in a separate process to predict the future RNG calls.

Careful observation of the original construction of the LFSRs already highlights that this transition matrix needs to be invertible.
If you want to try it yourself, think about how you would step each bit backwards immediately after going one step forwards, and what the different cases are that come up.
Consider the same toy LFSR as shown above:

110000

]–>

By simply modifying the way in which the data flows, a new variant can be constructed,
which allows us to run the same LFSR but in reverse.
These changes originate from the following considerations:

  1. In the ”forwards mode” each step sets the topmost bit xn−1x_{n-1}<!–> to the previous state’s bottommost bit value –>x0x_0.
    As such, to get the value back into the lowest bit position, simply reverse that arrow.
    This means the feedback arrow now originates from the topmost bit xn−1x_{n-1} instead.
  2. If a bit is just shifted from above with no XOR between, then
    this step is reversible by just flipping the direction of the shift.
    No further modification is necessary.
  3. However, if the bit is a combination of both the upper bit and feedback bit,
    then we reverse the step by computing xi+1(t+1)=xn−1(t)⊕xi(t)x_{i+1}^{(t+1)} = x_{n-1}^{(t)} oplus {x_i}^{(t)}.
    Visually this is consistent with the first step, where we reversed the feedback bit origin,
    keeping all XORs at the same locations, feeding them with the new source value.
    This will cancel out the feedback state added in the forwards mode, and reverse the shift as though no modification happened.

In simpler terms, this amounts to us just flipping almost all arrows from the previous
LFSR diagram to obtain the following ”reverse mode” LFSR:

110000

]–>

In mathematical terms what we have just shown is that if <!–>TT–> exists which corresponds to a single forwards step,
then we can always construct another matrix T−1T^{-1} which perfectly reverses the previous step.
i.e. we have found an inverse:

T−1T=IT^{-1} T = I<!–>

As such, for any given –>TT<!–> originating from an LFSR we know that –>T−1T^{-1} exists.
This can then either be generated by the construction above, or alternative methods
such as Gaussian elimination.
Usually for Gaussian elimination we only transform the matrix into an upper triangular matrix.
However in GF(2)GF(2)<!–> without any numeric issues we can directly solve for the inverse matrix using following pseudo code:

–>

def invert(M):
  # Extend with the identity matrix on the right
  system = [M | I]
  # Iterate over all columns in the original M
  col = 0
  for row in M.num_rows:
    # Find pivot row, which hasn't previously been applied
    for pivot_row in range(row, M.num_rows):
      if system[pivot_row][col] == 1:
        break
    # Move the pivot to the current row
    system.swap_row(pivot_row, row)
    # Cancel all other rows with a 1 in the current column
    for cancel_row in range(M.num_rows):
      if system[cancel_row][col] == 1 and cancel_row != row:
        system[cancel_row] += system[row]
    # Move to the next column
    col += 1
  # Return the part which was previously the identity
  return system.I
Regarding the Invertibility of <!–>TiT_i–> in taus88

If we consider the LFSRs as given by the boost library – and the parameters
used to instantiate them – we will notice that they operate on n=32n=32 bit words,
while the actual LFSR sizes are 31, 29 and 28 respectively. This means that a
couple of bits are unaccounted for. In this case these are the least significant
bits, which will just be copies of the bits the LFSR would have previously
output – or in terms of the linear equations: the least significant bits are
linearly dependent on the higher significant bits.

This causes the matrices TiT_i<!–> to have a –>rank(Ti)<32=ntext{rank}(T_i) < 32 = n which in
turn means that they are strictly speaking not invertible.
This is also why the pictures above show some empty columns for the least significant bits.

HOWEVER: As we know the lower bits to always just be linear combinations
of the higher bits, we can reduce the transition matrices of the individual
LFSRs to 31×3131times31<!–>, –>29×2929times29<!–> and –>28×2828times28 respectively.
This restores the full rank and hence my claim that the transition matrices are invertible by construction holds.
Using these reduced matrices to compute the internal state from the
observed outputs, we can compute the remaining lower bits from the values of the
higher bits, allowing full state restoration. This however is not strictly an extra step.
When the computed state is advanced as is (e.g. with the linearly dependent bits filled to 0)
then the full state of all bits is available after a single forward step,
as a single step includes computing lower bits by the linear combination of the higher bits.

Combining multiple LFSRs

We’ve seen that we can solve the state of a single LFSR as a linear equation of
the form Ax=bAx=b where all components are computed modulo 2. Remember, however, that
the full RNG result is determined by 3 independent LFSRs whose output we XOR together.
We can model this as having 3 different states s1,s2,s3∈GF(2)ns_1, s_2, s_3 in text{GF}(2)^n<!–> each with a corresponding transition matrix –>T1,T2,T3T_1, T_2, T_3. If we then stack all
these state vectors together, we obtain a big vector s=[s1s2s3]⊤s = [s_1 quad s_2 quad s_3]^top describing the entire state at once.
For this new state vector we can again derive a transition matrix which we know to be invertible:

s(t+1)=[s1(t+1)s2(t+1)s3(t+1)]=[T1000T2000T3][s1(t)s2(t)s3(t)]=Ts(t)s^{(t+1)} = begin{bmatrix} s_1^{(t+1)} \ s_2^{(t+1)} \ s_3^{(t+1)} end{bmatrix} = begin{bmatrix} T_1 & 0 & 0 \ 0 & T_2 & 0 \ 0 & 0 & T_3 end{bmatrix} begin{bmatrix} s_1^{(t)} \ s_2^{(t)} \ s_3^{(t)} end{bmatrix} = T s^{(t)}

To obtain the final result <!–>o(t)∈GF(2)no^{(t)} in text{GF}(2)^n–> from the current
”hidden state” s(t)s^{(t)}<!–> of the RNG we can then simply calculate:

–>o(t)=[III]s(t)o^{(t)} = begin{bmatrix}I & I & Iend{bmatrix} s^{(t)}<!–>

where –>I∈GF(2)n×nI in text{GF}(2)^{n times n} is the corresponding identity matrix. If we just look at this, we
might think that the entire thing turned non-invertible again. And this would be
true, if we only look at a single output. But what happens if we step the random generator
multiple times? The first output stays as it was, the next are:

o(t+1)=[III]s(t+1)=[III][T1000T2000T3]s(t)=[T1T2T3]s(t)begin{align*} o^{(t+1)} &= begin{bmatrix}I & I & Iend{bmatrix} s^{(t+1)} \ &= begin{bmatrix}I & I & Iend{bmatrix} begin{bmatrix} T_1 & 0 & 0 \ 0 & T_2 & 0 \ 0 & 0 & T_3 end{bmatrix} s^{(t)}\ &= begin{bmatrix}T_1 & T_2 & T_3end{bmatrix}s^{(t)} end{align*}

and likewise:

<!–>o(t+2)=[T12T22T32]s(t)o^{(t+2)} = begin{bmatrix}T_1^2 & T_2^2 & T_3^2end{bmatrix}s^{(t)}–>

Thus, we realize that if we observe 3 full outputs in a row we obtain the following
system of equations:

[o(t)o(t+1)o(t+2)]=[IIIT1T2T3T12T22T32]⏟As(t)begin{bmatrix}o^{(t)} \ o^{(t+1)} \ o^{(t+2)}end{bmatrix} = underbrace{begin{bmatrix}I & I & I \ T_1 & T_2 & T_3 \ T_1^2 & T_2^2 & T_3^2end{bmatrix}}_{A} s^{(t)}

In other words: To figure out what the state of the PRNG registers was at any
time step tt, we need to observe the results of 3 consecutive calls and solve
the linear system of equations:

s(t)=A−1[o(t)o(t+1)o(t+2)]s^{(t)} = A^{-1} begin{bmatrix}o^{(t)} \ o^{(t+1)} \ o^{(t+2)}end{bmatrix}

Implemen­tation Hurdles

The earlier result of inverting the observation matrix <!–>AA–> already works. In fact,
it was the first solver implementation I built in Python.
For the observations, I used the Factorio Lua API to generate 3 consecutive
random numbers. That was enough to recover the internal state and
predict future RNG outputs; see: First recording of the method working

Before we try to implement it with only the available resources in-game, we still have to inspect two theoretical hurdles:

  1. Currently we need the result of consecutive calls.
    These might not be available to us.
  2. Moreover, the full result width i.e. all 32 bits at once of the PRNG calls are required for our observations.
    With pure game mechanics, these are not necessarily observable.

As such, let’s take a look at both of these issues, and how we can address them.

Consecutive calls

In the previous derivation we utilized the states s(t)s^{(t)}<!–>, –>s(t+1)s^{(t+1)}<!–> and –>s(t+2)s^{(t+2)} which correspond to using the full width of 3 consecutive calls.
As we are not necessarily the only system in the simulation requesting
RNG values at any given time, we need to consider a non-isolated case.
There are several ways to tackle this:

  1. If we have a method of counting the calls made between observations, we can skip
    the non observed results by generalizing the previous result to use o(t+n) = Tns(t)o^{(t+n)}~=~T^{n} s^{(t)} instead,
    where nn is the number of calls we skipped until the next measurement.
  2. Alternatively, we try to force the measurements to occur consecutively.
    This can be done by disabling all other sources in-game which can interfere
    with the measured calls, doing our necessary calls in order and computing / manipulating from there.
  3. The latter can be extended further by venturing into the realm of sub-tick mechanics.
    Every 1/60th of a second, the game performs an update step, aka a tick.
    Within this tick all the simulation mechanics run in a fixed order.
    One of the triggered mechanisms is of course the creation of the crafting results within
    all machines finishing their item crafting cycle.
    If we now can harness the order in which the machines queue the item creation events,
    placing our entropy generators in a consecutive block within this queue,
    we force the RNG calls to be gapless, ensuring proper state reconstruction can occur.

For my implementation I chose to pursue both option 2 and 3.
The former, as it does not rely on internal update orders, is the fallback method which should always
work (as long as the devs do not change the RNG away from taus88).
Meanwhile, in theory, the latter approach allows for much faster state readout and more
robustness against extraneous outside calls.
In practice, however, it appears somewhat flaky, breaking at seemingly arbitrary times.

Full result width

For our Entropy Generators we will use crafting recipes which have some
randomization in their outputs.
This has the drawback that whenever we measure such an output, we do not obtain information about the
entire PRNG call. Instead, the only thing we can measure are some simple
questions about the output, depending on the chosen method. Some examples are:

  • The number of output items. It involves randomness if either
    the recipe yields non-integer item stacks (e.g. recycling recipes,
    which return 25% of the items required to craft a single input item or the item itself in case it is a self-recycle recipe), or it is a recipe with inherently random outputs (e.g. uranium processing,
    where there is a 0.7% chance of a U-235 being produced and a 0.7% chance of not producing a U-238).
  • The quality level of the output. We can measure if it rose in level,
    and if yes by how many at once.

I’m going to focus on the first of the two methods, just observing the amount of
produced items – as this was the only source of information I had available when
I started this project. The thing to realize is that answering any of these
questions yields us only information about some of the top bits of the RNG
roll result.

Let’s stick with the example of refining uranium ore into U-235 and U-238.
For this we have 2 production results:

  • U238 occurs with 99.3% probability as a result and
  • U235 with a 0.7% chance.

To generate both outputs, the RNG is queried twice for a single crafting cycle.
Once per item to generate two consecutive RNG calls.
Because these probabilities are very extreme, we gain important knowledge whenever the low-probability event occurs.
The resulting item gets generated if the respective inequality holds,
where ri∈[0,232−1]r_i in [0, 2^{32}-1]<!–> is the computed RNG roll:

–>r1≤⌊0.993⋅232⌋=  4 264 902 52410=11111110 00110101 00111111 011111002r2≤⌊0.007⋅232⌋=  30 064 77110=00000001 11001010 11000000 100000112begin{alignat*}{} r_1 leq leftlfloor 0.993 cdot 2^{32} rightrfloor &=;& 4,264,902,524_{10} &&= 11111110 00110101 00111111 01111100_2\ r_2 leq leftlfloor 0.007 cdot 2^{32} rightrfloor &=;& 30,064,771_{10} &&= 00000001 11001010 11000000 10000011_2 end{alignat*}

In particular:

  • If U238 was not generated, i.e. <!–>r1r_1–> is greater than the given threshold, we know that the first 7 bits are 1.
  • Likewise if U235 was generated, then the first 7 bits of <!–>r2r_2–> have to be 0.

Otherwise, we have no meaningful information about the rolled bits.

Cool! But which recipe will yield us the highest amount of information each time it completes?
The more information we obtain with a single crafting cycle, the fewer crafts we require and the faster and more efficient we can determine the internal state.

Like we saw above, the only information we can observe is determined by the topmost bits, and whether we obtained the item or not.
If we now estimate that the rolls are actually evenly distributed, we can compute the expected amount of information gained with each roll and observation:

0.007⋅7 bit⏟no U238+0.993⋅0 bit⏟U238=0.049 bit0.993⋅0 bit⏟no U235+0.007⋅7 bit⏟U235=0.049 bitbegin{align*} underbrace{0.007 cdot 7,text{bit}}_{text{no U238}} + underbrace{0.993 cdot 0,text{bit}}_{text{U238}} &= 0.049,text{bit}\ underbrace{0.993 cdot 0,text{bit}}_{text{no U235}} + underbrace{0.007 cdot 7,text{bit}}_{text{U235}} &= 0.049,text{bit} end{align*}

This means we can expect a total of <!–>0.0980.098–> bits per successful crafting cycle.
If we study the above pattern a bit longer, we might notice that the number of leading bits
obtained in each positive case (i.e. where the probability is 0.7%0.7%<!–>) is –>⌊−log⁡2(p)⌋lfloor-log_2(p)rfloor<!–> where –>pp is the probability of the event occurring.
A similar thing holds for p>0.5p > 0.5 where the result is instead leading 0 bits observed.
This allows us to compute the expected number of bits we measure for an event with probability pp.
Overall, it can be written as:

E[bits]=p⌊−log⁡2(p)⌋+(1−p)⌊−log⁡2(1−p)⌋mathbb{E}[text{bits}] = p leftlfloor -log_2(p) rightrfloor + (1-p) leftlfloor -log_2(1-p) rightrfloor

This function is also shown below.
Its plot indicates the following:

  • The event which has the highest expected number of observed bits is situated at the even 50% split.
    At that point we can in fact always observe the most significant bit.
  • While events closer to 0/1 allow us to infer more bits whenever they succeed,
    the likelihood of the events occurring diminishes too fast,
    decreasing the total number of expected observed bits per event instead.

2025-10-07T19:10:49.360938
image/svg+xml

Matplotlib v3.8.4, https://matplotlib.org/

0

312

116

18

14

12

3312

1156

78

34

1

Event likelihood 
p

0.0

0.2

0.4

0.6

0.8

1.0

Expected #Bits ()
Ep

Expected Bits observed for Events of Probability 
p

What we just calculated can be seen as a discretized version of Shannon entropy.
As such, we have a measure applicable to all available recipes allowing us to identify those which yield the largest amount of information per craft.
By extracting the relevant recipe data from the raw game dump4 we can programmatically compute the entropy for each recipe.

Doing so yields a table of recipes with their corresponding expected bits of information per craft, alongside how long each craft takes.
The following highlights a small selection of recipes with random outputs, for the table containing all recipes with random outputs see here.

Recipe Bits per Craft Craft­ing Time Items Returned
–> 4.5 0.5

3.75x Steel Plate
2.5x Iron Gear Wheel
2.5x Stone Brick
2.5x Electronic Circuit
2.5x Pipe

–> 3 0.03125

1.25x Electronic Circuit
1.5x Iron Plate
1.5x Iron Stick
0.75x Steel Plate

–> 2.05 0.2

1x Iron Gear Wheel (20%)
1x Solid Fuel (7%)
1x Concrete (6%)
1x Ice (5%)
1x Steel Plate (4%)
1x Battery (4%)
1x Stone (4%)
1x Advanced Circuit (3%)
1x Copper Cable (3%)
1x Processing Unit (2%)
1x Low Density Structure (1%)
1x Holmium Ore (1%)

–> 2 0.03125

0.5x Electronic Circuit
0.5x Iron Gear Wheel

–> 1 0.03125

0.5x Iron Plate

–> 0.5 0.2

1x Iron Plate (25%)

–> 0.1 1

1x Yumako Seed (2%)
2x Yumako Mash

–> 0.098 12

1x Uranium 235 (0.7%)
1x Uranium 238 (99.3%)

Excerpt of recipes with random results producing bit observations.

As we can see, there are waaay better recipes for extracting information from the game.
One might think that scrap recycling would yield a lot of information due to the many different items which can be produced.
Yet with a total entropy of 2.052.05<!–> it is only slightly above recipes like recycling repair packs which have an entropy of –>22.
This is due to the fact that recycling repair packs (and similar recipes with ingredient count =4n+2= 4n+2)
yield exactly one bit of information of the RNG output in either case,
as it creates a perfect 50/5050 / 50 split on whether the additional item will be created or not.
There are obviously alternative recipes such as the oil refinery which has an entropy of 4.54.5.
These, however, are also quite a bit more expensive and slower than repair pack recycling.

As such, I chose to implement the state readout using the repair pack recycling method, as repair packs are cheap, fast to craft, and unlocked early in-game.

Actual Implemen­tation

To actually implement the reversal and manipulation of the RNG, I’ve split the computation into the following steps:

  1. Sampling the current RNG through observations,
  2. Computing the current internal RNG state,
  3. Predicting the future internal states,
  4. Calculating corresponding quality levels for each future call, and finally
  5. Making use of the predicted levels with some adapters.

As already alluded to in Inverting an LFSR, we will need to compute a matrix-vector product for both of these steps.
Now the question is how do we get the matrices, and where do we get the vectors from?

Sampling the RNG

The current state will be computed from observations made when recycling repair packs.
Each recycling operation will yield exactly 2 bits of information, 1 for each resulting item.
This in turn means that we require 88/2=4488/2=44 recycling operations to have enough information to fully reconstruct the state.
As each result provides us with exactly one bit of information – the topmost bit of the RNG call – the 88 observations can be written as:

o(t),o(t+1),…,o(t+87)∈GF(2)o^{(t)}, o^{(t+1)}, dots, o^{(t+87)} in text{GF}(2)<!–>

A single unit measuring –>o(t+2k)o^{(t+2k)}<!–> and –>o(t+2k+1)o^{(t+2k+1)} may look as follows:

Screenshot of a single entropy measurement unit.
A single entropy measurement unit.

It performs the following steps:

  1. The inserter will move exactly 1 repair pack into the recycler.
  2. The recycler will recycle the item, and upon completion query the RNG for 2 new integers, determining whether extra items (either 0 or 1) are produced.
  3. Depending on whether the items are produced or not, they are placed into the provider chest.
    This chest is set to read the contents, providing the observation to the red wire.
  4. These observations directly correspond to the topmost observed bits o(t+k)o^{(t+k)} due to the 50% chance of output.
    Further processing occurs through the decider combinators below.
  5. Before the next call can occur, we clear the provider chest by making use of the trash unrequested option, alongside the enable/disable signal we can send over the green wire connected to it, which temporarily pauses the auto trashing behavior when the requestor chest is disabled. Otherwise it requests no items.

The above unit will therefore always provide us with 2 bits of information.
To reconstruct the full state quickly, we copy this unit 44 times, yielding a total of 88 bits.
This then looks like:

All entropy units joined together.
All entropy units together.

Of note here is the manner in which the individual units are queried.
There are 2 approaches:

  1. Either each is triggered with a 1 tick delay, ensuring that they are always queried in the same order.
    This, however, requires us to not have any other RNG running in the meantime,
    as RNG calls which occur in between will mess up the expected ordering.
  2. Alternatively, we can use same tick shenanigans.
    By splitting the red wire connecting the inserters with a 1 tick delay combinator in front of every inserter, this can be achieved.
    Connecting the inputs to the delay first creates a shared circuit network.
    Then sequentially connecting all outputs of the delays to the corresponding inserters will create a standalone network for each inserter.
    As the game needs to update the networks in some manner, I bank on the fact that
    it will iterate through the list ordered by the network ID.
    This ensures that the RNG calls all happen in the same tick, with no ticks interfering.
    Note that the wire construction can also be done using a 2-stage blueprint.
Note – Regarding the Same Tick Behavior

While the same tick stuff seems to work in practice, I have not actually confirmed that this is how the game works under the hood.
It can be brittle at times, as it seems to arbitrarily break at random times.
In those cases, simply reconstructing the wires allows it to work again.

Determining the current state

Each unit produces only single bit observations as a result, so we need to apply a similar strategy as we did before.
This time though, we only use the first row of the linear equation defined above for computing observations from the state:

P1[T1k000T2k000T3k]⏟Tk[s1(t)s2(t)s3(t)]=o(t+k)P_1 underbrace{begin{bmatrix}T_1^k & 0 & 0 \ 0 & T_2^k & 0 \ 0 & 0 & T_3^kend{bmatrix}}_{T^k} begin{bmatrix}s_1^{(t)} \ s_2^{(t)} \ s_3^{(t)}end{bmatrix} = o^{(t+k)}

where <!–>P1=[p1i]∈GF(2)1×96P_1 = [p_{1i}] in GF(2)^{1 times 96}–> is the first row of the matrix <!–>P=[III]P=begin{bmatrix}I&I&Iend{bmatrix}–>.
If we now let Ak=P1Tk∈GF(2)1×96A_k = P_1 T^k in text{GF}(2)^{1 times 96}<!–> be the matrix which denotes performing –>kk RNG steps followed by an observation of the topmost bit,
then we can write the observations we gather above to follow the subsequent equation:

[ A0 ⋮ A87 ][s1(t)s2(t)s3(t)]=As(t)=[o(t)⋮o(t+87)]begin{bmatrix} rule[3pt]{5mm}{0.2pt} A_0 rule[3pt]{5mm}{0.2pt}\ vdots\ rule[3pt]{5mm}{0.2pt},A_{87},rule[3pt]{5mm}{0.2pt} end{bmatrix} begin{bmatrix}s_1^{(t)} \ s_2^{(t)} \ s_3^{(t)}end{bmatrix} = A s^{(t)} = begin{bmatrix}o^{(t)} \ vdots \ o^{(t+87)}end{bmatrix}

Now, as we consume 88 calls when we perform our observation, it would be beneficial to instead directly calculate the state the RNG will be in after our observation, rather than when we started.
This can be achieved by making use of the inverse transition matrix T−1T^{-1}<!–> which causes some shift in the time index, creating a new matrix –>A~tilde A<!–>:

–>As(t)=A T−88s(t+88)=A~s(t+88)A s^{(t)} = A, T^{-88} s^{(t+88)} = tilde A s^{(t+88)}

Shifting the time <!–>tt–> to be relative to the next RNG call by means of substituting <!–>t~=t+88tilde t = t + 88–> we then arrive at the equation:

<!–>A~s(t~)=o(t~−88,…,t~−1)tilde A s^{(tilde t)}=o^{(tilde t-88,dots,tilde t-1)}–>

This can be read as us computing the next internal RNG state from the previously done observations.
Now since there are only 88 bits which are actually linearly independent, we cannot compute a full inverse.
However, a pseudo-inverse will suffice.
Especially since we are interested in the states after – for which the lowest bits are entirely described by the most significant bits.
This means that even a single step forward will deterministically set those previously unknown bits, so we are all fine.

The really neat thing about this entire endeavor is that matrix AA<!–> and similarly –>A~tilde A do not depend on any dynamic state.
As such, A~−1∈GF(2)88×96tilde A^{-1} in text{GF}(2)^{88 times 96}<!–> can be precomputed in Python and subsequently used in Factorio.

Hence we only need to implement a matrix-vector multiply in –>GF(2)text{GF}(2) in-game.
To do so, remember that any matrix-vector product can be seen as a weighted sum of the matrix columns weighted by the entries in the vector:

Ax=∑k=1nxk[A]k=x1a→1+⋯+xna→nAx = sum_{k=1}^n x_k [A]_k = x_1 overset{rightarrow}{a}_1 + dots + x_n overset{rightarrow}{a}_n<!–>

In this case, as we are performing our computations in –>GF(2)GF(2)<!–>, each entry is either 0 or 1, meaning the multiply can be represented with a simple conditional, while the sum is substituted with an XOR over all the weighted vectors:

–>s(t+88)=⨁k=087o(t+k)⋅[A~−1]ks^{(t+88)} = bigoplus_{k=0}^{87} o^{(t+k)} cdot [tilde A^{-1}]_k<!–>

Here –>[A~−1]k[tilde A^{-1}]_k<!–> is the –>kk<!–>-th column of –>A~−1tilde A^{-1}<!–> while –>o(t+k)o^{(t+k)}<!–> is the –>kk-th bit observation performed.
In practice this equation is computed in 2 parts.

First, each measurement unit computes a pointwise scalar-vector multiplication of o(t+k)∈{0,1}o^{(t+k)} in {0, 1}<!–> and vector –>[A~−1]k[tilde A^{-1}]_k.
This is done via the decider combinator mentioned above doing “further processing”.
If the resulting item was not observed (=0= 0<!–>) then the roll was above the threshold and we have –>o(t+k)=1o^{(t+k)} = 1, meaning this column needs to be accumulated otherwise it is not.
The vector [A~−1]k[tilde A^{-1}]_k is stored in the constant outputs of the decider combinator, where only the bits which are 1 are actually output.
As such the vectors are represented by 96 different signals.

A single decider combinator computing a single scalar-vector multiplication.
A single decider combinator computes a single scalar-vector multiplication.

Finally we require the XOR of all these vectors to obtain the final state.
This can be achieved via implicit addition.5 Summing the values of a single signal and extracting only the last bit of this sum is equivalent to taking the XOR over all of them.
As such, by wiring all decider outputs together, a single arithmetic combinator can perform the bit extraction by ANDing the pointwise sums with 1.

As each signal now corresponds to a single bit of the 3 32-bit LFSR states,
we can make use of a set of decider combinators to sum up all the corresponding bits of each active signal.
Thus we have 3 signals, each containing the current state of the game’s RNG.

The final XOR sum of all the scalar-vector multiplications.
The final XOR reduction.

Looking into the future

We perform a similar action to compute the future states of the individual LFSRs.
However, as all LFSR states require fewer than 32 bits, we can store the lookup table in a more compact fashion.
For each LFSR we need to compute the following:

Tks(t)=s(t+k)T^ks^{(t)} = s^{(t+k)}<!–>

This is computed for –>k=1,2,…,Nk=1,2,dots,N<!–>, where –>TT<!–> and –>ss are different between the 3 sub LFSRs.
From this we can again rewrite these matrix-vector multiplications as:

s(t+k)=⨁j=132sj(t)⋅[Tk]js^{(t+k)} = bigoplus_{j=1}^{32} s^{(t)}_j cdot [T^k]_j<!–>

where –>[Tk]j[T^k]_j<!–> is the –>jj<!–>-th column of –>TkT^k<!–>. These columns can be stored as 32 bit integers, as –>Tk∈GF(2)32×32T^k in GF(2)^{32 times 32}.
Hence the skip-ahead equation can be implemented in parallel for all kk<!–> steps as follows:

  1. Split the packed state –>s(t)s^{(t)}<!–> into its individual bits –>sj(t)s_j^{(t)}.
    Represent these as individual signals again (arithmetic combinator on the left).
  2. The pointwise scalar vector multiplication sj(t)s_j^{(t)} with all the different
    vectors [Tk]j[T^k]_j<!–> will either include all the vectors in the corresponding –>kk-th state or not,
    thus we can implement this via a decider combinator again.
    This time, however, I store the constants in a separate constant combinator – it can output more signals at once.
    I opted to predict 1000 forward steps in parallel.
The same matrix column from different T^k transition matrices.
The same column from different <!–>TkT^k–><!–> transition matrices.
  1. Now we have 32 nets each full with –>kk 32 bit wide values which need to be XORed together.
    Unlike before we cannot utilize the implicit addition here,
    as the vectors are now not represented by 32 different signals but instead via a single 32 bit signal value.
    Thus we have to use more arithmetic combinators.
    I’ve opted to use a binary tree to pairwise XOR sets of vectors together,
    as this is a known fast reduction strategy for prefix sums (which this is).
Parallel look-ahead for a single LFSR
Parallel look-ahead for a single LFSR.

Quality prediction

Now that we have the next RNG call results before the actual in-game calls
happen, we need to make them usable for our purpose. This basically means implementing
some form of the rollQuality function from the game.
A reverse-engineered version of the function can be seen below, implemented in pseudo-C++:

// Fixedpoint value of the module effect from -32.768 to 32.767
// e.g. 10% quality would be a value of 100
typedef EffectValue int16_t;

// Stub of relevant quality prototype fields
struct QualityPrototype {
    ID<QualityPrototype, uint8_t> id;
    ID<QualityPrototype, uint8_t> next;
    double nextProbability;
}

// Mapping from the ID<...> to QualityPrototype
PrototypeList<QualityPrototype>::indexToPrototype;

// Function which determines crafting result quality
ID<QualityPrototype, uint8_t>* QualityPrototype::rollQuality(
  ID<QualityPrototype, uint8_t> qualityID,
  EffectValue bonus, 
  RandomGenerator* generator,
  IDIndexedData<uint8_t, ID<QualityPrototype, uint8_t>> 
    const* unlockedQualities
) {
  // If no bonus, do early return -> no RNG call!
  if (bonus == 0)
    return qualityID.copy();
  
  QualityPrototype* quality = indexToPrototype[qualityID];

  // Roll the RNG exactly once
  double roll = RandomGenerator::uniformDouble(generator);

  // If roll < threshold we upgrade to the next quality
  double threshold = (double)((float)(bonus) / 100f);

  // Find highest quality which beats the threshold
  uint8_t nextIndex;
  while ((nextIndex = quality->next.id.index) != 0) {
    if (!unlockedQualities->data[nextIndex])
      break;

    // Scale by next upgrade probability, base game = 0.1
    threshold *= quality->nextProbability;
    if (roll > threshold)
      break;

    quality = indexToPrototype[nextIndex];
  }
  return quality->id.copy();
}

If we take a look at how the function is implemented in the game we can see that
it basically just takes the RNG roll and compares it against some thresholds. It
stops as soon as it finds a threshold which is no longer beaten by the roll.

This means that each craft which involves quality rolls will take exactly 1 RNG call.
And for each call we can compute the expected quality level by just comparing against
all the thresholds, which stay constant during the game. Thus they can be precomputed
in-game with some arithmetic combinators.

Computing the quality levels from RNG states in-game.
Computing the quality levels from RNG states in-game.

The calculator below shows the required threshold for each quality level, as well
as the expected amount of each quality level for a given bonus.
Note that when the threshold exceeds 232−12^{32}-1, i.e. the uint32_t maximum value,
we always upgrade, which is indicated by placing the thresholds in brackets.

]–>

]–>

Result Probability uint32_t Threshold
<!–> 75.200% 1,065,151,889
–> 22.320% 106,515,188
<!–> 2.232% 10,651,518
–> 0.223% 1,065,151
<!–> 0.025% 106,515

Adapters

Yippee, we can now compute the relevant RNG outcomes before the corresponding calls even occur in-game.
Now the question is what can we do with that?
I have by now experimented with several – what I call – adapters,
which take in these predictions, and do some funny stuff with them.

Beginning with the first that I’ve implemented:

Predicts quality levels before the craft.

It takes the sequence of future output qualities, and displays them next to the assembler,
similar to the “Next Up Pieces” queue in Tetris.
Each time a craft is completed, it advances the window into the future outputs by one,
keeping the display relevant at all times.

The next one was the following:

Selects assembler by next up quality.

This one takes the same list, but instead assigns only a single crafter to fabricate the next item.
The assembler which is selected depends on the next output quality.
In turn this leads to the 5 assemblers outputting the items in a sorted fashion,
where each belt only ever carries a single type of quality, ordered from left to right in increasing quality.

Finally, the goal that I’ve been interested in from the get-go – and the hook already seen at the start of this post:

Full automation of legendary items.

This method encapsulates the entire prediction and crafting loop into a closed system,
which can do the entire thing (predict and craft) autonomously.
We have seen how we can compute the current state, and predict the next NN states.
But how does this let us force RNG values of our desire?

The answer is the simplest of all: It doesn’t. At least not directly.

Instead, we can make use of the fact that the sequence of the RNG outputs is deterministic.
By consuming the bad RNG calls which would need to occur before our desired one, we can “force” the RNG to next output a desired value – such as one which upon use in the quality rolling code immediately upgrades from common to legendary quality.
For this we need something to consume the RNG calls.

One automatable aspect is again the creation of partial item stacks.
Unlike before however, we do not care about observing the output of these crafts, but only the number of calls each crafting cycle makes.
Additionally quick crafting cycles lead us to maximize the number of calls per second.

As luck would have it we already saw a recipe which consumes many calls and has a tremendously fast crafting speed: Scrap Recycling.
It takes 12 calls per completed craft, with a base speed of 0.2 seconds.
Note that this holds for the number of completed crafts, i.e. crafts completed by productivity count as well.
While this for one means that we scale the calls consumption rate of each recycler with the infinite scrap recycling productivity,
this simultaneously also requires additional handling of the productivity.

An array of 10 recyclers controlled by combinators ingesting both gears and scrap to skip rng calls.
Consuming scrap and gears to skip bad calls.

Using scrap adds complexity due to the following considerations:

  • We have multiple recyclers, so we need to figure out how much scrap each gets, and how many get an additional one?
    The last part helps reduce the number of items of the last stage.
  • Recycling scrap steps 12×12 times rng calls per craft which is not fine grained enough to resolve an exact state.
    The remaining number of necessary calls are padded with crafts only eating a single call each.
    Here these are gears.
  • We also need to consider crafts completed due to scrap recycling productivity which occasionally leads to multiple crafts finishing for a single input item.
    This causes the RNG to be queried for multiple recipe results, i.e. a multiple of the 12 calls.
  • Each recycler is started with at least 1 gear before scrap to reset the previous productivity progress allowing us to avoid tracking that as well.
The exact function implemented

To implement the exact function I first wrote some code to simulate it, then let AI derive a closed formula which I could simplify and translate to combinators.
The relevant notebook can be found here.
In short: we require a function F(R,P,C)=(Nbase,Rextra,Crem)F(R, P, C) = (N_text{base}, R_text{extra}, C_text{rem})<!–> which computes given –>RR<!–> recyclers and productivity level –>PP<!–> for any desired number of calls –>CC,
how many scrap NbaseN_text{base}<!–> each recycler gets where –>RextraR_text{extra}<!–> many get an additional one while –>CremC_text{rem}<!–> is the number of additional gears consumed.

The equations can be written as:

–>Ktarget=⌊C12R⌋Nbase=⌊10Ktarget+910+P⌋Kbase=⌊Nbase(10+P)10⌋ΔK=⌊(Nbase+1)(10+P)10⌋−KbaseCgap=C−12RKbaseRextra=⌊Cgap12ΔK⌋Crem=Cgap mod (12ΔK)begin{align*} K_{text{target}} &= leftlfloorfrac{C}{12R}rightrfloor \ N_{text{base}} &= leftlfloorfrac{10K_{text{target}}+9}{10+P}rightrfloor \ K_{text{base}} &= leftlfloorfrac{N_{text{base}}(10+P)}{10}rightrfloor \ Delta K &= leftlfloorfrac{(N_{text{base}}+1)(10+P)}{10}rightrfloor – K_{text{base}} \ C_{text{gap}} &= C – 12R K_{text{base}} \ R_{text{extra}} &= leftlfloorfrac{C_{text{gap}}}{12Delta K}rightrfloor \ C_{text{rem}} &= C_{text{gap}} bmod (12Delta K) end{align*}

where <!–>KK–> are craft completions, <!–>ΔKDelta K–> the step size if one more craft completes, <!–>RR–> the number of recyclers, and <!–>Cgap/CremC_text{gap} / C_text{rem}–> the number of RNG calls.

There are two additional considerations to make.
First of all, while we could increase the amount of calls simultaneously predicted in the forwards pass,
this will bloat the save / blueprint and does not scale well past a couple thousand calls per pass.
Instead, we can simply feed the output of the last computed RNG states back into the RNG forwarder in a feedback loop.

The two combinators feeding the output from the simulation back towards the simulator input.
Feeding the forward simulated RNG registers (right) back into the simulator (top).

The first 2 adapters could simply ingest the output of the quality prediction module.
To somewhat decouple the prediction and skipping ahead, I decided to decouple the 2 systems, by buffering known good offsets.

For this, as before I compute the threshold which needs to be passed, and now unlike before: Filter out the relevant call indices / offsets and only store those instead.
This is done by remapping the passing signals into a sequence of new signals, one for each good offset, as seen in the image below.
The remapping is done at 1 signal per tick.
Once all passing signals of this iteration are consumed, the above feedback loop is triggered to advance to the next 1000 steps.

A view into the buffer combinator filled with filtered good quality call offsets.
A buffer stores all “good” RNG offsets. Prediction state after about a minute.

As this buffer stores absolute offsets from the first time we measured the RNG, we need to compute the number of calls to actually skip to arrive at the next index.
For this we fetch the current and the next index from the buffer and subtract their offsets, yielding the delta.
This delta is then what is actually fed towards the skipper.
The next number of steps is fetched only after the skipper has finished with the current cycle of skipping and creating the next item.

Two selector combinators indexing into the buffer, computing the difference between two adjacent calls.
Computing the number of calls to skip subtracting two adjacent buffered offsets.

Lastly, we have the issue that the simulated registers may diverge from the actual game RNG state – for instance if any other process has consumed a RNG call unbeknownst to us.
While we can not prevent such intermittent calls, we can at least detect them.
In this instance our predictions will diverge from the actually observable crafting results.
As such, when too many results (2) differed from our expected quality, we can simply restart the machine automatically, triggering another full state observation and forwarding future states from there on.

The assembler output inserter connected to the thresholding circuit on the right.
Divergence detector and auto resetter.

Below we now see the entire machine in its full glory.
I’ve highlighted the different modules corresponding to the individual segments we constructed previously.
The general data flow can be read as starting from the bottom left (the assembler) going clockwise:
Readout (lime), prediction (blue), filtering (yellow), buffering (purple), skipping (black).

The final overarching view of the fully automated crafter.

Limitations

Now to the important part, that you may be wondering about:

Sweet! Can I use this to now do <insert RNG manipulation target> in my save-game?

The short answer: Very unlikely.

But why?
It’s not that I want to keep this tech for my self.
In fact, here is the world download, a blueprint string and the relevant cleaned up python code for you to play with.
No, it rather has to do with the way that Factorio currently handles their random generators.

For this we can take another look into the game binary.
After browsing a bit we encounter the Map object, which among other things as references to the individual surfaces (i.e. the different layers of the world, like the planets Nauvis, Fulgora, and so on).
The RNG states however are not stored per surface, but rather globally for the entire map.
And notice that I am speaking of states (i.e. plural) as there are actually multiple RNGs in game,
each responsible for some of the game’s logic.

The Map in particular stores the following six RNGs (names from the pdb).
Try to guess what each one is responsible for:

  • Map.aiRandomGenerator
  • Map.entitiesRandomGenerator
  • Map.generalRandomGenerator
  • Map.mapRandomGenerator
  • Map.triggerRandomGenerator
  • Map.unsafeRandom

To be honest, I still don’t know what some of them do, I was only interested in the one relevant to item crafting procedures.

Additional RNGs

There are more generators in the binary, the full list I have currently found in addition to the ones mentioned above is:

  • GlobalContext.randomGenerator
  • LightningMeshGenerator.random
  • SelectorCombinatorControlBehavior.random
  • SoundRandomizer.randomGenerator
  • SpacePlatform.asteroidsRandomGenerator

To try and track all the RNGs and what potential call paths are which lead to a random number call,
I made use of binary ninja and its python scripting to extract a call tree originating from the RandomNumberGenerator::getInt call and its derivatives.
The corresponding scripts for scraping are available here.

Here we can already see a saving grace for RNG manipulation.
Not all random effects are handled by the same RNG, and thus we can at least isolate some of them from the rest of the game logic.
For instance, the RNG responsible for the biter spawning and pathing is separate from the RNG we are interested in.

In particular this is the generalRandomGenerator, which is responsible for the item creation.
As its name implies, it is the general random generator, meaning there is still some overlap with other game logic.
Either completely unrelated to item creation, or through other recipes which also have probabilistic outputs.
This is exactly the issue with the non-isolated cases I talked about previously.

First a non-exhaustive list, of instances unrelated to quality rolling:

  • Floor tiles changing via the MapGenerator::clearEntitiesAndSetTile function, which randomly chooses from variants. This can happen due to manual edits via the editor, through the freezing logic changing tiles, tile ghosts being constructed, or a space platform building some flooring.
  • Name randomization of entities like labs and train stops,
  • Player manually mining ore (particle spawning) or walking over dusty ground (creating dust particles),
  • Particles in general i.e. ParticlePrototype::getRandomVariation and Smoke constructor,
  • Selector combinators initialize their own RNG with a random seed,
  • Mining drills with a single ore tile running out randomly shuffle all remaining tiles they mine,
  • Lightning strikes on Fulgora,
  • Spidertron leg placements when walking

Secondly, all machines requiring any sort of randomness share the same RNG.
Any time another (by my machine unexpected) recipe uses the RNG
– for instance your scrap recycling line on Fulgora, uranium processing on Nauvis, or any other quality rolling –
then the state of the RNG will change, causing the predictions of my machine to diverge from the actual game state,
making it impossible to reliably manipulate the RNG towards any specific goal.
Moreover, consider that in a game about automation one usually scales up to produce large volumes of items,
leading to potentially thousands of calls occurring in just a second, outpacing my capabilities of precomputing the RNG fast enough.

While it might be possible to account for all / many of the randomness sources above,
e.g. by dynamically disabling any other production lines doing random calling using for example a logistics group,
and waiting for daytime on Fulgora it may be possible to apply this in an actual game save,
it should be taken into account from the get-go, rather than being retrofitted into an existing save.

If anyone wants to give it a shot, feel free to try and let me know how/if it works out.

Factorio 2.1

With the last major update to Factorio, the way the RNG is used has changed.
While they still use taus88 as the underlying generator, they have fundamentally rewritten major parts of the item creation logic, reusing a single call for multiple item outputs.

Additionally from what I can currently tell, now every single crafting operation uses the RNG, even if it is deterministic,
to fill the shared call field in case anything later on will require it.
This means that no other machine can run in parallel, as it will always interfere with the RNG state.

Moreover, due to the sharing of the rolls, using scrap recycling to skip the RNG forwards is no longer useful, as the RNG is queried the same amount for any recycled item, and dealing with scrap recycling productivity and its many outputs increases the overhead a bunch. As such this could be replaced with the recycling of any other simpler/cheaper item instead – at least it’s no longer directly bound to Fulgora.

Lastly, while the update is still in the experimental branches I have been somewhat on a rollercoaster ride seeing different changes to the RNG system.
At one point, the Map::generalRandomGenerator was used to control the FISH motion.
This of course would be a huge problem, as it meant that any fish on the map generated an unknown number of RNG calls, leading to it continuously desynchronizing the RNG state from the predictions without any feasible way to account for it shy of removing all fish from the map (without generating any new chunks with new fish).
Thankfully, this was changed in a later update (its gone in version 2.1.13) with a new seventh RNG on the Map object, called Map::fishRandomGenerator. Guess what its job is 🙂

However its not all bad.
For instance, with the forced move away from scrap recycling, and the addition of universe wide signals (allowing us to send when Fulgora lightning storms start to other planets) we are no longer bound to any specific planet, and could instead build the manipulator on Vulcanus, gobbling up however many resources the skipping now requires, sending a signal to any other planet to craft local legendaries when the RNG is in the right state.

For now, I will leave it at that, as the game may further change while it is still in experimental,
so any updates to the cracker might just get broken by the next update without notice.

Conclusion

This marks the completion of a two+ year project, finally reaching the fully autonomous gamblen’t I wanted from the beginning.
Though to be fair, most of the latter part was me procrastinating on writing and publishing this post.
In the meantime (while I was dragging my feet), others have also looked into the RNG, who I’ll link here for reference:

  • @kovaxis in the Factorio forums, reaching and stopping at a similar point as I did initially, where the state is computed with an external python script from some in-game observations.
  • @d4s_over_dt4 on the Factorio discord, who built a combinator circuit to compute the RNG state, read out from 3 placed selector combinators (which each query the general RNG to seed each combinators own state), without further followup integration or verification.

Boy there were quite some tangents along the way which did not even make it into this post,
as its long enough already, such as me partially recreating cnide with improved handling for subnets just for documentation and simulation purposes as with syntax highlighting in vscode,
or the first implementation attempt where I did all the matrix math in game,
including the creation of the matrix and Gaussian elimination of said matrix.6

Thanks go out towards

  • @earthcomputer and co. who unknowingly inspired this project, as their “Mess Detector” reads out Minecrafts RNG state with only in-game mechanics,
  • @redruin1 with their factorio-draftsman library allowing for easy procedural blueprint creation,
  • the Binary Ninja team for a decompiler which does not shit itself7 actually works when encountering the relatively chonky factorio.exe,
  • and everyone close to me who bullied me into finally finishing this writeup after hearing me rambling about RNGs for the past 2 years 🙂

  1. See Factorio Friday Facts #375↩
  2. Trivially breakable for security researchers seems to mean that they know how to break them, hence its easy, even if still takes some setup to understand.↩
  3. This part of the project is where I realized that structures like Field, Ring, Monoid are basically the mathematical variant of programming using generics, where we can write algorithms which work for any type which satisfies the necessary properties.
    And just like when programming anything, the naming often sucks 🙂 But somehow sticks… ↩
  4. Factorio actually provides a way to dump the data by running with command line arguments.
    One of these is --dump-data which outputs a JSON file containing the processed prototypes as they are loaded in-game.
    This can then be parsed for further processing, such as extracting the various recipes.
    However, I just used the data bundled with draftsman.↩
  5. In Factorio, when multiple devices write a signal to the wire,
    then the resulting signal value on the wire is the sum of all the individual signals.
    This means any summation can be done implicitly.↩
  6. The Gaussian elimination in game is actually also included in the world download, with some instructions on how to use it.↩
  7. Unlike ghidra, where even 64 GB RAM were not enough to successfully decompile the game exe.↩

–>


Source: Hacker News

Reverse-engineered Jev-like model

Jevlike

Train a small model that chooses among a changing list of text options.

A Jev-like model takes a piece of text and a list of N text options. It returns one probability for each option. It does this in one pass instead of writing an answer word by word. Jev is TypeSafe’s commercial model for this kind of task. TypeSafe has not published its design. This repository is an independent starter model with the same input and output shape.

Demo

The same option-attention head can score controller buttons from image patches. This ten-second film joins two selected five-second windows: live deadly_corridor combat on the seven Doom buttons, then a chess controller walking to and playing moves with five keys. The diagram shows the tensors used for each decision. The Doom window came from the supplied joint checkpoint, which averaged 0.60 kills and -97.50 reward across its ten recorded episodes. The chess window came from the stronger chess-only checkpoint, which scored 4 wins, 46 draws and 0 losses in 50 sampled games against a random mover, but 0 wins, 2 draws and 48 losses against Stockfish level 0. The windows were selected for activity and are not typical-play or competence claims.

Install the game extras and record a fresh 640 by 480 Doom trace from the released joint checkpoint:

uv pip install -e '.[games]'
python examples/doom/play.py examples/checkpoints/joint-imitation.pt --episodes 10 --game-seconds 35.3 --device cpu --capture-resolution 640x480 --output runs/doom.mp4 --trace runs/doom-trace.json

Render the trace in the same visual layout. This writes a silent film because the author-owned soundtrack source is not part of the repository.

(cd examples/film && npm install && npx playwright install chromium)
examples/film/make-film.sh runs/doom-trace.json runs/doom-film.mp4 10

The release includes the Doom example, the chess example, the single-game checkpoints and the shared 12-option checkpoint. Both games import the visual scorer from jevlike.vision; there is no second model copy in either example.

Architecture

Each option becomes a query vector, which is a short list of numbers representing its text. The query assigns attention weights to the context tokens. Those weights make one context vector for that option. A shared dot product turns each option and context pair into one score. A softmax, which converts scores into probabilities that sum to one, runs across the options.

Each option queries the context, receives an attended context vector, and becomes one probability.

The default encoder learns byte embeddings from scratch. An encoder is the part that turns text into vectors. The optional Hugging Face path uses a frozen pretrained encoder, whose existing weights stay fixed while the small scorer learns.

Data format

Use one JSON object per line:

{"context":"The customer needs a refund.","options":["refund","sales","technical support"],"label":0}

label is the zero-based index of the correct option. Each row may have a different number of options, with a minimum of two.

Quickstart

Run these commands from the repository root. They create local synthetic data, train on it, evaluate the saved model and score one new menu.

uv venv
source .venv/bin/activate
uv pip install -e '.[dev]'

jevlike-data synthetic --output data/synthetic
jevlike-train data/synthetic/train.jsonl 
  --validation data/synthetic/validation.jsonl 
  --output runs/synthetic.pt
jevlike-eval runs/synthetic.pt data/synthetic/test.jsonl
jevlike-predict runs/synthetic.pt 
  --context "Choose the exact badge amber badger. Badge: amber badger." 
  --option "azure crane" 
  --option "amber badger" 
  --option "gold heron"

The evaluation prints top-1 accuracy, which is the fraction of correct first choices. Top-3 accuracy is the fraction with the right answer among the three highest scores. Expected calibration error compares confidence with observed accuracy. The command also prints a shuffled-context control, which pairs each menu with the wrong context. A useful model should beat that control.

Use your own data

  1. Export train, validation and test JSONL files in the format above.
  2. Keep all options that the model will see at prediction time in each row.
  3. Split related records together. For example, keep all records for one customer or one target page in one split. This prevents near-duplicates from leaking into the test set.
  4. Run jevlike-train with your train and validation files.
  5. Run jevlike-eval once on the held-out test file. Held-out means the file was never used for training or model selection.

The default byte encoder truncates context to 192 bytes and each option to 32 bytes. Raise --context-tokens or --option-tokens when your text needs more room. Training supports CPU, Apple MPS for a Mac GPU, and CUDA for an NVIDIA GPU through --device.

Use a frozen pretrained encoder

Install the optional dependency and name any compatible encoder from Hugging Face:

uv pip install -e '.[transformers]'
jevlike-train data/synthetic/train.jsonl 
  --validation data/synthetic/validation.jsonl 
  --output runs/qwen-head.pt 
  --encoder hf 
  --hf-model Qwen/Qwen2.5-0.5B 
  --rank 256 
  --batch-size 8

The checkpoint stores the trained scorer head and the encoder name. It does not copy the frozen encoder weights. Loading the checkpoint therefore needs access to the same Hugging Face model.

--rank sets the width of the small scorer head. A wider head has more trainable weights and uses more memory.

Wikispeedia example

scripts/get_wikispeedia.sh downloads the public SNAP archives and builds next-click JSONL files. The data stay outside this repository.

scripts/get_wikispeedia.sh
jevlike-train data/wikispeedia/jsonl/train.jsonl 
  --validation data/wikispeedia/jsonl/validation.jsonl 
  --output runs/wikispeedia.pt

Cite Robert West and Jure Leskovec, Human Wayfinding in Information Networks, WWW 2012. Review the source data terms on the SNAP dataset page.

What to expect

In the experiments that led to this starter, the one-pass scorer reached about 98% accuracy on synthetic menus. On target-disjoint Wikispeedia next-click data, a frozen Qwen2.5-0.5B encoder plus the scorer reached 26%, against about 8% for shuffled and random-encoder controls. A small model trained from scratch on 40,000 clicks reached 29%. At eight options, one pass was about 100 times faster than a small decoder forced to write 400 tokens.

These numbers describe local experiments, not this quickstart run. We did not show equal quality with Jev or reproduce TypeSafe’s private training method.

Limitations

  • This is a research starter, not a copy of Jev.
  • Accuracy depends on data quality, split quality and the encoder.
  • The byte encoder is cheap but weak on language meaning.
  • The pretrained path may download a large model and needs more memory.
  • One-pass scoring requires the complete option list before prediction.
  • The speed comparison used a small local decoder rather than a large commercial model.

Licence

Code is released under the MIT License. Downloaded datasets and pretrained models keep their own terms.


Source: Hacker News

My temporary PHP fix from 2014 has nearly 20M installs. Today I'm deprecating it

Twelve years ago, I wrote 174 lines of PHP as a stopgap for AOL’s content management system. I put it on Packagist in case anyone else needed the same patch, and somehow it’s been installed nearly 20 million times since. Today I marked it deprecated.

A temporary shim

In 2014, we were in the middle of upgrading AOL’s CMS from PHP 5.2 to 5.3. Part of that upgrade was dropping version 1 of the pecl_http extension, which gave us a function called http_build_url(). A CMS deals with a lot of URLs, and ours called that function in dozens of places. I wasn’t touching those. The function seemed straightforward enough to reproduce, so I wrote my own http_build_url(), defined only if the real one didn’t already exist. The old code never knew anything had changed.

Composer was just taking off at the time, which made sharing it easy. I figured it would earn its keep for a year or two, until the PHP community moved on to something better.

That’s a lot of installs

Well, it wasn’t temporary. It’s been installed from Packagist nearly 20 million times, and it still picks up over 400,000 installs a month.

Packagist install statistics for jakeasmith/http_build_url as of September 15, 2026: 19,864,271 total installs and 401,308 in the last 30 days. Daily installs climb steadily from near zero in 2014 to about 13,000 a day in 2026.

And it turns out Composer is only part of the picture. WPML, the market-leading multilingual plugin for WordPress, bundles the polyfill directly in its codebase, and WPML says it’s installed on over 1.5 million sites. The domain-name library idna-convert depends on it too, which is how it ships inside the source of SPIP, a French content management system, and how it ended up packaged in Debian and Ubuntu. Between all of them, there’s a pretty good chance you’ve visited a website that is still running my code.

I never imagined it would go this far.

Coming back to it

I didn’t grasp how far it had spread until a few months ago, when I looked at the package for the first time in years. I knew it had users. By 2021 I’d been out of PHP for a while, and the downloads were surprising enough that I asked for a new maintainer. Three people offered. Shortly after I asked, we lost a family member unexpectedly, and it turned our world upside down for a while. I never followed up, and that’s on me. By the time things settled, other goals had taken over, and I forgot about the package for years.

Along with the numbers, there were a handful of GitHub issues, including one where joining a path onto a URL with a trailing slash strips every letter “a” out of the path. So much for straightforward. Under a comment that reads // Workaround for trailing slashes, my code tacks an “a” onto the path so there’s always a last segment to cut off, then cuts it off with a find-and-replace. When the path ends in a slash, that last segment is just the “a”, and the find-and-replace takes every other “a” in the path with it. I can’t believe the bug went unnoticed for as long as it did.

So I had a decision to make. I could dive back into PHP after almost a decade away, hand the package to one of the people who’d offered, or let it keep sitting there.

None of the above

It was always meant to be temporary, so I’m retiring it. The PHP League’s URI library has been the community’s answer for years, and PHP 8.5 now ships a standards-compliant URI API in the language itself (thanks to jawira for pointing me at it). Both are better than a 174-line shim from 2014. Maintaining the package would only delay the move everyone should be making, and handing it over would add a risk on top of that. I don’t doubt anyone who offered, and ozh has kept a fork going for YOURLS. But a widely installed package with a new maintainer nobody downstream has vetted is exactly what attackers look for. Veritasium’s video on the xz Utils backdoor is the best telling I’ve seen of how that plays out.

The package will keep installing, but it won’t get new fixes, including for the missing-”a” bug. After this long without a change, even a one-line fix could have unintended consequences for someone, with no one around to support it. The README shows how to switch.

I wrote this code to ease a painful migration, for myself and anyone else going through the same one. Thank you to everyone who sent a pull request or offered to take it over, and to the people who kept filing issues long after I’d stopped reading them. It was a good run for a temporary fix.

P.S. We never migrated AOL’s CMS off the “temporary” polyfill. It ran there until the whole platform was shut down around 2020.


Source: Hacker News

Speeding up gearhash on ARM64

← Writing

Speeding up gearhash on ARM64 (2× faster)

tl;dr: As of version 0.1.4, the
gearhash crate has gained a NEON backend
which makes it roughly 2× faster on ARM64 at typical chunk sizes. It is
selected automatically on aarch64 and backwards compatible, so consumers of the
crate don’t need to do more than just update. Read on if you’re interested in
the details of how this was achieved, or skip straight to the final
results
.

How it all started

At the end of 2019, I was building a personal backup system, and as part of
this, became interested in a technique called content-defined chunking. The key
idea behind it is that instead of chunking files on fixed chunk boundaries, you
run a sliding window hash function across the file and trigger a chunk boundary
whenever the hash has a particular value. The downside is that this gives you
variable length chunks over a distribution, but the upside is that your chunking
is now much more resilient to byte sequences being inserted or removed from the
middle of files.

Anyway, as part of this I came across the FastCDC
paper
. Its building block is the GEAR
rolling hash. Because I like fast things, I spent quite some time trying to work
out how to convert the serial algorithm published in the paper into a SIMD
algorithm. I ended up publishing the result of this as
gearhash, a small Rust crate with
optimizations for SSE4.2 and AVX2.

When I wrote the crate, ARM64 was not really a target worth optimizing for. AWS
had offered ARM64 instances for a year, but only the first-generation Graviton
A1 family, built on Cortex-A72 cores and marketed for scale-out workloads rather
than general compute. Graviton2, the first generation with a competitive core,
was announced at re:Invent the same month as my first commit and did not reach
general availability until May 2020. Apple announced the M1 in November 2020.

Fast forward to today, a lot has changed. Apple has pushed ARM64 into the
mainstream of consumer hardware. AWS has shipped several further Graviton
generations and says that for three years running more than half of the new CPU
capacity it added has been Graviton. GitHub Actions added free ARM64 runners for
public repositories in 2025. On all of those machines, the gearhash crate was
falling back to the scalar loop.

On top of this, while gearhash initially had virtually no production users
aside from myself, it has since become a core part of the Xet
client
, Hugging Face’s storage
protocol for large files on the Hub, which has replaced Git LFS as the default.
For gearhash this means we’re now doing between 10k and 20k downloads per day.
This renewed interest in the crate helped me find the motivation to see where I
can push things further.

The insight that unlocked parallelization

The gear hash kernel is defined as a serial function over 64-bit unsigned
integers:

hash = (hash << 1).wrapping_add(table[byte as usize]);

Two properties make this difficult to vectorize:

  1. It is a serial dependency chain. Every byte’s hash depends on the
    previous byte’s. There is no data parallelism to extract from a single
    stream.
  2. The table lookup is a gather. 256 × 8 bytes is 2 KB, far too large for
    any in-register permute. Every byte costs a real load.

After banging my head against this for a bit, I ended up making an observation
about the first property: the hash is 64 bits wide and shifts left by one bit
per byte, so after 64 bytes the starting value has been shifted out completely.
That means you can start hashing at any offset in a buffer with hash = 0, warm
up over 64 bytes, and from then on the hash is bit-identical to a pass from the
start.

What this enables is that a chunk can be split into strips: seed lane 0 with
the real incoming hash, seed every other lane by hashing the 64 bytes that
precede its strip, and run all strips in lockstep. When a lane reports a match,
you just need to work out which match is earliest, which is where most of the
complexity in the implementation ended up being.

Beginning the port to NEON

I started out by doing a straight port from the SSE4.2 implementation.
aarch64::uint64x2_t is two 64-bit lanes, the same as x86_64::__m128i, so the SSE4.2 structure
maps over almost mechanically.

The one thing that did not map over is the mask extraction. NEON has no
equivalent of pmovmskb, so getting the lane comparison results into a scalar
register takes a narrowing shift and a move, which I wrapped in a small
movemask helper.

The result was disappointing: 0.92×, slower than the scalar code.

To understand why, we need to take a look at the loop-carried latency on ARM64.
Per iteration the NEON version would do this:

add.2d   v1, v1, v1      ; h << 1, which LLVM emits as an add to itself
add.2d   v1, v1, v_g

On Apple cores each of these are ~2 cycles each (per Dougall Johnson’s M1
tables
), so ~4 cycles
per iteration, and an iteration covers 2 bytes (one per lane), which comes out
to ~2 cycles per byte.

The scalar version, hash = (hash << 1) + table[b], compiles to a single
shifted-register add, add x0, x1, x0, lsl #1, with ~2 cycles of latency. That
is also ~2 cycles per byte.

Which means that the vector version does the same amount of work per unit of
critical path as the scalar one, but on top of that has to pay for the loads and
the mask extraction. It cannot come out ahead.

To win on NEON, the dependency chain itself has to get shorter.

Shortening the chain

If the chain is 2 ops per 2 bytes, why not make it 2 ops per 4 bytes by writing
out two steps of the per-byte update and multiplying through:

h₁ = (h << 1) + g₀
h₂ = (h << 2) + (g₀ << 1) + g₁

With this, h₂ depends on h through a single shift and a single add,
provided you precompute G = (g₀ << 1) + g₁. G depends only on table
lookups, not on h, so it is off the critical path.

Result: 0.92× → 1.13×, better, but still well short of the expected 2×.

The reason hiding in the disassembly

add.2d  v2, v1, v1      ; h << 1
add.2d  v2, v3, v2      ; h₁ = (h<<1) + g₀
shl.2d  v1, v1, #2      ; h << 2
add.2d  v3, v3, v3      ; g₀ << 1
add.2d  v1, v1, v4      ; (h<<2) + g₁      <-- on the h chain
add.2d  v1, v3, v1      ; ... + (g₀<<1)    <-- also on the h chain

Turns out, LLVM had just gone and reassociated it! I wrote (h << 2) + (G₀ + G₁) and it emitted ((h << 2) + G₁) + G₀. This is a legal transformation of
course, but it puts a second add back on the dependency chain.

The somewhat naughty fix

You cannot stop the compiler reassociating a sum, but you can (try to) stop it
seeing one. The combined term is built from two table lookups, and those arrive
in general-purpose registers anyway, so the combining can just happen there:

let (t00, t01) = (table[b00 as usize], table[b01 as usize]);
let (t10, t11) = (table[b10 as usize], table[b11 as usize]);
 
// Combining the two table entries in scalar registers keeps the vector operand
// opaque, which stops the compiler from reassociating the addition below into two
// dependent vector adds on the loop-carried `h` chain.
let g1 = vcombine_u64(
    vcreate_u64((t00 << 1).wrapping_add(t01)),
    vcreate_u64((t10 << 1).wrapping_add(t11)),
);
 
let h2 = vaddq_u64(vshlq_n_u64::<2>(h), g1);

Checking disassembly now showed only shl.2d → add.2d on the chain.

Result: 1.13× → 1.46×, finally starting to be meaningfully faster, but not quite
fast enough!

From latency-bound to throughput-bound

Encouraged by the result of unrolling to two steps at once, I tried the same
with four intermediate states, each still computed directly from h:

let h1 = vaddq_u64(vshlq_n_u64::<1>(h), g[0]);   // hk == (h << k) + g[k-1]
// ...
let h4 = vaddq_u64(vshlq_n_u64::<4>(h), g[3]);

This halves the length of the dependency chain per byte again, so I expected
another large step. Measuring it however, there was no difference at all.
It was at this point that I suspected the limit may no longer be the latency
between iterations, but instead simply the CPU throughput.

Following that hunch, my focus shifted to try and reduce instruction counts
instead. The loop was now doing two loads per byte: one for the byte itself and
one for its table entry. While the latter is unavoidable, we can now can
replace those four consecutive byte loads with one unaligned 32-bit load, then
peel one byte off each word per step with a shift.

Result: 1.46× → 1.63×, another large step towards the 2× goal.

Testing four states with one branch

With the loads optimized, the next largest block of instructions per iteration
was the boundary test: at every step, the code checks for a chunk boundary by
masking the hash and comparing it to zero.

While I had unrolled to four intermediate states per iteration, it was still
probing them one by one. That is four separate moves out of the vector unit,
each with a branch waiting on it.

Because the common case is no match, what we can actually do is combine the four
tests inside the vector unit and make one trip out. If none of the four
positions in either strip is a boundary, then the iteration can move on after one
move to a general-purpose register and one branch:

let t = vandq_u64(
    vandq_u64(vtstq_u64(h1, maskv), vtstq_u64(h2, maskv)),
    vandq_u64(vtstq_u64(h3, maskv), vtstq_u64(h4, maskv)),
);
 
if movemask(t) == u64::MAX {
    i += UNROLL;
    continue;
}

Only in the uncommon case when that check fails does the code look at the four
states one by one. With the 16-bit mask the Xet client uses for its 64 KiB
chunks, that happens about once every 8192 iterations.

Result: 1.63× → 1.81×. Quite happy with this, and here is where I stopped for
now.

Final results

Everything combined, on the crate’s 11-bit benchmark mask, that takes the NEON
path from 0.92× for the direct port to 1.81× in the version that I published as
0.1.4.

However, one thing I realized while working on this which is quite obvious in
retrospect, is how dependent the benchmark is on the mask density. That’s
because the optimized path has a fixed cost per call that the scalar path does
not, and sparser mask means more boundaries and more calls. To see how much that
matters, I ran benchmarks across a range of masks with 4 to 20 bits set.

Throughput of the scalar and NEON paths against average chunk size on an Apple M4. NEON is slower than scalar below about 350-byte chunks and flattens out near 4,350 MB/s above 64 KiB, while scalar stays near 2,000 MB/s throughout.

bits set mean chunk scalar MB/s NEON MB/s ratio
4 16 B 996 250 0.25×
8 256 B 1864 1641 0.88×
11 2 KiB 1981 3565 1.80×
16 64 KiB 1999 4277 2.14×
20 1 MiB 2000 4347 2.17×

The crate’s own benchmark, at 1.8×, sits on the steep part of the curve, which
keeps rising until it flattens out at about 2.17× between 64 KiB and 1 MiB. The
16-bit row is the mask the Xet client uses, at 2.14×.

Sidenote: Below roughly 350-byte average chunks the per-call cost outweighs the
gain and the NEON path becomes progressively slower than scalar. I wouldn’t
expect anyone to use this type of configuration, but falling back to the scalar
path for masks with few bits set seems like a cheap way to close that gap.

What’s next

With NEON now roughly twice as fast as scalar, it feels like it’s time to take
another look at the x86 backends. Who knows, some of the tricks I learned along
the way on the NEON implementation might carry over. Stay tuned!


Source: Hacker News

Why a fast-growing German AI startup is moving its parent company from the US

By&nbspDoloresz Katanich
Published on

Share

Comments

Add Euronews on Google

Share
Close Button












In a highly unusual move, fast-growing German AI start-up Langdock has moved its legal headquarters from the US to Germany as it seeks to become a “European hyperscaler”.

Berlin-based Langdock has reversed the usual path taken by European start-ups by moving its parent company from the US to Germany.


ADVERTISEMENT

ADVERTISEMENT

The decision comes amid growing debate over how Europe can build its own AI infrastructure, retain promising technology companies and reduce its dependence on US cloud and AI providers.

Many European start-ups establish US holding companies in an effort to make it easier to secure funding. Langdock has now defied this trend and dismantled its US holding structure and brought its parent company under European law.

“Langdock has reorganised its corporate structure into a Societas Europaea (SE), registered in Germany, replacing the previous US holding-company structure,” a company representative told Euronews Business. The company said the process began in early 2026 and cost several million euros. It is now complete.

Founded in Berlin in 2023, Langdock is an enterprise AI platform serving about 13,000 organisations. It gives employees access to several AI models and allows companies to connect them to workplace data and applications, create AI agents and automate tasks.

Langdock said creating a US-registered parent company was initially a “vital step” that helped it attract investors and benefit from the support and network of US start-up accelerator Y Combinator. However, its operations and customer data remained in Germany.

Langdock has now replaced its US parent company with a European SE. The previous structure meant customers’ lawyers had to check whether the US parent created any legal or data-protection risks, even though Langdock said the US company had no employees, infrastructure or access to its production systems.

US laws, including the Cloud Act, have created concerns across the market that American authorities could seek access to customer data.

Removing the US parent makes Langdock’s European structure clearer to customers, the company said, particularly “in geopolitically uncertain times”. It added that it was now large enough to meet the stricter governance requirements of an SE.

“We believe Europe is a strong place to build a global technology company, and we want to contribute to its sovereignty and competitiveness,” the company told Euronews Business.

About 80% of Langdock is owned by founders and employees who live in the EU. The company said the new structure would not change its plans to serve customers worldwide or raise money from international investors.

Langdock says its annual subscription revenue reached a rate of $50 million (€42 million) in August, up from $1 million (€870,000) in October 2024. The figure estimates how much subscription income the company would receive over a full year if its current sales continued.

Langdock said its long-term ambition was to build “a sovereign, full-stack AI platform that can compete with US hyperscalers over time”. It plans to launch three new services by the end of the year and use its own data centre in Germany to run open-source AI models and provide computing power. The company said it would start small and expand as customer demand grew.

Despite its rapid growth, Langdock has a long way to go before it can compete with US tech giants. Its $50 million annual revenue run rate compares with the $128.7 billion (€108 billion) in revenue generated by Amazon Web Services in 2025.

Langdock’s decision to relocate its parent company offers a test of whether European regulation and concerns about digital sovereignty can become a competitive advantage, rather than simply a burden, for the region’s AI companies.

Go to accessibility shortcuts

Share

Comments

Add Euronews on Google

Read more


Source: Hacker News