It looks like that is in reference to the embedded interactive code blocks. If you use uBlock Origin you can use the element picker to remove the annoying image.
I believe that Python is as popular and widely used as it is because it's old enough to have an expansive ecosystem of libraries. It's easy enough to implement one in pure Python and possible to optimize it later (Pydantic is a great recent-ish example, switching to a Rust core for 2.0).
That same combination of Python + (choose a compiled language) makes it quite difficult for any new language to tap into the main strength of Python.
Python is popular because of it's expansive ecosystem of libraries, which only exist because the language is duck typed. If it was statically typed, the expansive ecosystem of libraries wouldn't exist.
There is a factor of 3x difference in dev speed between the two typing systems.
It's not just its age, it's how easy it is (was?) to jump in and start writing useful code that could be revisited later on and be able to read it and understand it again.
All of these efforts to turn it into another Typescript are going to, in the end, kill the ease of use it has always had.
F# has a similar whitespace syntax to Python, but is statically typed and can be compiled AoT.
Bubble sort Python:
mylist = [64, 34, 25, 12, 22, 11, 90, 5]
n = len(mylist)
for i in range(n-1):
for j in range(n-i-1):
if mylist[j] > mylist[j+1]:
mylist[j], mylist[j+1] = mylist[j+1], mylist[j]
print(mylist)
Bubble sort F#:
let mylist = ResizeArray [ 64; 34; 25; 12; 22; 11; 90; 5 ]
let n = Seq.length mylist
for i = 0 to n - 2 do
for j = 0 to n - i - 2 do
if mylist[j] > mylist[j + 1] then
let temp = mylist[j]
mylist[j] <- mylist[j + 1]
mylist[j + 1] <- temp
printfn "%A" mylist
var mylist = [64, 34, 25, 12, 22, 11, 90, 5]
let n = mylist.len
for i in 0..n-2:
for j in 0..n-i-2:
if mylist[j] > mylist[j + 1]:
swap(mylist[j], mylist[j + 1])
echo mylist
Nim feels like a really amazing language.
There were some minor things that I wanted to do with it. Like trying to solve a codeforces question just out of mere curiosity to build something on top of it.
I felt like although it was similar to python. You can't underestimate the python's standard library features which I felt lacking. I am not sure if these were skill issues. Yes these are similar languages but I would still say that I really welcome a language like SPy too.
The funny thing is that I ended up architecting a really complicated solution to a simple problem in nim and I was proud of it and then I asked chatgpt thinking no way there can be anything simpler for it in nim and I found something that worked in 7-10 or 12* lines and my jaw dropped lol. Maybe chatgpt could be decent to learn nim imo or reading some nim books for sure but the packages environment etc. felt really brittle as well.
I think that there are good features of both nim and SPy and I welcome both personally.
If either/both are not "great enough" in some particulars you want, why not raise a github issue? (Or even better look into adding said particulars yourself? This is really the main way Python grew its ecosystem.)
But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?
Hypothesis keeps a database of failures to use locally and you can add a decorator to mark a specific case that failed. So you run it, see the failure, add it as a specific case and then that’s committed to the codebase.
The randomness can bite a little if that test failure happens on an unrelated branch, but it’s not much different to someone just discovering a bug.
You can either use the @example decorator to force Hypothesis to check an edge case you've thought of, or just let Hypothesis uncover the edge cases itself. Hypothesis won't fail a test once and then pass it next time, it keeps track of which examples failed and will re-run them. The generated inputs aren't uniformly randomly distributed and will tend to check pathological cases (complex symbols, NaNs, etc) with priority.
You shouldn't think of Hypothesis as a random input generator but as an abstraction over thinking about the input cases. It's not perfect: you'll often need to .map() to get the distribution to reflect the usage of the interface being tested and that requires some knowledge of the shrinking behaviour. However, I was really surprised how easy it was to use.
As far as I remember, hypothesis tests smartly. Which means that possibly problematic strings are tested first. It then narrows down which exact part of the tested strings caused the failure.
So it might as well just throw the kitchen sink at the function, if it handles that: Great, if not: That string will get narrowed down until you arrive at a minimal set of failing inputs.
> Which means that possibly problematic strings are tested first.
Hypothesis uses a the same probability distribution for all the 200 (or so) random cases it generates for a test. The first case has the same distribution as the 200th.
However, Hypothesis gives a pretty large weight in the probability distribution to inputs that are generally 'problematic'. Of course, that's just a heuristic: eg empty lists and 0 and empty strings or strings with apostrophes in them and NaN or infinity often are problematic, but that's just a guess: hypothesis doesn't know anything specific about your code.
The heuristics work remarkably well in practice, though.
Once Hypothesis has found a failing test case, then it tries to shrink it down.
If you know it will fail on apostrophe you should have a specific test for that. However if that detail is burried in some function 3 levels deep that you don't even realize is used you wouldn't write the test or handle it even though it matters. This should find those issuses too.
> But let's say employee names fail on apostrophe. Won't you just have a unit test that sometimes fail, but only when the testing tool randomly happens to add an apostrophe in the employee name?
If you just naively treat it as a string and let hypothesis generate values, sure. Which is better than if you are doing traditional explicit unit testing and haven’t explicitly defined apostrophes as a concern.
If you do have it (or special characters more generally) as a concern, that changes how you specify your test.
Either your code shouldn’t fail or the apostrophe isn’t a valid case.
In the former, hypothesis and other similar frameworks are deterministic and will replay the failing test on request or remember the failing tests in a file to rerun in the future to catch regressions.
In the latter, you just tell the framework to not generate such values or at least to skip those test cases (better to not generate in terms of testing performance).
I think what they meant is, "won't Hypothesis sometimes fail to generate input with an apostrophe, thus giving you false confidence that your code can handle apostrophes?"
I think the answer to this is, in practice, it will not fail to generate such input. My understanding is that it's pretty good at mutating input to cover a large amount of surface area with as few as possible examples.
Hypothesis is pretty good, but it's not magic. There's only so many corner cases it can cover in the 200 (or so) cases per tests it's running by default.
But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code, even if you haven't done anything specifically to address this problem.
Also, even with Hypothesis you can and should still write specific tests or even just specific generators to cover specific classes of corners cases you are worried about in more detail.
> But by default you also start with a new random seed every time you run the tests, so you can build up more confidence over the older tests and older code
Is it common practice to use the same seed and run a ton of tests until you're satisfied it tested it thoroughly?
Because I think I would prefer that. With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in production.
You can think of property based tests as defining a vast 'universe' of tests. Like 'for all strings S and T, we should have to_upper(S) + to_upper(T) == to_upper(S+T)' or something like that. That defines an infinite set of individual test cases: each choice for S and T gives you a different test case.
Running all of these tests would take too long. So instead we take a finite sample from our universe, and only run these.
> With non-deterministic tests I would always wonder if it's going to fail randomly after the code is already in production.
You could always take the same sample, of course. But that means you only ever explore a very small fraction of that universe. So it's more likely you miss something. Remember: closing your eyes doesn't make the tiger go away.
If there are important cases you want to have checked every time, you can use the @example decorator in Hypothesis, or you can just write a traditional example based test.
the more the test runs the less likely it is there is an uncovered case left. So your confidence grows. Remember too anything found before release is something a customer won't find.
Yes, you should note the seed you use for a run in the logs, but you should use a new seed each run (unless trying to reproduce a known bug) so you cover more of the search space.
Hypothesis is a search algorithm for finding ways that we have misunderstood our code (whether that be in the codebase, or the claims we have made about it in the spec/tests). Conceptually, that's the inverse of "a unit test that sometimes fails": it's a search that sometimes succeeds at finding out that we're wrong. That's infinity% more effective than a search procedure which is hard-coded to check a single example; especially when that example is chosen by the same dev (with the same misconceptions!) that implemented the code-under-test!
Now, as for what would actually happen in that situation, when using Hypothesis:
- As others have indicated, Hypothesis keeps a database of the failures it's found. Committing that database to your project repo would act like a ratchet: once they've successfully found a problem, they will continue to do so (until it's fixed).
- If you don't want to commit your database (to avoid churn/conflicts), then the same would happen per working-copy. This is fine for dev machines, where repo directories can live for a long time; though not so useful for CI, if the working-copy gets nuked after each run.
- Even if we disable the failure database, property failures will always show the inputs which caused it (the "counterexample"), along with the random seed used to generate it. Either of those is enough to reproduce that exact run, giving us a ready-made regression test we can copy/paste to make the failure permanent (until it's fixed). As others have said, Hypothesis properties can be decorated with `@example(foo)` to ensure a particular input is always checked.
- Even if we disable the failure database, and don't copy the counterexample as a regression test, property checkers like Hypothesis will still "shrink" any counterexamples they find. Your example of apostrophes causing problems is trivial for Hypothesis to shrink, so if/when it happens to find some counterexample, it will always manage to shrink that down to the string "'"; essentially telling us that "employee names fail on apostrophe", which we could either fix right now, or stick in our bug tracker, or whatever.
The point about a datalake is to separate computer and storage. Postgres isn’t a compute layer it’s an access layer.
Your compute asks Postgres “what is the current data for these keys?” Or “what was the current data as of two weeks ago for these keys?” And your compute will then download and aggregate your analytics query directly from the parquet files.
but most serious compute engines already speak Iceberg, what do they gain from interfacing with PG now?
My understanding is the opposite - PG cuts it as a compute layer for small amounts of data, and this is where it excels.
I also assume `pg_lake` was built mainly with the intention of creating/writing tables, and the ability to read comes "for free" as an extra, since Iceberg integration is already written.
reply