Congratulations on getting first rank on front page.
Congratulations on getting your hands dirty and doing everything yourself like computing gradients manually, badly shuffling (non Fisher-Yates), badly js transpose (double swapping), it is a great way to learn.
Congratulations on completing a full pipeline, that's the hard part then it just swapping pieces for better pieces.
I advise non-technical readers not to attach much value to the results of this neural network as it is probably inferior to the even simpler naive Bayes.
The Concat operation mean it is especially sensible to dropping or adding a word as it will offset the remaining words and give a totally different vector.
Using word vectors mean it doesn't forgive any spelling mistake as a spelling mistake will usually correspond to <unknown> vector.
Using a feed forward neural network means formulaic titles with a single word substitution from a good positive example from the training set will often work.
It is trained by gradient descent using a squared error loss, on ~1000 examples one example at a time without cross-validation using a custom written neural network library. (Almost all these bad choices can be solved by using a framework).
It seems to have successfully over-fit as it return Good ~1.0 for positive examples from the training set.
Thank you for summarising and thx to OP for publishing his work. The combination of both was a nice opportunity for me (and others?) to learn something from mistakes. :-)
I really like this comment. Thanks a lot for taking the time to going through the code and figure all that out. The way I learn things is by doing. Since I started college, the only way I can understand a mathematical equation is by turning it into code. The machine learning class I took was a basic one. We learned about knn, kmeans, adaline, perception, linear regression... the semester ended with a multilayer perceptron with only one hidden layer. This Dejavu NN you see in my code is like the fourth iteration of me trying to really understand NNs. Some of the things you said I already knew but didn't took the time to apply: like cross validation. Tht shuffle algorithm I didn't knew, thanks for that. Anyways, if all the criticism I got from my projects were like yours i'd be such much better now.
None of these projects are perfect, but helped me to better understand things. I did implemented a professional solution for a company using machine learning, but I used scikit-learn (back then). I have some experience with tensorflow also, but implementing things makes me fell like in control. But I know whena a solution is good enough for production and when it's not. I made clear on the README of this project that this is not at all good for production. I took the time to show every single step of the process and tell it how it is. I had limited time and limited resources, and with the little I had I made a "fun" experiment. I enjoy doing this kind of thing. I have to squish this kind of projects in my free time, cause I have a full time job and I'm finishing my final thesis (I'll defend it two week from now).
Sorry if I turn this into a journal entry, but I kind got hurt yesterday when the other guy shted all over my project That's not at all what you did and I appreciate that. Thanks again.
This. Exactly this. No sophisticated tokenization. No interesting architecture using attention. And the author is completely clueless about overfitting... and even cross entropy loss. He could have gotten better results just using a bag of words approach.
But this ends up on frontpage anyway. Welcome to HN.
By now it's clear from your history of behaving aggressively in HN threads that you don't mean to use the site as intended. I've banned the account. https://news.ycombinator.com/item?id=21245546
I'm sad to do that because you are knowledgeable about a number of things. Many of us could learn from you if you would share what you know without putting other people down. But the aggression subtracts more than the knowledge adds. We can't have users behaving like this sort of asshole in comments, least of all in Show HN threads, where the idea is to teach people things and expressly not to shit on them and their work: https://news.ycombinator.com/showhn.html.
Other users here know things and are willing to talk about them without being mean. GistNoesis modeled this wonderfully in the GP comment. We'll learn what we can from them instead. But if you decide that you want to use HN in the intended spirit, as described in the site guidelines and especially the one that says Be kind, you're welcome to email [email protected] and let us know.
So all in all, only 3 of today's top 10 had good titles... Either the titles could have been better but the content was too interesting, or this tool has very low recall.
I think this is a prime example of where AI could go wrong. When people just talk about social media AI curation they don't really understand it. But I personally really wish social media would do less AI curation, who knows what gems we've missed, just because they're maximising for our instant satisfication.
Kinda spooky even, who knows, social media totally might have already killed companies that sounded too different or even just political ideas that differ from mainstream (or sponsored) views?
I think the problem is that the title is not a good indicator for current-event related submissions. "More Intel speculative execution vulnerabilities" may be a bad blogpost, but it's an important current event, so it still gets to the top regardless of the title selection.
Categorizing submissions to different types, and repeat the experiment, you'll find the program may predict blog/article and "Show HN" submissions with higher accuracy.
> This project is far from credible. All the things I did were to satisfy my own curiosity. With that being said, the bigger limitation I can see is that I only had access to a few stories. I also cannot validated the neural network prediction, cause in order for me to do that, I would have to write a content, come up with a title and then post it choosing words that triggers a good value on the neural network and post that history on a Friday noon, to see if my story succeed.
This is somewhat interesting. If what is stated in the comment itself is true then that means the poster had to solve to find x and y such that 'This comment is going to collect the most votes yet is predicted to be Bad: x - Good: y' would give Bad: x and Good: y as the output when passed through the neural net. Maybe they also manipulated other parts of the comment to find x and y.
I didn’t read the code in the post and don’t have any deep familiarity with machine learning, but I have implemented a naive bayesian classifier to do something similar for tweets. The scores you get from that method don’t add up to 1 either.
So that's a neural network thing? If the input is 'word space' it trains to detect buzzwords?
I'm honestly interested - do we hobble our neural networks with our choice of training space? If for instance, the HN-NN-input space including 'prepositions and word-frequency statistics', would the network train for sentence sophistication? Just because those stats were in front of it?
That is how it works here, you collect upvotes if you use fancy words. That's why everybody uses the word "orthogonal" here all the time. Have you ever seen that word anywhere else?
> That's why everybody uses the word "orthogonal" here all the time. Have you ever seen that word anywhere else?
Yes, frequently. It is common in various areas of math -- mathematical background is not something "normal people" usually flaunt, but there are good reasons to expect programmers to have much more such background than average.
Well, here's the thing: a good Samaritan offered 2.6M stories from HN with score. I've downloaded the file (almost 500M) and I'm now processing it. It is taking a long time to just process it. I don't know if I'll be able to train the neural network with all that data. As I said on the repo, the project is was a quick thing, just to test a theory. My question is: do you think is worth feed the NN more data so it can make better predictions? Please up vote this comment so more people could give their opinion.
Depends - if you really want to explore the tech, this is precisely the way to do it. I would be interested in the results, especially comparing them to your initial results.
It would be nice to include an analysis of how good is this with real data. Perhaps pick all the submission from yesterday and show the correlation between the real points and the prediction.
The distribution of points is very 0 heavy. It can be a problem to represent it and to model it.
Interestingly, a couple months back I saw a reddit thread where someone collected data that showed that posts with small spelling errors will gain (in some cases significantly) more upvotes - for whatever reason.
I tried feeding https://rachelbythebay.com/fun/hrand/ to this. The results were... well, the input was very repetitive and the NN was only trained on 1.5k titles, so what can I say.
Some of the best:
1.0000 0.0001 The police found programming to raise sheep
1.0000 0.0001 The police found PS4 in the cloud
1.0000 0.0001 The future of API in the cloud
1.0000 0.0001 The economics of iPad in the cloud
1.0000 0.0001 The economics of iPad in the cloud
1.0000 0.0001 My framework for Windows is patented
1.0000 0.0001 My framework for SDK is patented
1.0000 0.0001 My framework for Heroku is patented
1.0000 0.0001 I bootstrapped my Heroku in 2 years
1.0000 0.0001 Google kills Y Combinator in space
1.0000 0.0001 Coming soon: bigger blog is patented
Some of the worst:
0.0000 1.0000 Apple has Y Combinator in 2 years
0.0000 1.0000 China creating PS4 at a coffee shop
0.0000 1.0000 China creating PS4 at a coffee shop
0.0000 1.0000 Choose Heroku for developers
0.0000 1.0000 Fixed: NoSQL at a coffee shop
0.0000 1.0000 Followup to the Arduino without warning
0.0000 1.0000 Followup to the marriage without warning
0.0000 1.0000 How I made Android at Stanford
0.0000 1.0000 How I made Obama on the freeway
0.0000 1.0000 How I made bitcoins in the cloud
0.0000 1.0000 How I made iPod at Stanford
0.0000 1.0000 How I made iPod on the freeway
I did this by porting this project to node (which was shockingly easy - took 3 minutes - because there were no JS libraries or frameworks used :D :D :D :D) and running it at the console. 5000 lines of output and repro instructions over at https://gist.github.com/exikyut/1714ad98a136d77d8674944410a4... (the output got pasted first for some reason, sorry)
While reading this, I see a lot of negative comments, but here's something to think about.
Machine Learning and AI isn't about producing a perfect solution; that results in things like overfitting. Instead, think of Machine Learning as a human being. No person is going to be prefect and no single person is going to be able to please everyone. Instead, it's about building a solution that generally speaking, provides "good results". Subjective, 100%.
The project just started. Once the OP starts adding new features to the data set and improving the data set itself, I'm sure the results will start getting better and better. At which point, it'll be a good system to "test your subject lines" before posting.
What's the worse it could do? Tell you a subject is great when a bunch say it sucks? I've clicked on many head lines that I thought "sucked" but ended up finding the content very useful.
Great job to the OP and keep it up. This type of work isn't easy but certainly can be fun. Fk all of the negative opinions and keep on keeping on.
Yup, that's a big one. A topic where it is very noticeable is when it concerns monopolies.
Negative post or comment about monopolies when Americans are the online majority? Expect to get downvoted into oblivion.
Posted when Europeans are the online majority? Up you go.
To be clear: this isn't a complaint about downvoting, I'm just pointing out the phenomenon. For example, its possible to go to bed with +7 and wake up to -4 (a rather stark difference) because a certain comment was posted during the European evening, and thus was 'exposed' to American HNers longer than it was to European ones.
Makes me wonder if there is ever any non-toy usage of AI sentiment analysis. People (managers, customers, marketing, etc) think is the hottest thing since sliced bread, but every time I've looked at it, the results are meaningless noise, like in the myriad of examples in the comments here.
It's a cool bullet point in the slide deck, and it gives you some metrics to graph on dashboards, but I'm unconvinced that it means anything.
I'd argue that titles, as long as they're neutral and accurate, have nothing to do with the upvotes you'll get. This is a perfect example of using data to try to find something that simply doesn't exist or that bear so little weight in comparison to the other variables that you can safely ignore it.
Well yeah, that's exactly what I said. Gaming the system by tweaking your title according to a random github "neural network" won't help you if your content is shit, and if it's not shit the title will be good enough.
this neural network is politically biased and possibly racist. it also prefers swedish girls over swedish boys.
I do have to agree though, that cheese is better than mouldy cheese. and the fact "comacho for president for ever" nets a score of "Bad: 0.0415 - Good: 0.9614" which i think is fair reflection of voter behaviours and general consensus among humans. 'man' and 'not man' its not so fussy about, 'woman' gets same score as 'man', but 'not woman', oh thats just bad (Bad: 0.9909 - Good: 0.0097)
all in all a great tool to analyse your titles, it will surely help this community grow and mature over time, finally. Thanks a lot for creating this.
In response to this article, I was expecting insightful comments on the neural network itself and how to improve it, but instead I'm mostly reading funny attempts to play with the scoring :)
I spent significant time on binary text classification, specifically with HN titles. You can actually get up to 65-70% post popularity accuracy just by looking at the post title.
I am currently creating a filter that filters HN news and similar sources using a similar classfier. It learns on the fly and the accuracy of guessing my 'taste' is about 75%-80%. Better accuracy for this is explainable by the fact that my interests are more focused and the classifier has easier time predicting posts I would be interested in.
I feed it a bunch of articles currently on the front page and tabs I had open that I might submit and got bad on all of them. Then I started typing stereotypical Hackernews click bait and got a pretty solid score.
Paul Graham Rust VC IPO growth
Bad: 0.0026 - Good: 0.9976
So far adding any kind of grammatical structure to the random list of keywords that turns it into a title that makes sense completely ruins the score...
Very cool. I had the same idea a couple of years back and implemented a very similar interactive tool [1]. If you find the topic interesting, then you might also enjoy the analysis explained in that blog post.
BTW, I tried a bunch of single word titles (example: red, green, blue, title), and I always seem to get the same result: Bad: 0.4528 - Good: 0.5472
So, apparently, if you want to maximize your "score" with the lowest mental effort, just spam thousands of single word title posts, and then, it's a coin flip for each one :)
> I also cannot validated the neural network prediction, cause in order for me to do that, I would have to write a content, come up with a title and then post it choosing words that triggers a good value on the neural network and post that history on a Friday noon, to see if my story succeed.
Adam Neuman got away with billions with WeWork bankrupt
Apple and Google help China's surveillance of dissents
Google Chrome monitors straight viewers
Solar panels can not slow climate change, only nuclear power can
SpaceX to to build new Silicon Valley on Mars
Would have been nice if some keywords were likely to increase or lower the score prediction. For instance, popular things like "rust" would probably increase popularity.
Adam Neuman got away with billions with WeWork bankrupt
Apple and Google help China's surveillance of dissents
Google Chrome monitors straight viewers
Solar panels can not slow climate change, only nuclear power can
SpaceX to to build new Silicon Valley on Mars
"Can a neural network predict how many up votes your HN post will have?" or "Can an optimizer find a parametrization of a nonlinear manifold corresponding to tech zeitgeist?"
This is an interesting question of what context titles should be evaluated in.
If your model is "pick a title, and then predict how it will do", with the title being an independent variable, then the overwhelmingly negative assessment is quite correct. Most of the time, Stephen Hawking (or any other celebrity) hasn't just died, and that title would be a lie or a hoax.
To predict these obituary articles better, you'd need to build in an assumption that the content of the title was true, which would cause a lot of problems in the general case.
IDK but I know for a fact that I can comment with the same message in different ways and get upvoted or downvoted accordingly just by wording the the comment differently
you are trying to use a small simple neural network to predict the behavior of thousands of other completely different and significantly more complex neural networks: the average Hn user.
According to this tool, HN would not give a fuck about "Windows 11 will use a Linux kernel", "Mark Zuckerberg's lizard tail falls off during interview" or "Google Search to be rewritten in PHP", but "Trump nukes North Korea" would at least be 0.5719 good.
So... I guess the neural network has some learning to do?
Congratulations on getting your hands dirty and doing everything yourself like computing gradients manually, badly shuffling (non Fisher-Yates), badly js transpose (double swapping), it is a great way to learn.
Congratulations on completing a full pipeline, that's the hard part then it just swapping pieces for better pieces.
I advise non-technical readers not to attach much value to the results of this neural network as it is probably inferior to the even simpler naive Bayes.
The model of the neural network is simplistic :
Concat(Word Vectors)-Dense(120,act=sigmoid)-Dense(60,act=sigmoid)-Dense(2,act=sigmoid)
The Concat operation mean it is especially sensible to dropping or adding a word as it will offset the remaining words and give a totally different vector.
Using word vectors mean it doesn't forgive any spelling mistake as a spelling mistake will usually correspond to <unknown> vector.
Using a feed forward neural network means formulaic titles with a single word substitution from a good positive example from the training set will often work.
It is trained by gradient descent using a squared error loss, on ~1000 examples one example at a time without cross-validation using a custom written neural network library. (Almost all these bad choices can be solved by using a framework).
It seems to have successfully over-fit as it return Good ~1.0 for positive examples from the training set.