> It turns out that their algorithm makes a successful prediction about 70 per cent of the time. That’s far from perfect but much better than random guessing which is right only half the time.
From the graph, it looks like only about 27% of requests are fulfilled in the best case (jobs). In which case I can do better than 70% just by constantly predicting "no".
(I assume that this is just bad reporting. I haven't read the reference.)
Edit: skimmed the referenced article. Average success rate is 24.6%. The 70% they give (well, 67.2%) is "the probability that a classifier will rank a randomly chosen positive instance over a randomly chosen negative one".
They got a ROC AUC score of 0.67. This means that for a randomly chosen denied request and a randomly chosen accepted request, they will give a higher score to the accepted request 67% of times.
No, reread the parent's statement. The test involves distinguishing exactly one known accepted request and exactly one known rejected request. Probability of an acceptance does not come into play here; hence a random algorithm would choose the accepted request correctly 50% of the time.
What the algorithm performs poorly at is determining whether any single arbitrary request is accepted or rejected; that's the test that would require around 80% success rate.
"Since the AUC is a portion of the area of the unit square, its value will always be between 0 and 1.0. However, because random guessing produces the diagonal line between (0, 0) and (1, 1), which has an area of 0.5, no realistic classifier should have an AUC less than 0.5."
So it appears that not 0.8 but 0.5 is the randomness threshold; therefore 0.7 is not so bad.
Unfortunately not. IIRC, any classifier with AUC < 0.5 can be improved by just inverting its output. Real classification quality requires being significantly above 0.5 -- 0.8 is a common threshold.
Random pet-peeve. I dislike how the graph in the article has markers at each decile.
This is model-generated data so they could put an infinite of markers at arbitrary locations. The use of markers implies to the viewer that is a data-generated figure, which it is not.
>We find that Reddit users with higher status overall (higher karma) [...] are significantly more likely to receive help
There could be a common cause to these two factors (a certain way of writing, for instance), that could explain the correlation. I can't imagine someone basing a judgement on a stat reported on someone's user page.
GP means that things like being an effective communicator and knowing how to play to the crowd will factor into karma and into asking a question that gets answered.
That doesn't mean score-viewing doesn't happen, but it isn't likely to be the only mechanism at play.
Reddit Enhancement Suite, a ubiquitous browser extension, shows the link karma, comment karma, and account age of any username that you hover over.
So, it's totally possible that, like others have said, people deciding which request to fulfill take karma/account age into consideration to avoid gaming the system with multiple accounts.
That seems fairly self explaining. By the definition of having more karma, they're better acquainted with the site and have successful experience appealing to the audience.
I'm sure it could be a factor (nothing's preventing someone from doing that, after all) but my original point was essentially as characterized in this post below https://news.ycombinator.com/item?id=7793607
The tone of the article, I think, was suggestive of a causal relation, which surely doesn't (necessarily or plausibly) hold.
even if it is "threshold" its simply a binary relation and not an explanatory variable at the margin. so, yes...you might be able to weed out some data...but of what is left its unliklely to differeniate, except by proxy (as alluded to in the other comments) for something else.
beside the other reasons, a simple alternative might also be that users with high reputation usually are known by the regulars. If you recognize someone’s name for posting a lot of interesting stuff on your favourite website you probably help him out before other guys.
> Althoff and co used a standard machine learning algorithm to comb through all the possible correlations.
What exactly is a "standard machine learning algorithm"?
I'm sure that probably means that they used something from scikit-learn but "comb[ing] through correlations" isn't as simple as clicking the Go button.
The rest of the article does start to get into labeling, holding out a test set, and some of the data cleanup (the real combing working).
I guess I was just hoping for more detail of how it worked and not that it worked. I get that this wasn't meant to be a PhD thesis on supervised machine learning, but the mechanics of data analysis are really interesting as a process of discovery.
Curious to know what others think. How did the balance of how vs that work for you?
It's a logistic regression model, a basic statistical technique which wouldn't have even come under "machine learning" a few years ago. Later they use some kind of LASSO regression to penalise the inclusion of redundant features.
"Combing through the correlations", it seems, literally means calculating the (Pearson) correlation between two variables (success vs. an input feature) and adding some interpretation, as they do on p6 of the arXiv paper. For test/training data, it looks like they just used a 30/70% split rather than k-fold cross-validation and holdout, but I'm sure it makes no difference either way and in this case (as often) is trivial to design and implement. Presumably their AUROC could be increased just by dropping in an SVM or a Random Forest in place of the logistic regression.
From what I've skimmed of the paper you're over-estimating the complexity of the study.
Great Comment: Having studied graduate statistics within the stats department and a data mining within the CS dept, It amazing how well the CS crowd has rebranded statistics into something you can talk about at a bar without people's eyes glazing over.
Here's a project I put out in January, 65% accuracy, that does the same thing. These guys actually contacted me back then, surprised to see no mention.
Are you suggesting that these people aren't being truthful about their need, or, are you saying it's amazing that internet access is so [relatively] cheap? Or... ?
Internet access costs less per month than a takeaway pizza in my country. Libraries and other community centres provide free access to computers with broadband internet.
One of the surprises for me when, some years ago, I found myself in a developing nation (with unstable power supplies and non-potable tap water) was that every street corner in the town I was staying in had a (hand painted) advert for an internet cafe. It wasn't cheap compared to food prices - but since then globally food prices seem to have gone up and internet prices gone down considerably.
I just wanted to express how bizarre this situation is but not imply anything about (possible) reasons. My first thought was like just terminate the internet contract, sell the phone or computer or whatever you are using and buy you 50 kg of rice. That would be even healthier than pizza!
But I decided not to phrase it this way because this is obviously a very simplistic view and the people on the internet will tell you that in full detail even if you are aware of it. They might have free access to internet. It might be a temporary situation and not be economical to sell and later buy back the computer. Corn is much cheaper than rice. You have to cook rice and that requires energy. Rice alone is not healthy. Maybe they just do this to get in contact with others not because of having no money for food. They might need the computer for work. They just got robbed late at night, no money left but they still have a phone.
Further, a single pizza is unlikely to make much difference to one's nutritional status. I see these sorts of things as closer to emotional support than life-saving interventions.
Hmm, these people could be making the more rational choice. Depending on what they do for money, internet access could provide a higher return than food.
Higher return than staying alive? (Ignoring the fact that they can possibly dual use the internet to get free stuff, especially food, making it really a rational choice.)
From the graph, it looks like only about 27% of requests are fulfilled in the best case (jobs). In which case I can do better than 70% just by constantly predicting "no".
(I assume that this is just bad reporting. I haven't read the reference.)
Edit: skimmed the referenced article. Average success rate is 24.6%. The 70% they give (well, 67.2%) is "the probability that a classifier will rank a randomly chosen positive instance over a randomly chosen negative one".