I've typed and deleted this post a few times trying to find a way that it doesn't sound kind of pompous but if it helps save one person alot of money then screw it, I'll sound pompous....
I get asked quite a bit on how to start doing algorithmic trading and the first thing I always tell people is don't.
I think I've said this many times now but the number of people who come at it with the thinking "I'm a computer scientist. I'll just fire up R or python and apply some machine learning to the markets and watch the money roll in" is staggering.
I mean each day 100's of Phd's start with clean market data, more data sources than you could possibly think of and statistical back testing systems that have 1000's of man hours put into them, trying to find a way to make money.
After all of that if you really want to I wrote this in response to an Ask Hacker News a little while ago
- expect to take a year to figure out some edge in the market
- most decent trading strategies that a normal person can use come from economic/market insights first and technology second.
The site: https://www.quantstart.com/ is also decent at bringing you up to speed on the math you'll need to know though I believe that the material there oversells how easy it is to find a decent trading strategy.
I mostly agree with this. Treating markets as an exercise purely in data science is a _bad idea_. Risking money without a solid understanding of market mechanics and trading conventions is a recipe for disaster. Taking a Bayesian view won't save you either - this is a land where some new "six sigma" event happens every six months.
That said, the same logic that holds for identifying profitable strategies within an institution holds for individuals: unless you have better gear, don't fish a crowded pond. As an individual, your small size is, in some regards, an advantage. Institutions routinely pass on strategies that don't have capacity (there's not enough liquidity to make the strategy's returns worthwhile relative to their trading level) or strategies that aren't quite up to their standards (but might be up to yours). There's also a huge class of strategies that have somewhat choppy but long run consistent returns. Traders and funds worried about MoM track records won't touch those.
Fees are the biggest barrier to entry. Institutions enjoy substantial discounts and an ability to amortize costs across a much wider base, making the strategy performance hurdle rate proportionally higher for smaller traders.
Quantopian (www.quantopian.com) gives you the ability to trade through robinhood with long trades at 0 commission. There has been some skepticism in the Q forums on how well robinhood's execution is (possibly the effect of a "you get what you pay for"-attitude).
Trading real capital with algorithms is more difficult than it sounds. Mentally if you're algorithm is doing stock selection as well as trade execution, you no longer understand what you own. During a period of extended drawdown, you have to be mentally tough enough to believe that your algorithm has what it takes to dig itself out.
In a sense, it is like the start-up game. You have to believe you're doing something well enough that in the end you'll be right (and not bankrupt).
I use Interactive Brokers. They're probably the best retail brokerage out there. They're cheap -- the best rates on commissions and by far the best rates for margin. And they have a reasonably well-documented and supported API which you can write your programs against. The API is actually pretty much industry-standard so if you're interested in a third-party program, they probably support it.
If there's any downside, it's their data. You can't get that much of it historically (1 year max on the minute bars, far less on the second bars, and it takes forever to download it because of the throttling). It's also not...I'm going to say correct...historically. I mean, it is a correct record of what trades happened when. But it includes trades that you won't see when they happen, making it less useful if you're looking for a stream of events as they are happening. Also compared to a broker like Lightspeed (who I haven't tried) their data is expensive.
The software is a big mess, but it's usable. It's always in this weird state where it's 50-75% of the way to being completely awesome, but there's just a few things missing that stop it from being so. Also if I have one nitpicky complaint it's that I can't direct route complex options orders.
This level of defeatism is the exact reason I got into the markets algo trading.
People like you stay out.
More to the point, just because a genius mathematician and code breaker started a hedge fund it doesn't at all push out any of the little guys. The market is so large he can't possibly be trading all instruments at once, and "scaling" is a problem for huge hedge funds. Especially ones that have to answer to their shareholders. Even though this is a huge fund, it is highly inflexible.
What you're saying is akin to "Google already does email and they're loaded with geniuses, what makes you think you have an edge over that?"
In fairness, if someone told me they were going to make a killing on free webmail supported by ads and data mining, I think "How on earth do you intend to complete with Google?" would be a perfectly legitimate question.
In fairness, by that logic nothing would ever get done.
Microsoft? You're never going to win over IBM. Xerox? You'll always be ancillary to Kodak. GM? Ford is already there.
The Fugger's Banking company? Good luck against the Venetians and the Florentines. Same old story.
At the end of the day, you either try something new or you don't.
> In fairness, by that logic nothing would ever get done.
Or it will get done by people who have thought long and hard about how they are going to compete with the giants, rather than someone naïve sap.
The fact that the question is being asked does not imply that there are no good answers to it, but it is unlikely that someone who hasn't spent time considering how to improve on the incumbents is going to beat them.
Indeed, but nobody (I certainly wasn't) was advocating to not do your homework and evaluate your risk. Sometimes, however, things just happen, for example James Simons stumbled upon the industry by chance.
How many companies were motivated by killer instinct? Did Gates want to kill IBM? Did Xerox plan to kill Kodak (they didn't but they serendipitously started a revolution)?
Comparative advantage. Even if they're absolutely better than you at every possible strategy, there are still going to be opportunities in markets that it isn't worth their time to invest in.
This isn't true. There are a non-zero number of traders who use IB and post positive returns.
For example Taaffeite Capital Management (who gained some publicity for good returns on Brexit) are a < $10mm fund who use IB[1].
Sure, bigger funds have better market access, and it will always be impossible to implement a high frequency trading approach. But these platforms are about as good as a small player can get.
There are plenty of MIT PhDs very successfully losing money.
I think this goes directly to the "if you use these platforms you are a fool" comment. It seems to me that some people who aren't fools use the platforms, and some who are, don't.
Hey, that's not fair, you're arguing against half my point!
I said if you use these platforms _and don't have a reason to think you've got an advantage_ you shouldn't be doing so.
Lun and his peers are an exception to this. If you want to make a second argument and say he shouldn't be trading go ahead (and I'll try to back you up), but my original point was about the 99.9% who aren't Lun and are clearly just fish.
Don't get impressed by credentials and buzzwords. A lot of people had and are having success without credentials. The investing, trading domain is large enough to accommodate different approaches and wide variety of skill set. It is not the domain of a few chosen or privileged ones.
Incorrect for the reasons pointed out above, they're trading size, you're likely not, it's a completely different game for them in a completely different field due to their liquidity needs. They are not your competition.
This has been on my mind recently as someone who was in the front office for a few years (but not making trades) but is now on the outside.
I mean each day 100's of Phd's start with clean market
data, more data sources than you could possibly think of
and statistical back testing systems that have 1000's of
man hours put into them, trying to find a way to make money.
It seems to me that the Tiger Rule applies here. You don't have to outrun the tiger, you just have to outrun your buddy.
Is the choice really that one can be in the cohort you mention, or you can buy an index fund (or whatever the equivalent is in the market you're interested in)... and that's it?[1] Is the market so efficient that there is no middle ground where a smart and methodical person can make more money than the index plodders without being obliterated by the big players?
That seems really unlikely to me.
[1] (I don't think that's what you are saying... but you've given me a chance to try and express something I've been thinking about. Thank you for that)
Don't think in terms of there being one tiger that stops as soon as it gets something to eat. The market's more like a sea full of countless sharks, where all the sharks have to survive by eating other sharks.
You're not wrong to think that, if there's any way to consistently make money by trading, it's by exploiting inefficiencies in the market. But remember that those inefficiencies come from people. Are you confident enough in your skills, your education and your resources to be sure that you'll be one of the ones finding and exploiting inefficiencies, rather than one of the ones who's creating inefficiencies for others to find and exploit?
you can make more money, you just have to take more risk, and it takes being quite knowledgeable, experienced, and careful to know what a good risk is and how to diversify it/hedge it.
As long as the target you're optimizing for is different than what those 100 of PhD's (and most of the market) are optimizing for then maybe you've got a chance. You also don't need to pay their salary.
So I agree wrt/ to longer time periods. Most of those PhD's are probably trying to predict minute to minute moves, or daily moves. A lot of them would be out of a job if they lose over 90 day periods.
Not sure how much of an edge can be gained over buying and holding some reasonably diverse equity portfolio. If anyone is thinking they'll beat some HFT hedge fund they're almost certainly not going to. If they're thinking of beating a reasonably diverse/optimized portfolio over longer terms that's maybe possible IMO but the difference isn't going to be huge. To amplify the difference requires taking on more risk, for example by leveraging, options etc.
If I want to decide whether long term investing in the US vs. Turkey, Greece, Brazil, Russia, or the UK, or gold :) it's not clear machine learning can give me useful insight. The historical data has its limits. Predicting geo-political processes and things like interest rates for the long term seems like a very difficult problem. I would suspect the machine answer to this question would be something like P/E is lower so it's a good investment but who knows. So I agree with the economic/market insight comment as well.
> I would suspect the machine answer to this question would be something like P/E is lower so it's a good investment but who knows. So I agree with the economic/market insight comment as well.
This is spot on (tried it once :).
ML doesn't really help much at the macro level. MPT, perhaps combined with some decent economic insight, is a lot more useful.
Hi Delaney, you're my favorite quantopian. Just watched all the lectures and I think jupyter notebooks are the best way to learn combined with a lecture for nearly any programming related subject.
Thanks, happy they've been helpful. I agree that they're a super powerful teaching tool. We actually got the idea when seeing a prof at Harvard using them to teach the entire class: homeworks, lectures, everything.
This was my experience as well. Doing "quant" trading on the side quickly becomes a second job, as you are always adjusting, training, simulating, and even just observing the models you're using.
It's not impossible to make money though. When it comes down to it, the trade-off for the hours you put in and the money you make only really pays off if you're trading with someone else's money, and lots of it.
The rapid rise in popularity of algorithmic trading among everyday coders, the massive growth of /r/wallstreetbets, and the popularity of Quantopian tells me one thing: Go long on retail brokers (AMTD, ETFC, IBKR)
That was my first thought. If I had the resources, I'd built up that subreddit. Youtube videos. HN posts. Some trading libraries on Github. Some forums. Promotional deals to start out.
Previously there were other pyramid schemes, penny stocks, FX trading forums and so on. As those went out of fashion there is a new cohort of young people with great imagination who think they can beat the stock market. Some surely see a great potential there ready to exploit. They'd be silly not to.
Or to put it more obviously when everyone goes digging for goal, don't follow them, but start selling shovels. Then maybe take some profits and put some ads out in the saloons with stories about awesome finds of massive amounts of gold.
Random idea, but what if you could combine market data with news articles, and (obviously only for the past few years) data from blogs/reddit/hn/twitter/etc.? Sometimes there is fascinating insight to be learned in obscure places on the Internet (hn itself being a great example); I imagine a system that would collect and analyze that kind of information would be quite interesting, despite the huge amount of noise. Such a hypothetical system reminds me of this story: https://www.facebook.com/notes/robin-sloan/julie-rubicon/985...
This has been made many, many times already. Zipline that we are talking about here is created by Quantopian that actually provides aggregated twitter harvesting datafeed free for this year.
Been done tons of times. Some of them make quite a bit of money, others not. The real game-changer is if you're effective at separating out the real signal from the noise (and the noise floor is high in these sorts of feeds).
You would probably have as much luck correlating features like whether your neighbors kid was at the bus stop or had to run to catch it each day and the amount of times your neighbor complained about someone not cleaning up their dog's poop that week. Bullshit random edge case features don't get any better or worse if they come from the internet.
In regular trading(algorithmic or not) you really need some non-trivial insight that for some reason no one else will have.
Yup. Incidentally this is exactly the same effect that makes model selection within a portfolio challenging. Two worthwhile reads on that front: the multiple comparison problem [1] and regression towards the mean [2].
Regardless of your objective - hosting a contest like this or allocating to individual institutions and strategies - you'll always face this problem. Good allocation decisions are tactical, taking both the logic of the strategy and the broader portfolio into account. Bad allocation decisions chase returns.
"Yes, but then... "Finance novice beats hedge fund pros, winning $100k in Quantopian trading contest"
Not sure if you know, but that winning algorithm was taken offline within one month for poor performance. It was probably just overoptimised on past data to win the competition and then failed miserably out in the real market.
Equally, there are many athletes that play at amateur/semi-professional level and suddenly they are top level starts (e.g. Chris Smalling, Jamie Vardy). So yes, it's possible. It's just that the odds are not in your favor :) should you keep chasing your childhood athlete dream because someone did it? Probably not :)
I don't know anything about algorithmic trading, but I'm just wondering, are the people who do make money lots of money out of it, those who have servers close to the data source, who do high frequency trades.
You can make money having by having better access like the HFT firms or by having data not widely available. You can also make money by applying well known principles more intelligently than others. This latter approach usually requires a lot of money. You can't afford retail brokerage costs when you're in a highly crowded and competitive trade.
The best way to make some money as a personal trader is to take advantage of the liquidity premium in one way or another. Because you're trading money in the 5 or 6 figures rather than the 7 or 8 figures, you can take advantage of smaller opportunities without thoroughly distorting the market with your own trades. These smaller trades usually require research and market insight rather than clever algorithms. These trades are usually on financial instruments that don't have a lot of easily accessible data to build an automated trade on top of.
> You can't afford retail brokerage costs when you're in a highly crowded and competitive trade.
This is very important. Most brokerage charge around $7 per trade, which makes high-volume trading very very expensive and prohibitive.
Robin Hood is an amazing alternative that charges nada for trades, and once they have an API[1], I think they'd be a great choice for small-time developers looking to do some (low-frequency) algorithmic trading.
Can you give an example of the kind of trade you can make in that kind of market? Doesn't matter if it's an old one, I'm just curious where to start looking.
algorithmic trading doesn't equal high frequency trading. people conflate these a lot even though they know better. high frequency trading requires algorithmic trading. but algorithmic trading can implement warren buffet or suzi orman's style if thats what you wanted.
Agreed, HFT has become a catch-all term for anyone trying to make a marketing/political statement about trading. Like the term "big data" - it's been used to describe so many things that it no longer describes anything.
Another thing that gets ignored is the the difference between a trading strategy and an execution algorithm.
I wish I could get some rest API and websockets into the debt and credit default swaps market, really inefficient markets that reveal a lot about sentiment
don't need high frequency at all, just pretty average latency actually (for now anyway)
IB probably has this, not sure though. But those data costs are a huge deterrent for me!
CDS and debt are not really inefficient markets. Some bonds are less liquid than others but there is currently a rapid conversion to electronic trading going on. CDS is also a very mature market. Not saying that those markets don't contain signal but they are not very inefficient.
and I think the lack of this perpetuates market inefficiencies. I think insights into CDS can be a leading indicator into equities and equity futures, yet CDS are restricted to OTC markets in the US
CDS isn't an OTC instruments. Swaps are now traded on SEFs like MarketAxess or TradeWeb. Just because you don't have access to the data don't think that professionals don't. But this isn't OHLC data for listed equities. Expect to pay. Actually high quality realtime equity data (ITCH,PITCH,OpenBook Ultra,etc...) costs real money as well.
Who's going to provide that data for free when they can charge a LOT of money for access to it. Ever notice why free data on Google or Yahoo are 15 min delayed?
I know what the exchanges charge for data access, and even if you pay it is in an antiquated format.
Tradier has been providing equity and option tick data for free for years in a clean RESTful API with the capability of websockets and streaming. So thats the answer to your question of who.
Thats part of their business model. I have not used their data but I can tell you that all tick data is not created equally.
We are unlikely to ever see CDS data for free since individuals cannot trade them as most retail investors are not Eligible Contract Participant. Some brokerages give away data to get people to trade with them so there its about customer acquisition.
You're pretty much spot on sans the "clean market data" part. I work for a very large electronic trading firm (and have been in HFT professionally the past 8.5-9 years of my career) who has a team dedicated to just grooming this data. It is a lot of work and it is noisy. Coming from some exchanges, it is even often wrong. Look at the entire mess the recent leap second did to some exchanges.
Can't find the original comment but somebody wrote here time ago that some HFs let you run on their infrastructure (keeping 85/90% of any profits) if you can prove to have a valid strategy. Just out of curiosity, I could not find any other information about this, is it a thing that actually happens?
There are places (like Quantopian, but also more professionally-oriented) that will not only provide infrastructure but also capital if you come to them with a trading algorithm that does well on back tests.
The next step down are firms that'll give you a platform including hardware and software for the "infrastructure" bits: often called an "algo container", but lots of brokers have an API you can use to avoid needing to write feed handlers, etc. eg. Pico.
Then there are heaps of providers who'll rent you servers, connectivity, rackspace, etc. You do all the software. Lucera is a trading-oriented "cloud" provider. OptionsIT or Fixnetics are infrastructure providers. Or you just go straight to the data centers -- any decent finance-oriented datacenter will have a POP for most of the venues, and you can just cross-connect.
No relationship with any of the firms named -- just examples off the top of my head.
Yep. I am currently doing this, but under an NDA, at an HFT firm.
80% of profits seems a bit rich, though. If they're supplying top spec colo servers, a super fast network to connect them, and people to stare at it, I'd say something a bit lower (but still pretty chunky) would make more sense.
I suppose the key for them is they have a bunch of existing infrastructure that pays for itself, so any extra cream is good. Also they have capital sitting around, so why not?
The strategies have to be more than unusually profitable, though. I don't think I've seen a down day in the 3 months I've been sitting here, so your average trend following sharpe ~1.5 or so ain't gonna impress people.
I agree with you. I have spent my nights and weekends into trying to find an edge and am still looking. As a retail trader, it is very difficult to find anything but random data all over. I hope I will prove myself wrong someday, but as of now, I think a retail trader finding an edge is nearly impossible.
Disclaimer(s): QuantStart.com founder here, background as a quant dev at a small fund.
I should probably nuance my statement that it is easy to find trading strategies by saying that it is easy to find new trading /ideas/. There are a huge number of freely available trading ideas on forums, pre-print servers (arXiv, SSRN), blogs etc. The trick is knowing how to implement them properly, accounting for any transaction costs and adjusting the parameters of the model. This is often where the stated performance falls down. It takes a lot of time to carry out this sort of research.
Long-term profitable strategies are tricky to find, due to the ever-present spectre of "alpha decay". This is where your strategy's edge is "arb'd out" - everyone else knows what you're doing and so there's no tradeable edge anymore. Hence it is necessary to have a portfolio of strategies and gradually phase out the ones that aren't doing well, and bring in new ones over time.
That being said there are a large number of trend following funds (known as Commodity Trading Advisors, or CTAs, in the industry) that all broadly do the same thing (follow "trends" in the commodity futures markets) and have great years every now and then. There are some well-known "retail" quant traders who do well by trend following, but it does require quite a bit of capital to trade in futures.
The philosophy that I do try to emphasise is to always be learning and researching new ideas. Also, as you mention, I'm pretty keen on discussing the math(s)/statistics aspect because once you have a solid math capability, it is easier to see where potential edges might exist and how to really assess whether it is a true "edge" or just a statistical anomaly.
I believe someone else in a grandchild comment below said that there are many areas that bigger quant funds won't touch because of institutional incentives. If you have $10bn assets under management (AUM), then you're not going to care about investing $100-200k, even if the returns are good, because it won't move the needle on your monthly reports.
The trick is to niche down into markets that you can spend a lot of time researching to find a distinct edge, that won't likely be touched by bigger funds. One area that is becoming interesting recently, due to the prevalence of satellite data/AI/deep learning-esque VC-backed startups, is building commodity supply/demand models. A good example is forecasting oil supply/demand by analysing large quantities of storage tank heights in global refineries [1].
Also, a small related-to-Zipline plug: I've recently started a free Python-based MIT-licensed open-source backtester [2], predominantly as a learning tool for programming and quant trading. There's about 4-5 of us working on it at the moment and it's in an early alpha stage, but we're always looking for people willing to help.
I never understood this. If its possible to be a profitable independent day trader, and we know it is because many are, then it should be possible to code the rules you follow and become a profitable algo trader.
> If its possible to be a profitable independent day trader, and we know it is because many are, then it should be possible to code the rules you follow and become a profitable algo trader.
There's actually a logical error here: if trading results were essentially random, a certain subset of traders (including day traders) would, at any given time, have profitable records. But you would not be able to derive (and, therefore, implement in code) any set of rules which would make automated trading profitable. You would just either be lucky, or not, as an algorithmic traders just as you would as any other kind of trader.
Now, I'm not saying that trading profits are random, but the existence of some profitable traders does not mean that there are rules you can deduce from their behavior that will guarantee profitable trading when implemented by someone else (either in an automated system or otherwise.)
Agreed. Probably a large fraction of 'profitable independent day traders' mention by parent depend on luck. It's one thing to post a profitable month/year, whole different thing is a profitable decade.
It's relatively easy to have a profitable period on a strongly bull market. But it's irrelevant if any 'unexpected' (people call such events unexpected despite the fact that they tend to happen regularly over time) event such as 2008 crisis will completely wipe you out.
> If its possible to be a profitable independent day trader, and we know it is because many are
You're forgetting about the many that aren't profitable. You're also forgetting about the majority that are less profitable than the market average. The latter is probably the easiest to overlook. Yes, they are technically profitable, but so are packaged portfolios of stock held for long periods of time. If you can't be more profitable than someone who puts in zero effort, then whats the point of putting in more work?
A lot of day trading is done based on intuition, which is difficult to translate to code. Also, keep in mind your survivorship bias... Most "day traders" lose money and quit the game.
There are lots of successful software developers too, but writing out a series of rules to automate writing code is not going to happen soon. Unless day trading is substantially easier than programming, an autonomous robotrader that can make money seems unlikely.
Leaving aside the question of how many profitable day traders there actually are, there are profitable professional poker players out there but no bot can compete with them. Now scale up the complexity of poker by at least an order of magnitude and you end up with the financial markets.
Sure, it is possible to to program very niche behaviour, but we are nowhere near any sort of program that can act as general "day trader".
The comments here are interesting...Looking at your question from a different angle, yes, of course it's possible to be a profitable algo trader - set your first investment to buy("SPY") and do nothing else for 20 years. What you are asking though depends entirely on your risk tolerance and what you are benchmarking your strategy against.
You vastly underestimate the complexity of a discretionary trader's intuition and experience. You cannot just replicate years of human experience with computer code so easily.
This statement is too general. You could of said the same thing about chess, there are chess Grandmasters who devote their lives to studying the game yet computers play chess at a much higher level than any human.
Chess is rational, following a easily understood set of rules, and both players have perfect information. The big problem has always been analysing all future possibilities.
The stock markets are very far from a rational, perfect information game with simple rules.
If you honestly think that living a real human life, with all the concurrent decisions that are simultaneously and relentlessly made on a micro and macro level throughout every second, every day is the same as a single game of chess, then by all means, go trade the stock market and show us how it's done.
An organisation at the scale of IBM was able to create, after many attempts, and vast investment, a computer that can beat Grandmasters. That insight isn't useful to an individual trying to do the same.
There are many strategies that are difficult to encode into an algorithm. When I briefly did day trading independently, my most profitable strategies played on public perception and (over)reactions to news. Those things are very, very difficult to properly automate (though many companies try, with varying degrees of success).
Its not as simple as converting rules followed by independent day trader into algorithms - most of them just do not rely on just technical analysis but also on fundamentals which is hard problem to decipher with all the hype around any stocks in the news and analyst of institutional investors influencing with the media news through various sources.
The best for you to start learning - would be to get a start on how the market works - my suggestion would be - https://www.amazon.com/dp/B000THOD1G/
It's a slightly different thing though, being an independent trader isn't the same as becoming an independent trader.
A trader that is independent may still have advantages that prevent some random programmer from bootstrapping his/her way to also being an independent trader.
Just thinking out loud though, I'm not a trader of any sort.
I get asked quite a bit on how to start doing algorithmic trading and the first thing I always tell people is don't.
I think I've said this many times now but the number of people who come at it with the thinking "I'm a computer scientist. I'll just fire up R or python and apply some machine learning to the markets and watch the money roll in" is staggering.
I mean each day 100's of Phd's start with clean market data, more data sources than you could possibly think of and statistical back testing systems that have 1000's of man hours put into them, trying to find a way to make money.
After all of that if you really want to I wrote this in response to an Ask Hacker News a little while ago
https://news.ycombinator.com/item?id=11352562
TL/DR - focus on time periods greater than a day
- expect to lose money
- expect to take a year to figure out some edge in the market
- most decent trading strategies that a normal person can use come from economic/market insights first and technology second.
The site: https://www.quantstart.com/ is also decent at bringing you up to speed on the math you'll need to know though I believe that the material there oversells how easy it is to find a decent trading strategy.