Yeah, I think that'll happen. I think it's already happening. And if someone wraps it in a terrible app with ads, makes millions of dollars, and I never see any of the money, yes I'll be very sad. But I'm trying to take a bigger perspective here. There's a lot of experience and tools that could be built off this, and I can't possibly build them all.
And if I hid the code away so only I could use it, there's a good chance no one would ever create any of those experiences. Creating a successful app requires mobile UI people, devops, designers, marketing/SEO, monetization, etc. I don't want to do ANY of that... in the land of app creation, I'm more of a Tom Bombadil-type.
This is also my first project out of graduate school. I feel like I'll probably have other ideas down the road, and maybe I'll try to monetize those ones with what I've learned from this adventure.
This is AMAZING, my 3 year old draws the cutest pictures and I can animate them!
One thing with the browser version, it all works fine up until the animation and then its flipping my image upside down? Could this just be the photo metadata rotation/orientation??
It’s not metadata, you would see that when you upload the photo at the beginning.
I’m guessing that the nose key point is located below the shoulders in your child’s drawing?
That’s a known issue caused by how we ‘apply’ poses onto the characters. It’s fixed in the GitHub code but not in the browser version. Try moving the nose keypoint to the top of the characters head and that should resolve it
I assume you're referring to the demo at sketch.metademolab.com?
The server is returning an .mp4 file that is displayed by your browser, so you should be able to download it. On Chrome, if I maximize the video and click the three vertical dots, I get the option to download the video.
Is it actually possible to use the code and create an app for sale out of it. The MIT license says so I think, just wondering if anything else would speak against it.
Phenomenal. For what it’s worth, I think this would be a killer app for my phone. My kids would easily spend hours drawing and animating their creations.
that little toy I had as a kid did more to shape my career as than even hypercard.
FTA (and relevant):
> What truly made this the best toy ever made was the amount of patience and perseverance a kid needed to create these animations, just like real animators. This is also why it would later fall from popularity.
Hah! Cool to meet another kid who grew up with that. I wish there were more actual devices like that now, with their own weird UIs and knobs and buttons, instead of everything being an app mixed in on a tablet with endless other apps. When you think about it, what a far-out thing for a company that made Etch-A-Sketches to engineer and sell...
Yeah, the limited memory was a shortcoming. Still, in a way, that made it more in line with the notional paradigm of the Etch-A-Sketch... doing the drawing is the fun, but eventually you have to erase it and start over.
It's cool that this is possible, but it's missing some very important subtleties which make professional animation look "right" or even "real". As it is, there's no sense of compression or snap at the right parts of the movement. 2D character animation has a lot of exaggerated distortions of objects to make them look like they land on the ground or bounce. The sense of weight and inertia is critical.
These aren't necessarily great, but this page[0] does touch on some of the basic concepts. "Squash and stretch" is a key thing missing. Also while in the demo you're dancing not walking, a walk cycle is a very basic thing which is often done poorly by even supposedly professional companies. [1] It's really important for the "contact" part to seem like it's happening. Getting feet to plant themselves on the ground and look like they're not swimming or sliding around is a challenge.
[my father worked in animation for much of his career and would complain bitterly about badly done animation, so I guess I can't unsee things with his critical eye]
I'm a student of good animation myself (know the 12 principles of animation by heart, got a copy of The Illusion of Life right next to my desk, etc.) and have great respect for what professional animators can do.
And while I (maybe more than anyone else) am highly aware of all the ways this tool doesn't create high-quality animation, I'm okay with it. The point here isn't to create animations that rival those of professional animators; it's to lower the barrier-to-entry to the point where almost anyone can animate their own character.
And if a tiny number of users become more interested in animation through this tool, learn how to do it the proper way, and create animations your father would approve of... well I think that would be a phenomenal outcome.
It's a bit buried in the repo readme files, but the author @hjessmith used Rokoko Video [1] to create the animation files (BVH) from a phone video. Then he used Animated Drawings to make the drawings replicate his motion captured from the video.
Both Rokoko and Animated Drawings were new to me. I know what I'll be doing with my kids all weekend! Thanks for sharing the project.
Completely tangential point - Rokoko Video is fine for a free tool, but if you need something better and you're willing to pay, there are other options. I spent a couple of days evaluating various video-only mocap and in the end went with DeepMotion.
Rokoko Video wasn't very reliable (at one point I was standing perfectly still, but it thought I was on one leg twirling like a ballerina). Although it's very expensive, DeepMotion has been great (also if you were disappointed by DeepMotion 12+ months ago like I was, I'd suggest revisiting - it's leaps and bounds ahead of the earlier version I tried out).
move.ai looks very good too, but needs multiple cameras which was overkill for my use case. Plask seems to have regressed (the platform seems like it doesn't even work half the time) and Radical's aggressive tactics really turned me off.
The free tier only gives you 60 seconds of animation per month, and also isn't licensed for commercial use. Even for a hobbyist, 60 seconds wont go far!
Out of all the ML projects submitted to HN this is one of the funniest and most entertaining I've seen so far! Also very neat demo and presentation in the website! KUDOS to the author(s) of the implementation and paper!
This is a great project and it was fun playing around with it.
When Meta started discussing this in 2021, the HN sentiment [1] was quite...cautious, making connections to how Meta can't be trusted with kids drawings. Has something changed?
I mean it gets exhausting. We had Covid (and Q), Musk taking over Twitter, war in Ukraine, fear of war in Taiwan, ... accelerated climate change and so on. There are real limits to how many things one can be upset about.
Edit: My bad it has to do with my Adblocker. Its all good now. Cheers
Super cool! Thank you for bringing this to the world. Kids are gonna love this. I'm unable to get it running on the web demo unfortunately though :( Post image upload, I'm redirected to https://sketch.metademolab.com/canvas with a blank screen with the grid image. Without any error message, it isn't obvious if the issue is with the images I uploaded ( which probably is the case ). Will try the Python library and play around.
I want to animate video game characters with as much automation as possible because hand drawing frame-by-frame takes forever and ever. Is this my best bet?
It currently only outputs .mp4 and .gif files. You could probably use these as game assets, but I don't think it would be optimal.
The rendering code utilizes an MVC framework. If you need the animation output in a .obj format or something, it's as simple as creating a new View that supports it. Feel free to request this in a Github issue if you want me to prioritize it.
Currently it only allows for animating a character generated from a prompt. However, we’re working to use null text inversion [0] combined with inverse DDIM (used for pix2pix) to allow animating a pre-drawn character.
Small anecdote: I discovered the codebase for this project last week and assumed that it had already been advertised and popularised. Apparently that's just happening now.
I deployed the Docker container and got it working fine and I'm already thinking of plenty of cool use cases for the code.
I get the impression that a few startups / apps are going to be launched off of the back of this. A few ideas:
- Animated story book using GPT (I'm working on)
- Animated greetings cards using photos (works really well)
Glad to hear it was easy for you to get up and running, but where did come across the repo? It's been in 'stealth release' mode for a few months now, but nothing was publicly announced until today.
We are working on a startup (https://dibulo.com) which provides children an off-screen coloring experience combined with a digital reward. We have so many ideas and using pose-estimation is one of them. Definitely gonna play around with it!
So great to see more work on pose research! My ultimate dream is to have a tool where I can give it a cartoon character and a set of pose photos, and have the tool pose the character in those poses. This feels one step closer to that.
Very fun. Is there something like this but which can capture the motion from a webcam (without separate motion capture process as described in the readme) and project that onto the drawings?
As written now, it expects 3D motion data. Extending it to support 2D motion data is doable, but will take time. Feel free to open a github issue requesting it if you'd like me to prioritize it.
Code and dataset are here: https://github.com/facebookresearch/AnimatedDrawings
And a browser-based version of it is here: http://sketch.metademolab.com/