Incentive
My incentive to do this is that one of my commonly used 4K super-resolution subtitle group stop their activity, and I still want to be able to watch latest 4K anime, so I need to find a way to perform super resolution on existing 1080P anime video resources.
Real-time Shader
There are things like Anime4K and AnimeJanai which could be used as real-time shaders in video players to provide upscaled video watching experience.
I tried Anime4K. It works when the characters are already quite large in the frame as showed below:
However, when characters are small, the performance will become quite poor:
Though it has some drawback, Anime4K is the final choice in my case because it’s really fast and efficient compared to other approaches I tested. Generally, it could nearly achieve real-time processing of video frames, which means for a 20min video clip, it will also take about 20-30 minutes to generate a upscaled video file.
Offline video upscaling could be done using Anime4K GUI, which will produce an upscaled file in the same directory of the original video.
Generative AI Super-Resolution
I do some research and learn that generative AI super-resolution models may perform better for such cases since it could creatively “generate” some details when original resolution is low or original detail is not enough.
I use chaiNNer, a graphical node style image upscaling workflow editor, to experiment this approach.
Below are some models I tried using chaiNNer
- 2x AnimeSharpV4 Fast
- AnimeJaNai
For me it looks like the AnimeSharp model performs better for small characters/details, also we could add more pre/post processing nodes like Denoise and Sharpen to further improve the final result.
However, this approach has a huge efficiency issue: it’s too slow. For example, when using AnimeSharp model, it takes nearly 3 to 4 seconds to generate one single frame, this means it will cost more than a whole day to upscale a 20-minute-long anime video.
Video Upscale Model
I found SeedVR2 when searching about video upscaling, and done experiment in ComfyUI:
I give up this approach because it’s nearly unusable on personal laptop/PC, on my laptop with RTX3060 laptop GPU, it will approximately take about 1 hour to upscale a 3-second test clip.