Because the new version of PS tools has built-in AI-driven tools, such as “sky replacement” and other difficult image editing problems, it can be easily achieved with a click of the mouse, and the effect is far beyond manual repair.
Whether shooting people or scenery or anything else, the “sky” is a key element of photography. Isn’t it more flavorful if you add a picture of a plain landscape with the afterglow of the sky tones? For short film lovers, wouldn’t it be so happy to fly to the sky if the film can be modified to reach this level?
That’s right, today I’m going to introduce an AI processing tool based on native videos that can not only change the sky background with one click, but also create a variety of “Sky City”.
New ways to play AI movies This AI processing tool comes from the latest research of a Chinese postdoctoral fellow at the University of Michigan. Based on visual technology, it can adjust the sky background of the film and change the weather with one click.
For example, “StarCraft” and other sci-fi movies often appear in the vast starry sky and spaceships, and this technology can also be used to incorporate casually shot movies. The road movie becomes a science fiction movie in seconds, and there is no sense of contradiction. The blue sky of the film also turned into a gray tone with the spaceship, and the sense of doomsday came out immediately.
Of course, the gameplay is more than that. Anime fans can also create their own mobile castles. Those who like “Castle in the Sky” and “Hall’s Moving Castle” should be very familiar with the following scene. Or hang a super moon, it’s another scene. As long as the brain is big enough, using this AI technology, there are unlimited ways to play in film creation.
There is also a weather switching function, such as various weathers such as sunny skies, rainy and thunderstorms, etc., which can be switched at will. Are friends who like to play Vlog moved? Researchers said that they have now considered making plug-ins/scripts to facilitate industry or personal use.
Prior to this, the AI code of this technology has been open sourced on Github, readers who understand the technology can install and play first.
Technical Principle Different from traditional research, researchers propose a solution based entirely on vision. The advantage is that it can process non-static images without being restricted by shooting equipment, and without user interaction, and can process online or offline movies.
The above experimental videos were all shot in the field using smartphones and driving recorders. After processing, the film quality, motion dynamics, and lighting conversion all have high fidelity. For example, the floating castle and the super moon example, using a single NVIDIA Titan XP GPU card, can output a real-time processing speed of 640×320 up to 24fps, and a real-time processing speed of nearly 15fps at 854×480.
This tool is divided into 3 core modules:
Sky Matting Network: A film frame that detects the sky area of the film frame. Using prediction channels based on deep learning to produce more accurate detection results and a more visually effective sky template. Motion Estimation: A motion estimator that restores the dynamics of the sky. Sky movies need to be rendered and synchronized under real camera motion. Image Blending: Blend the sky template specified by the user into the Skybox of the movie frame. In addition, it can be reset and colored to make the mixed result more realistic in color and dynamic range. Sky mask framework: Using the advantages of Convolutional Neural Network (CNN), predicting sky hail under a pixel-level regression framework, it can generate two kinds of sky templates. The sky mask framework consists of a Segmentation Encoder, a Mask Prediction Decoder and a Soft Refinement Module. The purpose of the encoder is to learn to sample the intermediate features of the input image. The decoder trains and predicts the rough sky. The optimization module simultaneously receives rough sky templates and high-resolution inputs, and generates high-precision sky templates.
Dynamic prediction: Researchers directly predict the dynamics of the target at infinity, and create an image-mixed skybox (Skybox). By blending the 360-degree skybox template image into the perspective window, the virtual sky background is rendered.
Assuming that the sky mode movement is simulated by the matrix M 2 R 33, since the objects in the sky (such as clouds, sun or moon) should be in the same position, assuming that the perspective transformation parameters are fixed values and are included in the skybox background image, then use Update Lucas-Kanade and pyramid method to calculate optical flow, and then track a set of sparse feature points frame by frame. For each pair of adjacent frames, given two sets of 2D feature points, the RANSAC-based robust blur estimation is used to calculate the best 2D transformation with 4 degrees of freedom (limited to translation, rotation, and uniform scaling).
Image mixing: When predicting the sky template, the higher the output pixel value, the higher the probability that the pixel belongs to the sky background. Conventional methods usually use image masks to linearly combine the newly synthesized film frame and the background as the pixel-level combination weight.
However, because the foreground and background colors may have different tones and intensities, directly performing the above method may lead to unrealistic results. Researchers applied recoloring and relighting techniques to transfer color and intensity from the background to the foreground.
When synthesizing rainy weather images, the researchers added a dynamic rain layer (video source) and a fog layer on top of the result through screen mixing. The results show that only a slight modification of the skybox template and re-lighting factor can achieve visually realistic weather conversion.