Adding patterns and in depth talk about layer masks

[Avery]

When I make tutorials I often use “layer masks” which are probably something that is not the most beginner friendly thing in the world. Not because they are difficult to use, just because they are a concept that might be a bit difficult to wrap your head around if you never heard of it.
So even if you avoided layer masks until now, there is one major reason to use them: they are non-destructive. That means, anything they do to your image can be undone in no time. Why that is so important is something we will talk about later.

What is a layer mask?
Imagine your image is projected onto the layer it is on by a projector. Now you put a transparent foil in front of the projector - nothing happens. That is the same, as if you would put an all white layer mask on an image: in short, an all white layer mask will have your layer be fully visible.
Now imagine I use a black marker, to draw on that foil - those parts will miss from the projection, as those black markings don’t let the light pass.
If I use a black marker that is not fully opaque (which would lead to grey, if used on a white layer mask), only some of the image comes through, resulting in a semi transparent area, the more opaque the “marker” was, the less we would see.
Now the great thing about this foil is that I can just pull it off, in case I realize I screwed up, and never ever altered anything on the layer I used it on. Or in case I like what I did but need it for a different layer as well, I can simply duplicate this foil and add it to the other layer as well.

How do we make and use a layer mask?
Let us say, we want to make this gentle guy here, Nature_2, be a striped cat with a white belly instead of an all black one:

Before we do anything, we set the color mode from “indexed” to RGB, by going to “Image” - “Mode”. This will ensure we can have any color in this image, not just the preset color palette.
We of course then need to get the layers to put the masks on, so for white we usually start with a “Adjust color curve” to get the needed contrast and here I was also able to adjust the color to my liking via the other curves:

Proceed with the brown layer and here are my results, a little moved so you can see I moved them to the side so you can see them in the order I have them:

Now we right click on the layer we want to start with - here white - and choose “add layer mask”. We could use the layers alpha channel or a premade selection, but in this and most of our RM dev cases, a solid layer mask is a good first step, and I personally like to start with a “blank canvas”, so all black.

After this addition, the layer is still there, but as explained above, nothing of it “shines” through the mask:

Now we ensure, that the “edit layer mask” box is ticked, which means if we now draw on this layer, we actually just touch the mask and not the real image.

On which we use a white pencil to draw in the “white belly”, we talked about before (or whatever you are going for).
This is my layer mask:

and this is how the image looks like as of now:

If you want to check how your layer mask looks, you can either right click on the layer and use “show layer mask” or shift+click on the layer mask.

Now we proceed with the brown layer, it also needs a layer mask that has white parts where the brown fur stays and black parts for everything else:

Now imagine, we might have used the eraser instead of a layer mask and now decided, that we would want some additional white spots, we can just draw them in on the layer mask, if we erased these parts, they would be gone and if we did not save the layer unedited somewhere, we had to try to get it recolored to exact the same shade again so these spots would not stand out - not ideal.

Now there is a problem left: the outlines were not solid, but have anti aliasing by default, and the recoloring lead to some weird color differences and more solid lines, as there multiple semi transparent pixels add up.
How do we tackle such a problem?
We use the “select by color” tool on the transparent background of the default, unedited Nature_2 layer that is our bottom layer. On that selection, we use “Select”- “Grow selection” and then, for MZ busts, 2 is a good value:

The result is this:

We have all the transparent areas selected, +2 pixels, which means, we have selected the part that should be all black on every layer mask but (in this case not existing one) one bot the bottom layer.
We now bucket fill (fill wole selection) this part on these layer masks with black, and - tada!

or as full picture:

Such “dying in a pattern” just works on a solid color, so if that cat already was black and white as in MV, we would have a much harder time giving them such a makeover.

That being said, this method is also great to give your heroes multi-colored hair, dye a pattern into clothing or add scars to faces!

For example, here I did it all at once!
And without any problems, I could scribble onto the layer masks, until I got everything I wanted to be there and got rid of everything that I did not!

So hopefully this motivated you to add layer masks into your edit routine, they are a ofter underused tool that helps a lot!

 

[ShadowDragon]

I use layerMask sometimes, but its not exactly equal if they use Photoshop,
but is close.

the only thing I do a slightly different just in case, I copy the image, hide it,
and work on the copy one, to set a layer mask, if something goes wrong,
I always have a backup.

but there are some things I didn't know off, but I did more text effects than
graphics, but this has some cool things I need remember, as it can be used on
loads of things with little work.

thanks again for a usefull tutorial with tips