Stencil Buffer works with lines, but not textures.

[Neil]

Can someone tell me what I am doing wrong here?

I am rendering a scene. The application has two render modes, "Design" and
"Preview" If in design mode, a plane is rendered as a grid of lines. In
preview mode, the plane is rendered as a texture. When I render with no
stencils (Setup and Remove stencil function calls are commented out), both
planes function as desired.

But when I turn on my stencil, only the design mode works properly. In
preview mode the stenciled area becomes washed out, with no visible texture.

Here is my code:

if(mode == Mode.Design)
{
SetupStencil(device);

// Make sure the device is using correct vertex format
device.VertexFormat = CustomVertex.PositionColored.Format;

// Set the device index buffer and vertex buffer
device.Indices = DesignPlaneIBuff;
device.SetStreamSource(0, DesignPlane, 0);

// Draw the grid plane
device.DrawIndexedPrimitives(PrimitiveType.LineList, 0, 0, 596, 0, 300);

RemoveStencil(device);
}
else
{
SetupStencil(device);

// Set the devices index buffer and vertex buffer
device.Indices = ProductPlaneIBuff;
device.SetStreamSource(0, ProductPlane, 0);

// Draw the textured plane
device.DrawIndexedPrimitives(PrimitiveType.TriangleStrip, 0, 0, 44848,
0, 44846);

RemoveStencil(device);
}

private void SetupStencil(Device device)
{
//Enable the stencil buffer/tests
device.RenderState.StencilEnable = true;
device.RenderState.StencilFunction = Compare.Always; //always
pass the stencil test

//Set the render states as to what will happen if individual
states fail.
device.RenderState.StencilPass = StencilOperation.Replace;
device.RenderState.StencilWriteMask = -1;

//Disable writes to the zbuffer and disabling writes to the
color buffer
device.RenderState.ZBufferWriteEnable = false;
device.RenderState.AlphaBlendEnable = true;
device.RenderState.SourceBlend = Blend.One;
device.RenderState.DestinationBlend = Blend.One;

//Render the polygon to the stencil buffer (masking the
bits)
CustomVertex.Transformed[] verts = new
CustomVertex.Transformed[PointList.Count];

for (int i = 0; i < PointList.Count; i++)
{
verts[i] = new CustomVertex.Transformed(PointList[i].X,
PointList[i].Y, 0f, 0f);
}

// Set vertex format
device.VertexFormat = CustomVertex.Transformed.Format;

//m_pd3dDevice->DrawPrimitiveUP(D3DPT_TRIANGLESTRIP,6,verts,sizeof(rhwTUVert));
device.DrawUserPrimitives(PrimitiveType.TriangleStrip,
PointList.Count/3, verts);

//Enable depth buffer and back buffer writes
device.RenderState.ZBufferWriteEnable = true;
device.RenderState.ReferenceStencil = 0xf;// 0x7f; //The
reference value to test against the value in the stencil buffer
device.RenderState.StencilFunction = Compare.Equal;
device.RenderState.StencilPass = StencilOperation.Keep;

// Set appropriate blending states
device.RenderState.SourceBlend = Blend.SourceAlpha;
device.RenderState.DestinationBlend = Blend.InvSourceAlpha;

}
private void RemoveStencil(Device device)
{
// Reset render states
device.RenderState.AlphaBlendEnable = false;
device.RenderState.StencilEnable = false;
}

[legalize+ (Richard [Microsoft Direct3D MVP])]

[Please do not mail me a copy of your followup]

Post by Neil
But when I turn on my stencil, only the design mode works properly. In
preview mode the stenciled area becomes washed out, with no visible texture.

The problem isn't with stencil, its with how you're changing the alpha
blending state. In other words your function is called
"SetupStencil", but it does a lot more than change stencil state.

[Neil]

I didnt realize the alpha state had nothing to do with the stencil. I was
following a tutorial, and it did not explain stenciling very well, only gave
some example code.

So I removed the lines modifying the AlphaBlendEnable, and now the texture
shows up as a solid grayish brown color, which is probably what the texture
would look like if it were reduced to a solid color.

What exactly is the significance of the ReferenceStecil and the
StencilWriteMask values?

[legalize+ (Richard [Microsoft Direct3D MVP])]

[Please do not mail me a copy of your followup]

Post by Neil
I didnt realize the alpha state had nothing to do with the stencil.

The device is a big data pipeline with all the bits of state
configuring and controlling the pipeline. Check out my pipeline
poster to see how all the data flows through the device:
<http://www.xmission.com/~legalize/book/preview/poster/index.html>

Post by Neil
I was
following a tutorial, and it did not explain stenciling very well, only gave
some example code.
So I removed the lines modifying the AlphaBlendEnable, and now the texture
shows up as a solid grayish brown color, which is probably what the texture
would look like if it were reduced to a solid color.
What exactly is the significance of the ReferenceStecil and the
StencilWriteMask values?

You might want to read my chapter on the frame buffer:
<http://www.xmission.com/~legalize/book/download/14-Frame%20Buffer.pdf>

That will explain the stencil buffer and alpha blending. Texturing
itneracts with alpha blending because texturing can be the origin of
alpha values for source pixels. Source pixels and destination pixels
are combined into the render target through alpha blending.

[Neil]

Thank you for the detailed response, I have already made progress
using your input. Turns out the main problem wasnt the stencil at
all, but I had the vertex format set wrong prior to stenciling. Once I
set the vertex format to PostitionTextured (and also remembered to set
it back to Transformed before stenciling) It began functioning as I
desired. There are still a few weird cases but I will get to those
after reading your book chapter.

Your site looks very informative, the best explaning Direct3d I have
found so far. Is your book out or when is it expected out? It looks
like a book I would definetly put on my desk

[legalize+ (Richard [Microsoft Direct3D MVP])]

[Please do not mail me a copy of your followup]

Post by Neil
Your site looks very informative, the best explaning Direct3d I have
found so far. Is your book out or when is it expected out? It looks
like a book I would definetly put on my desk

You can get the completed chapters of my draft manuscript at the link
in my URL. Its not been published yet.