Bug 708274: Tweak antidropout code in the non-AA rasterizer.

In all the normal paths through this code, we apply floorf() to
the coords given to convert to int. Just in the rectangle
case (the specific case that is only used for anti-dropout)
we are currently using a mix of floorf() and ceilf().

This is causing the problem referenced in the bug.

The reason for using floorf and ceilf together is to avoid the
case where the vertical extent becomes 0 - but this is not
actually a problem in this case.

So, we amend the code to use just floorf initially, and we only
resort to using ceilf() if the extent would otherwise be 0.

Thus we still get anti-dropout, but don't affect cases where we
don't need it.

diff --git a/source/fitz/draw-edge.c b/source/fitz/draw-edge.c
index efa30791079c7e51310f54de0148014a3e2b0b20..e3788cdbc92cb323cda5ba6ecdad276f2cc7e5b8 100644 (file)
--- a/source/fitz/draw-edge.c
+++ b/source/fitz/draw-edge.c
@@ -262,12 +262,16 @@ fz_insert_gel_rect(fz_context *ctx, fz_rasterizer *ras, float fx0, float fy0, fl
        if (fy0 <= fy1)
        {
                fy0 = floorf(fy0 * vscale);
-               fy1 = ceilf(fy1 * vscale);
+               fy1 = floorf(fy1 * vscale);
+               if (fy1 == fy0)
+                       fy1 = ceilf(fy1 * vscale);
        }
        else
        {
-               fy0 = ceilf(fy0 * vscale);
+               fy0 = floorf(fy0 * vscale);
                fy1 = floorf(fy1 * vscale);
+               if (fy1 == fy0)
+                       fy1 = ceilf(fy1 * vscale);
        }
 
        fx0 = fz_clamp(fx0, ras->clip.x0, ras->clip.x1);