Small triangles could probably make a mess with epsilon, so perhaps that could be the issue. As far as I know, the only exceptions caused by a bad epsilon are StackOverflows caused by Node.build() calling itself, so if that’s the exception you’re getting then the epsilon may well be inappropriate for your small triangles; try changing the epsilon and see if that helps.
When you preform a CSG operation with FabianCSG, the triangles on the mesh will be broken down in to many smaller, unnecessary, triangles. This might add to any issues you were having with small triangles, and I found it rapidly slowed down the speed of operations as I preformed multiple subtractions on the same mesh; my second and third operations took many seconds. Could this be the source of your freezing?
I had a go at implementing the idea of checking the vertex type using floats and then double checking with BigDecimals using the code below. This worked with the [two cubes test][1] above, but not using a torus or with some other tests I tried (the application froze for several minutes, and eventually closed and logged a StackOverflow of Node.build()). I’m not currently working on that, so here’s the source code for what I have so far:
In Plane.java split():
int polygonType = 0;
int[] types = new int[polygon.getVertices().length];
for (int i = 0; i < polygon.getVertices().length; i++) {
Vector3f p = polygon.getVertices()[i].getPosition();
float t = normal.dot(p) - w;
int type = (t < -EPSILON) ? BACK : (t > EPSILON) ? FRONT : COPLANAR;
if(type == COPLANAR){
Vector3b a = new Vector3b(points[0]);
Vector3b b = new Vector3b(points[1]);
Vector3b c = new Vector3b(points[2]);
Vector3b n = b.subtractLocal(a).crossLocal(c.subtractLocal(a)).normalizeLocal();
BigDecimal bw = n.dot(a);
if(flipped){
n.negateLocal();
bw = bw.negate();
}
BigDecimal d = n.dot(new Vector3b(p)).subtract(bw);
type = (d.doubleValue() < -SMALL_EPSILON) ? BACK : (d.doubleValue() > SMALL_EPSILON) ? FRONT : COPLANAR;
}
polygonType |= type;
types[i] = type;
}
Other changes to Plane.java:
public static final double EPSILON = 1e-2;
public static final double SMALL_EPSILON = 1e-5;
private Vector3f normal;
private float w;
private Vector3f[] points;
private boolean flipped = false;
public Plane(Vector3f normal, float w, Vector3f[] p) {
this.normal = normal;
this.w = w;
points = p;
}
public static Plane fromPoints(Vector3f a, Vector3f b, Vector3f c) {
Vector3f n = b.subtract(a).cross(c.subtract(a)).normalizeLocal();
return new Plane(n, n.dot(a), new Vector3f[]{a, b, c});
}
public Plane clone() {
return new Plane(normal.clone(), w, points.clone());
}
public void flip() {
normal = normal.negate();
w = -w;
flipped =! flipped;
}
org.fabian.csg.Vector3b.java:
package org.fabian.csg;
import com.jme3.math.Vector3f;
import java.math.BigDecimal;
import java.math.RoundingMode;
public class Vector3b {
public BigDecimal x;
public BigDecimal y;
public BigDecimal z;
public Vector3b(BigDecimal argx, BigDecimal argy, BigDecimal argz){
x = argx;
y = argy;
z = argz;
}
public Vector3b(Vector3f v){
x = new BigDecimal(v.x);
y = new BigDecimal(v.y);
z = new BigDecimal(v.z);
}
public Vector3b subtractLocal(Vector3b v){
x = x.subtract(v.x);
y = y.subtract(v.y);
z = z.subtract(v.z);
return this;
}
public Vector3b crossLocal(Vector3b v){
BigDecimal tempx = y.multiply(v.z).subtract(z.multiply(v.y));
BigDecimal tempy = z.multiply(v.x).subtract(x.multiply(v.z));
z = x.multiply(v.y).subtract(y.multiply(v.x));
x = tempx;
y = tempy;
return this;
}
public Vector3b normalizeLocal(){
BigDecimal length = x.multiply(x).add(y.multiply(y)).add(z.multiply(z));
double lengthD = length.doubleValue();
if (lengthD != 1 && lengthD != 0){
length = sqrt(length, 1000);
x = x.divide(length, RoundingMode.HALF_UP);
y = y.divide(length, RoundingMode.HALF_UP);
z = z.divide(length, RoundingMode.HALF_UP);
}
return this;
}
public BigDecimal dot(Vector3b v){
return x.multiply(v.x).add(y.multiply(v.y)).add(z.multiply(v.z));
}
public Vector3b negateLocal() {
x = x.negate();
y = y.negate();
z = z.negate();
return this;
}
private static BigDecimal sqrt(BigDecimal A, final int SCALE) {
BigDecimal x0 = new BigDecimal("0");
BigDecimal x1 = new BigDecimal(Math.sqrt(A.doubleValue()));
while (!x0.equals(x1)) {
x0 = x1;
x1 = A.divide(x0, SCALE, RoundingMode.HALF_UP);
x1 = x1.add(x0);
x1 = x1.divide(new BigDecimal(2), SCALE, RoundingMode.HALF_UP);
}
return x1;
}
@Override
public String toString() {
return "(" + x.stripTrailingZeros() + ", " + y.stripTrailingZeros() + ", " + z.stripTrailingZeros() + ")";
}
}
[1]: BoolMesh.java (boolean operations on mesh) - #29 by Plerdi