[SOLVED] Project a PNG on a mesh

Thanks sgold for your explanation but it’s too high for me :frowning:.
Everything what I want is: like in a game. If you shoot on a wall then a whole should be shown at this poisiton.
Or maybe there’s an easier Possiblity to do that. I can draw it on plan and then rotate it. Maybe in the direction of normal of hit point.

1 Like

sure, you can just do rayCast and place Quad with circle based on normal of contact point.

But i thought you need paint circle in this model in certain static location?

now when you told “If you shoot on a wall then a whole should be shown at this poisiton.”
i think like i misunderstood what you want. because i didnt expect you want dynamic location of it calculated from contact point. if you want to have circle in point where your mouse(or gun or whatever) is pointing, then do like you said(raycast and place quad - rotate based on normal) or use projection i linked post(third topic post. but it also migh be harder anyway).

You’re right. I need it in a static location. But the approach is the same. Either the position comes from a file or by contact point makes no difference. Do imagine the following. I have scanned human body by a 3D scanner. On this body there are some stickers. These stickers are regonized by a previous software. I just get the mesh as STL file and a file with the coordinates of these stickers in the room(not the coordinates on the body).
Now I want to visualize the stickers on my model so that user can see it.
Unfortunately, my 3D programming skills are really low level and currently I haven’t the time to improved my skill to understand UV mapping and so on. I thought there’s an easy way to this.

i almost understand all, but:

stickers in the room(not the coordinates on the body)

you said you have sticker location from previous software, but where are exactly this coords? where are this stickers? they are “on body”? or somewhere near body?

lets assume one of stickers is located “near head” in space. (with distance 1f from head).

then anyway you can get 0f distance point “on head” in space just by raycasting to closest point on model. (you can use verticle/tris positions of model for example to get closest point).

then you can do 3 options:

  • option 1 - create Quad with transparent sticker and rotate it properly based on raycast normal - problem of this option is that sticker will not “wrap model”, it will be quad.
  • option 2 - use ParallelProjection i post in third topic post, from sticker location to model closest point too. - problem is that i didnt use it and i dont know if it will be easy, so i cant tell.
  • option 3 - get closest point, but here get closest verticle and check its UV coords, then add sticker image in UV coords that this verticle have - problem is that UV might be deformed and instead of circle you could have some oval shape. (maybe sgold have idea, but i dont)

so based on my knowledge option 2 will look best(but also if for example hand would be before chest, then sticker would be on both hand and chest so also have some issues), option 1 is fastest to do, but will not wrap model correctly and option 3 can have deformed stickers. so i would choose option 1 or 2.

ofc im not sure what Sgold want to achieve with UV idea, maybe he have idea for fix deformed images in UV. He is clever so maybe he have solution for it.

but i just say based on own knowledge :slight_smile:

You got it right. The sticker coordinate is just a point(x,y,z) in “room”/coordinate system and must not be match with model. It depends on the accurancy of the sensor. The point of the sticker isn’t excatly on the model(human body) but very close to it. The sticker represents a body point like knee, hip or waist.
I tried Option 1 and it looks very bad due to inaccurancy of the sticker coordinates and the skin imperfection. The sticker is often disappearing in the model or too far away from it.
Option 2 would be the best but I thnik is too hard to implement.
Option 3 is the best in my opinion. Only thing I have to do is to find some point in the near of my sticker and calculate the uv-coords and add texture to this region. A not perfect circle isn’t a problem. I think the distortion is low.

But in theory:

  1. I have to figure out all closest tris of my sticker point.
  2. With this tris I would create new mesh
  3. For this new mesh I calcluate UV-coords and add my texture to it

I think 1 is possible for me and for the calculation of the UV-Coords I would use Sgold function.

Ok, here’s my first try. I can find all closest point around my hit point but the texture isn’t correct. I think that’s due to the wrong uv coords?

    public static void calcUV(Mesh mesh) {
		FloatBuffer positions = mesh.getFloatBuffer(VertexBuffer.Type.Position);
		int numVertices = positions.limit() / 3;
		FloatBuffer uvs = BufferUtils.createFloatBuffer(2 * numVertices);
		mesh.setBuffer(VertexBuffer.Type.TexCoord, 2, uvs);
		for (int vertexI = 0; vertexI < numVertices; ++vertexI) {
			float x = positions.get(3 * vertexI);
			float z = positions.get(3 * vertexI + 2);
			float u = 12f * x;
			float v = 12f * z;
			uvs.put(u).put(v);
		}
		uvs.flip();
	}

	public static Geometry markHitPoint(Mesh mesh, AssetManager assetManager, CollisionResult result) {
		Vector3f point = result.getContactPoint(); 
		 point.multLocal(1000);
		 point = new Vector3f(point.x * -1, point.z, point.y);
		Mesh newMesh = new Mesh();
		float maxDistance = 20;
		List<Vector3f> position = new ArrayList<>();
		List<Vector3f> normals = new ArrayList<>();
		List<Integer> indices = new ArrayList<>();

		for (int i = 0; i < mesh.getTriangleCount(); i++) {
			Triangle tri = new Triangle();
			mesh.getTriangle(i, tri);
			boolean isClose = tri.get1().distance(point) < maxDistance || tri.get2().distance(point) < maxDistance
					|| tri.get3().distance(point) < maxDistance;
			if (isClose) {

				for (int j = 0; j < 3; j++) {
					Vector3f v = tri.get(j);
					int index = 0;
					if (position.contains(v)) {
						index = position.indexOf(v);
					} else {
						index = position.size();
						position.add(v);
						normals.add(tri.getNormal());
					}
					indices.add(index);
				}
			}
		}
		newMesh.setBuffer(Type.Position, 3, BufferUtils.createFloatBuffer(position.toArray(new Vector3f[] {})));
		newMesh.setBuffer(Type.Normal, 3, BufferUtils.createFloatBuffer(normals.toArray(new Vector3f[] {})));
		newMesh.setBuffer(Type.Index, 3,
				BufferUtils.createIntBuffer(indices.stream().mapToInt(Integer::intValue).toArray()));
		calcUV(newMesh);

		Geometry geo = new Geometry("OurQuad", newMesh);
		Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
//		mat.setColor("Color", ColorRGBA.Yellow);
		
		Texture2D tex = (Texture2D) assetManager.loadTexture("point.png");
		mat.setTexture("ColorMap", tex);
		
		geo.setMaterial(mat);
		System.out.println("called");
		return geo;
	}

The point.png is little blue circle of height and width of 40px. It’s working so far but the texture doesn’t do it.
Please don’t be irritated about the CollisionResult - Parameter that’s just for testing reasons. If it works this then it should be work for the sticker.

if i see correctly positions contains Triangle components.
then you get x/z subcomponents.

this is @sgold function so i belive it should work correctly.

i dont see how do you project this UV, it looks more like UV is generated based on XZAxis plane, instead of “sticker point to closest model point projection”.

but anyway if you could show some screenshot maybe i could understand what is trully wrong. (if its not about projection issue)

This is a human leg and I have the position(not exactly but very close) of the knee.
grafik
What I have done with my previous code: I figured out all triangles close of this point(in my case the hit point of the model) and it works well. Then I have created a new mesh with these triangles and created a new Geometry object with the new mesh and a material and added it to my scene. You can see the found triangles as a green patch in the model.
grafik
Now I want to add the following texture/image(height & width = 20px) to this patch:
grafik
I must not be perfect circle it’s even better with some distortions. Then it looks more realistic.

First, I’m impressed with how far you’ve gotten with this given your self-professed limitations earlier in the thread. Extracting a separate mesh is precisely how I’d have done this.

And so, with that, I will give you what I think the next steps should be in the general sense as maybe you will be able to figure it out from there.

When projecting a texture like this onto the knee, you will have some ‘surface normal’ in mind, ie: the direction that the quad would face if this were a quad. (Imagine if someone stuck a playing card to the knee.)

What you want to do then is project the 3D coordinates onto this ‘theoretical quad’ to get 2D texture coordinates and then normalize them.

You could obtain the surface normal from the center of the patch/new mesh or you could average all of the normals. (A weighted average might be the most correct… there is technically no perfect answer here.)

Once you have that, you can obtain an X and Y vector as:

Vector3f normal = ....
// Find a vector 90 degrees from both the normal and the world up vector
Vector3f left = normal.cross(Vector3f.UNIT_Y);
// Find a vector 90 degrees from both the normal and the left vector
Vector3f up = normal.cross(left);

In 3D space, left and up describe the orientation of the edges of your ‘theoretical quad’… from this we can project 3D coordinates into 2D coordinates.

// For every point in the mesh
float u = left.dot(point);
float v = up.dot(point);

Collect all of the u,v coordinates for each point, keep track of the minimum and maximum values for u and v… then go back and ‘normalize them’ so that they fall between 0 and 1. Something like:

u  =  (u - uMin)/(uMax - uMin);
v = (v - vMin)/(vMax - vMin);

Set this to the texture coordinate for that vertex.

5 Likes

so much perfect answer :slight_smile:

1 Like

Thank you very much it works almost perfectly and I would never have gotten so far without your help. Thanks @oxplay2, @pspeed, @sgold.
That’s the result and it looks great. That’s what I want.
grafik
The sticker which I use as texture for my patch:
marker-target-sticker
But, of course, there’s just one problem:

  1. If I zoom in my model the sticker disappears. I think that because there are two mesh on exactly the same position. How can I define in mesh its the number one and always visible?

Here’s the almost final version of method:

	    public static Geometry markHitPoint(Mesh mesh, AssetManager assetManager, CollisionResult result) {
    		Vector3f point = result.getContactPoint();
    		point.multLocal(1000);
    		point = new Vector3f(point.x * -1, point.z, point.y);
    		Mesh newMesh = new Mesh();
    		float maxDistance = 20;
    		List<Vector3f> position = new ArrayList<>();
    		List<Vector3f> normals = new ArrayList<>();
    		List<Integer> indices = new ArrayList<>();
    		Map<Vector3f, Integer> mapIndices = new HashMap<>();
    		for (int i = 0; i < mesh.getTriangleCount(); i++) {
    			Triangle tri = new Triangle();
    			mesh.getTriangle(i, tri);
    			boolean isClose = tri.get1().distance(point) < maxDistance || tri.get2().distance(point) < maxDistance
    					|| tri.get3().distance(point) < maxDistance;
    			if (isClose) {
    				for (int j = 0; j < 3; j++) {
    					Vector3f v = tri.get(j);
    					int index = 0;
    					if (mapIndices.containsKey(v)) {
    						index = mapIndices.get(v);
    					} else {
    						mapIndices.put(v, position.size());
    						index = position.size();
    						position.add(v);
    						normals.add(tri.getNormal());
    					}
    					indices.add(index);
    				}
    			}
    		}

    		newMesh.setBuffer(Type.Position, 3, BufferUtils.createFloatBuffer(position.toArray(new Vector3f[] {})));
    		newMesh.setBuffer(Type.Normal, 3, BufferUtils.createFloatBuffer(normals.toArray(new Vector3f[] {})));
    		newMesh.setBuffer(Type.Index, 3,
    				BufferUtils.createIntBuffer(indices.stream().mapToInt(Integer::intValue).toArray()));

    		// --------UV Calc Start-----------//
    		Vector3f normal = normals.stream().reduce(new Vector3f(), (total, next) -> total.addLocal(next)).normalize();
    		// Find a vector 90 degrees from both the normal and the world up vector
    		Vector3f left = normal.cross(Vector3f.UNIT_Y);
    		// Find a vector 90 degrees from both the normal and the left vector
    		Vector3f up = normal.cross(left);
    		float uMin = Float.MAX_VALUE, uMax = Float.MAX_VALUE * -1, vMin = Float.MAX_VALUE, vMax = Float.MAX_VALUE * -1;
    		List<Vector2f> uv = new ArrayList<>();
    		for (Vector3f p : position) {
    			float u = left.dot(p);
    			float v = up.dot(p);
    			uv.add(new Vector2f(u, v));
    			uMin = Math.min(uMin, u);
    			uMax = Math.max(uMax, u);
    			vMin = Math.min(vMin, v);
    			vMax = Math.max(vMax, v);
    		}
    		for (Vector2f textCoord : uv) {
    			float u = textCoord.x, v = textCoord.y;
    			textCoord.x = (u - uMin) / (uMax - uMin);
    			textCoord.y = (v - vMin) / (vMax - vMin);
    		}

    		FloatBuffer uvs = BufferUtils.createFloatBuffer(uv.toArray(new Vector2f[] {}));
    		newMesh.setBuffer(VertexBuffer.Type.TexCoord, 2, uvs);
    		// --------UV Calc End-----------//

    		Geometry geo = new Geometry("OurQuad", newMesh);
    		Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
    		Texture2D tex = (Texture2D) assetManager.loadTexture("marker-target-sticker.png");
    		mat.setTexture("ColorMap", tex);
    		mat.getAdditionalRenderState().setBlendMode(BlendMode.Alpha);
    	    geo.setQueueBucket(Bucket.Transparent);
    		geo.setMaterial(mat);
    		System.out.println("called");
    		return geo;
    	}
1 Like

i think we should start from different point.

that is:

I think that because there are two mesh on exactly the same position

maybe i missed something, but Why you need to have two mesh on exactly the same position?

my guess: Do you mean you have 2 character Geometries in same position, and one is transparent with circles(stickers) and second is solid color? You can solve it with one Geometry just writing simple shader that replace transparent pixels with some color.

my second guess: issue can be also about camera frustrumNear that can hide too close mesh.

my third guess: you have each Geometry for each sticker, but you dont need to, you can replace UV just on one geometry, no need 2.

my forth quess: you copy verticles from knee and add it over character again, just with UV, you also dont need do it, you can directly change it on character existing verticles. (yes it might be transparent in other places, but then you can set it as solid color in shader)

btw. sorry if i missed some part, or maybe if i dont look well at code you provided, but i need to know why there is need for 2 mesh in exactly same location. As i know it was just told about change UV on existing verticles, no need copy mesh, at least i dont see reason.

Also if mesh overlap, you can do simple solution by remove verticles from one, to avoid z-fighting or other issues. (but still i think it should be just one mesh)

in my response i marked bold keywords/sentences that might help you solve this issue.

i might sometime write too fast, but looking at your code i see what i guessed, you really copy mesh just for circle.

you use newMesh to create new Geometry.

you dont need to, you can just replace character mesh :slight_smile: (or if you want to leave what you have now, just remove this verticles from character then)

please note if you have 2 mesh in same position, there is z-fighting anyway, didnt you see flickering circle when you move camera? thats why you need just have one mesh, or remove part of one to avoid overlaping.

solution 1(best): instead of doing newMesh and Geometry from it, just replace UV for this verticles in character mesh TextureCoords buffer. if you will have issues with material, you need just simple shader update to make transparent pixels some solid color.

solution 2(worse): leave what you already have, but remove verticles from character that you dupplicate

solution 3(worst imo): make a little offset of your dupplicated mesh to not be in exact same location.

When done, you might also want look at camera frustrumNear to make possible zoom even more and avoid culling close mesh.

1 Like

One mesh is a submesh of what he extracted. It needs to overlay on top of the full mesh. At least that’s what I’ve understood.

To avoid z-fighting, you will have to make sure that it is drawn after the other mesh… putting it the transparent bucket is the easiest way.

If you still have issues then you can also change the depth test function for its material.

It doesn’t seem a z-Buffer fight. If I remove my character model then the sticker disappear anyway.
What do mean with frustrumNear and why disappears just the sticker when I’m zooming and not my character model?

So, either it’s bounding shape is wrong or you are near-plane clipping… but then you’d have been clipping the big model, too.

Probably you need to call updateBound() on the mesh.

2 Likes

You are so right. That did the trick. Thank you very much.

@pspeed @oxplay2

grafik

I will refactor my code and then post it.

4 Likes

nice work! :slight_smile: so issue was about near-plane clipping, but caused by bounding issue.

also i did not take in mind you would want to “overlay” multiple stickers on each other, then your current solution will work well :slight_smile:

@pspeed he said model is about to be just a solid color, so i thought ealier it would be better to reuse model buffer, but now i see copy buffer as new mesh is better in this case.

Near plane clipping is different.

This was because the model’s bounds were at 0,0,0 and probably 0 in size. So as soon as 0,0,0 was not in the camera frustum then the object would be “clipped”.

Near plane clipping is when pixels aren’t drawn because they are too close to the camera and so outside the range of the z-buffer.

2 Likes

Here’s my final solution:

	public static Geometry createTargetMarker(Mesh mesh, Vector3f markerPosition, AssetManager assetManager) {
		Mesh markerMesh = new Mesh();
		float maxDistance = 20;
		List<Vector3f> position = new ArrayList<>();
		List<Vector3f> normals = new ArrayList<>();
		List<Integer> indices = new ArrayList<>();

		for (int i = 0; i < mesh.getTriangleCount(); i++) {
			Triangle tri = new Triangle();
			mesh.getTriangle(i, tri);
			boolean isClose = tri.get1().distance(markerPosition) < maxDistance
					|| tri.get2().distance(markerPosition) < maxDistance
					|| tri.get3().distance(markerPosition) < maxDistance;
			if (isClose) {
				for (int j = 0; j < 3; j++) {
					Vector3f v = tri.get(j);
					int index = 0;
					if (position.contains(v)) {
						index = position.indexOf(v);
					} else {
						index = position.size();
						position.add(v);
						normals.add(tri.getNormal());
					}
					indices.add(index);
				}
			}
		}

		Vector3f normal = normals.stream().reduce(new Vector3f(), (total, next) -> total.addLocal(next)).normalize();
		// Find a vector 90 degrees from both the normal and the world up vector
		Vector3f left = normal.cross(Vector3f.UNIT_Y);
		// Find a vector 90 degrees from both the normal and the left vector
		Vector3f up = normal.cross(left);
		float uMin = Float.MAX_VALUE, uMax = -Float.MAX_VALUE, vMin = Float.MAX_VALUE, vMax = -Float.MAX_VALUE;
		List<Vector2f> uvCoords = new ArrayList<>();
		for (Vector3f p : position) {
			float u = left.dot(p);
			float v = up.dot(p);
			uvCoords.add(new Vector2f(u, v));
			uMin = Math.min(uMin, u);
			uMax = Math.max(uMax, u);
			vMin = Math.min(vMin, v);
			vMax = Math.max(vMax, v);
		}
		// Normalize UV coords
		for (Vector2f textCoord : uvCoords) {
			float u = textCoord.x, v = textCoord.y;
			textCoord.x = (u - uMin) / (uMax - uMin);
			textCoord.y = (v - vMin) / (vMax - vMin);
		}

		markerMesh.setBuffer(Type.Position, 3, createFloatBuffer(position.toArray(new Vector3f[] {})));
		markerMesh.setBuffer(Type.Normal, 3, createFloatBuffer(normals.toArray(new Vector3f[] {})));
		markerMesh.setBuffer(Type.Index, 3, createIntBuffer(indices.stream().mapToInt(Integer::intValue).toArray()));
		markerMesh.setBuffer(Type.TexCoord, 2, createFloatBuffer(uvCoords.toArray(new Vector2f[] {})));

		Geometry geo = new Geometry("Marker", markerMesh);
		Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
		Texture2D tex = (Texture2D) assetManager.loadTexture("marker-target-sticker.png");
		mat.setTexture("ColorMap", tex);
		mat.getAdditionalRenderState().setBlendMode(BlendMode.Alpha);
		mat.getAdditionalRenderState().setPolyOffset(-1f, -1f);
		geo.setQueueBucket(Bucket.Transparent);
		geo.setMaterial(mat);
		markerMesh.updateBound();
		return geo;
	}
4 Likes