How to read Colors from the Scene?

fm27 · September 18, 2012, 10:44am

Hello Community!

Today, I want to read the color from scene-objects so that I have their color as RGBAColor in my Program. I dont want do display it on the screen. Has anyone an Idea how I can do that? I tried 2 ways so far… without success

My first try:

I use raycasting
I can compute the collisions between ray and seen
I can compute the triangle which has been hit
Works fine… but Triangles does not provide any methods like “readColor” or “getColor”. How can I go on?

[java]

CollisionResults results = new CollisionResults();

Ray ray = new Ray(this.position, this.orientation);

this.se.getRootNode().collideWith(ray, results);

CollisionResult closestCollision = results.getClosestCollision();

float dist = closestCollision.getDistance();

Vector3f pt = closestCollision.getContactPoint();

String hit = closestCollision.getGeometry().getName();

Triangle ttt = closestCollision.getTriangle(null);

[/java]

My second try was to use a camera. I have been thinking that I can read the pixels from the 2D Projection which is computed by the camera.
This is very nice. The camera films the scene and I also have it on my screen. But how do I find the matrix with the color values that are displayed on my screen?
Cameras do have some interesting functions which may be helpful. read(JMEImporter …), getScreenCoordinates(Vector3f), getWorldCoordonates(Vector3f),…

[java]Camera cam2 = cam.clone();

cam2.setViewPort(.4f, .6f, 0.8f, 1f);

cam2.setLocation(new Vector3f(-0.10f, 1.57f, 4.81f));

cam2.setRotation(new Quaternion(0.00f, 0.99f, -0.04f, 0.02f));

ViewPort viewPort2 = renderManager.createMainView("PiP", cam2);

viewPort2.setClearFlags(true, true, true);

viewPort2.attachScene(rootNode);

[/java]
Can anyone please give me an example how to go on to read colors from the scene?
If it is not possible the way I tried… how can I read the color of objects from the scene?

Please tell me at least if my question is to stupid or not understandable.

Thanks.

zarch · September 18, 2012, 12:21pm

What you are trying to do is not simple - quite apart from anything else the “colour” of something depends entirely on the fragment shader used to render it.

This can include material colour, textures, lighting, shadows, potentially even other opaque objects partially in front of it or transparent/translucent objects in front of it…etc…etc.

This may be easier if you take a step back and explain what you are trying to achieve by doing this as there may be a simpler solution.

fm27 · September 18, 2012, 12:59pm

I write a software simulation of small robots. These robots shell navigate through the scene and they shell use several sensors for that, including a lightsensor and a colorsensor.

I can imagine that there are much parameters which influence the color of a polygon in 3D-space. But if this polygon is already rendered into 2D (which should be done by the “camera”)? Is it not possible to access the rendered 2D projection?

normen · September 18, 2012, 1:00pm

Its inside the graphics card, thats the issue. As mentined you have to render it back to a texture and getting that out of the graphics card is the issue.

fm27 · September 18, 2012, 1:28pm

Ok… looks like that I have to render it back… how can I do that? Do I have to access the viewport in such a way?

[java]viewport.getOutputFrameBuffer().getColorBuffer().getTexture().getImage().getData(id)[/java]

But I still dont find Colors there…

normen · September 18, 2012, 1:36pm

Look at TestRenderToMemory

fm27 · October 31, 2012, 2:25pm

Thanks! Looks complicated

Perhaps there is a another possibility:

Raycasting (and getting the hit-geometry and the hit-point)
Accessing the pixel of the texture (which is mapped to that geometry) at the hitpoint?

I would ignore shadows, lighting, other opaque objects ect. ect. in that case. I would read the value of the texture only! Is that simpler to do? What do I have to do?

normen · October 31, 2012, 2:28pm

@fm27 said:
What do I have to do?

Compute the texture coord by the closest vertices and their texture coordinates.

fm27 · December 1, 2012, 10:24am

Sorry to bother again… just for my understanding: Is the computation of the (final) color of a polygon only DIFFICULT (but possible) to do in software or is it IMPOSSIBLE without graphic card support (when using an OpenGL & LWJGL based rendering system)?

wezrule · December 1, 2012, 10:36am

The final color yes, i’m pretty sure that’s nearly impossible without graphics card support. Unless you create some sort of universal graphics card emulator.

If your just using simple colors and texture maps then you can get that information from raycasting like you suggested. Just get the color from the texture at the texCoord of the ray intersection and add any additional colors you used with that material.

Empire_Phoenix · December 1, 2012, 1:10pm

I did something similar, for critical hitzones, that where simply a painted texture,

here try to understand how it works and see if it fits your situation.

position is the local Hitposition in the mesh coordiante system

[java]

private static Vector2f getHitUVPoint(final Mesh mesh, final Vector3f position, final int triindex) {

final VertexBuffer textureBuffer = mesh.getBuffer(Type.TexCoord);

final VertexBuffer positionBuffer = mesh.getBuffer(Type.Position);

final IndexBuffer ib = mesh.getIndicesAsList();

assert textureBuffer != null && textureBuffer.getFormat() == Format.Float && textureBuffer.getNumComponents() == 2;

assert positionBuffer != null && positionBuffer.getFormat() == Format.Float && positionBuffer.getNumComponents() == 3;

final FloatBuffer textureFloatBuffer = (FloatBuffer) textureBuffer.getData();

// aquire triangle’s vertex indices

final int vertIndex = triindex * 3;

final int vertindex1 = ib.get(vertIndex);

final int vertindex2 = ib.get(vertIndex + 1);

final int vertindex3 = ib.get(vertIndex + 2);

final FloatBuffer positionFloatBuffer = (FloatBuffer) positionBuffer.getData();

final Vector3f px1 = new Vector3f();

final Vector3f px2 = new Vector3f();

final Vector3f px3 = new Vector3f();

BufferUtils.populateFromBuffer(px1, positionFloatBuffer, vertindex1);

BufferUtils.populateFromBuffer(px2, positionFloatBuffer, vertindex2);

BufferUtils.populateFromBuffer(px3, positionFloatBuffer, vertindex3);

System.out.println(position);

System.out.println(px1);

System.out.println(px2);

System.out.println(px3);

final double[][] A = new double[3 + 1][4 + 1];

A[1][1] = px1.x;

A[1][2] = px1.y;

A[1][3] = px1.z;

A[1][4] = position.x;

A[2][1] = px2.x;

A[2][2] = px2.y;

A[2][3] = px2.z;

A[2][4] = position.y;

A[3][1] = px3.x;

A[3][2] = px3.y;

A[3][3] = px3.z;

A[3][4] = position.z;

final Vector3f linearFactors = GaussJordan.solve(A);

// Texture coordinats

final Vector2f tx1 = new Vector2f();

final Vector2f tx2 = new Vector2f();

final Vector2f tx3 = new Vector2f();

BufferUtils.populateFromBuffer(tx1, textureFloatBuffer, vertindex1);

BufferUtils.populateFromBuffer(tx2, textureFloatBuffer, vertindex2);

BufferUtils.populateFromBuffer(tx3, textureFloatBuffer, vertindex3);

System.out.println(tx1);

System.out.println(tx2);

System.out.println(tx3);

while (tx1.x > 1) {

tx1.x–;

}

while (tx1.y > 1) {

tx1.y–;

}

final Vector2f targettex = new Vector2f();

targettex.addLocal(tx1.mult(linearFactors.x));

targettex.addLocal(tx2.mult(linearFactors.y));

targettex.addLocal(tx3.mult(linearFactors.z));

System.out.println(targettex);

return targettex;

}

[/java]

[java]

package de.visiongamestudios.shared.util;

import com.jme3.math.Vector3f;

public class GaussJordan {

// swap()

// swap row i with row k

// pre: A[q]==A[k][q]==0 for 1<=q<j

static void swap(final double[][] A, final int i, final int k, final int j) {

final int m = A[0].length - 1;

double temp;

for (int q = j; q <= m; q++) {

temp = A[q];

A[q] = A[k][q];

A[k][q] = temp;

}

}

// divide()

// divide row i by A[j]

// pre: A[j]!=0, A[q]==0 for 1<=q<j

// post: A[j]==1;

static void divide(final double[][] A, final int i, final int j) {

final int m = A[0].length - 1;

for (int q = j + 1; q <= m; q++) {

A[q] /= A[j];

}

A[j] = 1;

}

// eliminate()

// subtract an appropriate multiple of row i from every other row

// pre: A[j]==1, A[q]==0 for 1<=q<j

// post: A[p][j]==0 for p!=i

static void eliminate(final double[][] A, final int i, final int j) {

final int n = A.length - 1;

final int m = A[0].length - 1;

for (int p = 1; p <= n; p++) {

if (p != i && A[p][j] != 0) {

for (int q = j + 1; q <= m; q++) {

A[p][q] -= A[p][j] * A[q];

}

A[p][j] = 0;

}

}

}

// printMatrix()

// print the present state of Matrix A to file out

static void printMatrix(final double[][] A) {

final int n = A.length - 1;

final int m = A[0].length - 1;

for (int i = 1; i <= n; i++) {

for (int j = 1; j <= m; j++) {

System.out.print(A[j] + " ");

}

System.out.println();

}

System.out.println();

System.out.println();

}

public static Vector3f solve(final double A[][]) {

int n, m, i, j, k;

// read first line of input file

n = 3;

m = 4;

// declare A to be of size (n+1)x(m+1) and do not use index 0

// print array A to output file

GaussJordan.printMatrix(A);

// perform Gauss-Jordan Elimination algorithm

i = 1;

j = 1;

while (i <= n && j <= m) {

// look for a non-zero entry in col j at or below row i

k = i;

while (k <= n && A[k][j] == 0) {

k++;

}

// if such an entry is found at row k

if (k <= n) {

// if k is not i, then swap row i with row k

if (k != i) {

GaussJordan.swap(A, i, k, j);

GaussJordan.printMatrix(A);

}

// if A[j] is not 1, then divide row i by A[j]

if (A[j] != 1) {

GaussJordan.divide(A, i, j);

GaussJordan.printMatrix(A);

}

// eliminate all other non-zero entries from col j by

// subtracting from each

// row (other than i) an appropriate multiple of row i

GaussJordan.eliminate(A, i, j);

GaussJordan.printMatrix(A);

i++;

}

j++;

}

// print rank to output file

System.out.println("rank = " + (i - 1));

return new Vector3f((float) A[1][4], (float) A[2][4], (float) A[3][4]);

}

}

[/java]

fm27 · January 10, 2013, 8:12pm

First of all; big thanks for your reply and your code!

Unfortunately I dont get it running. Can you please tell me if I use your function correct?

I am working with raycasting ect. … so “closest” is the collisionresult of the raycasting and “hitgeo” is the geometry which has been hit by the ray.
[java]
Vector3f position = hitgeo.worldToLocal(closest.getContactPoint(), null);

Vector2f texturepoint = this.getHitUVPoint(hitgeo.getMesh(), position, closest.getTriangleIndex());
ImageRaster image = ImageRaster.create(hitgeo.getMaterial().getTextureParam(“DiffuseMap”).getTextureValue().getImage());
ColorRGBA color = image.getPixel((int)(texturepoint.x), (int)texturepoint.y);[/java]

And can you tell me… I newer saw this before (and my compiler as well): A closing i-tag opening i tag. Is this the same like A[i]?
And what is tx1.y - ? Is it decrementing ( tx1.y - -)?

fm27 · January 10, 2013, 8:29pm

(this post can be deleted - just wanted to edit my previous post)

Empire_Phoenix · January 10, 2013, 8:34pm

Well you just give it the mesh that was hit, the localhitpos and the triindex, all should be in the collisionresult, or reachble from it. ith then calcualtes the exact position for it, based on the three nearest vertexes.

fm27 · January 10, 2013, 8:43pm

Ok i think that this is what I do. I think that I still dont understand what the return value exacty is. I have assumed that the return value is the pixel index of the texture. Is that correct?

And again… one reason why I dont get it running (perhaps) is because I dont understand this code in your GaussJordan (my compiler also cannot compile it):

[java]while (tx1.x > 1) {
tx1.x–;
}
while (tx1.y > 1) {
tx1.y–;
}[/java]

What are you doing there? are you decrementing the tx1.x and tx1.y values? (because there is only one minus in your code)