Full allocated memoryMangement

Hi, this is a alternative prototype implementation for a different approch to bufferutils,

while not complelty efficient so far, it works quite nice for me at least(i had no more outofmemory)


Code:
/*
  • Copyright © 2009-2010 jMonkeyEngine

  • All rights reserved.

  • Redistribution and use in source and binary forms, with or without

  • modification, are permitted provided that the following conditions are

  • met:

    • Redistributions of source code must retain the above copyright
  • notice, this list of conditions and the following disclaimer.

    • Redistributions in binary form must reproduce the above copyright
  • notice, this list of conditions and the following disclaimer in the

  • documentation and/or other materials provided with the distribution.

    • Neither the name of ‘jMonkeyEngine’ nor the names of its contributors
  • may be used to endorse or promote products derived from this software

  • without specific prior written permission.

  • THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS

  • "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED

  • TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR

  • PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR

  • CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,

  • EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,

  • PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR

  • PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF

  • LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING

  • NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS

  • SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

package com.jme3.util;

import java.lang.ref.PhantomReference;
import java.lang.ref.ReferenceQueue;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.nio.Buffer;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.nio.DoubleBuffer;
import java.nio.FloatBuffer;
import java.nio.IntBuffer;
import java.nio.ShortBuffer;
import java.util.Collections;
import java.util.Comparator;
import java.util.HashMap;
import java.util.Map.Entry;
import java.util.Vector;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.logging.Level;
import java.util.logging.Logger;

import com.jme3.math.ColorRGBA;
import com.jme3.math.Quaternion;
import com.jme3.math.Vector2f;
import com.jme3.math.Vector3f;

/**
*

  • <code>BufferUtils</code> is a helper class for generating nio buffers from
  • jME data classes such as Vectors and ColorRGBA.
  • @author Joshua Slack
  • @version $Id: BufferUtils.java,v 1.16 2007/10/29 16:56:18 nca Exp $
    */

public final class BufferUtils {

private static ByteBuffer backingBuffer;

public static void initialize(final int bytes) {
	System.out.println(&quot;Trying to allocate memory: &quot; + bytes);
	backingBuffer = ByteBuffer.allocateDirect(bytes);
	final CleanupThread clthread = new CleanupThread();
	clthread.setDaemon(true);
	clthread.start();
}

static final AtomicInteger bytesAllocated = new AtomicInteger();
static final ReferenceQueue collectedBuffers = new ReferenceQueue();

/**
 * 
 * This is needed, else the Reference is collected before the buffer, so it
 * 
 * cannot be added to the Referencequeue
 */

static final Vector&lt;BufferReference&gt; referencetrash = new Vector();

private static final boolean detailPrint = true;

public static void dalloc(final ByteBuffer buf) {
	synchronized (referencetrash) {
		referencetrash.remove(buf);
	}
}

/**
 * 
 * Creates a clone of the given buffer. The clone's capacity is equal to the
 * 
 * given buffer's limit.
 * 
 * 
 * 
 * @param buf
 * 
 *            The buffer to clone
 * 
 * @return The cloned buffer
 */

public static Buffer clone(final Buffer buf) {

	if (buf instanceof FloatBuffer) {

		return clone((FloatBuffer) buf);

	} else if (buf instanceof ShortBuffer) {

		return clone((ShortBuffer) buf);

	} else if (buf instanceof ByteBuffer) {

		return clone((ByteBuffer) buf);

	} else if (buf instanceof IntBuffer) {

		return clone((IntBuffer) buf);

	} else if (buf instanceof DoubleBuffer) {

		return clone((DoubleBuffer) buf);

	} else {

		throw new UnsupportedOperationException();

	}

}

/**
 * 
 * Generate a new FloatBuffer using the given array of Vector3f objects. The
 * 
 * FloatBuffer will be 3 * data.length long and contain the vector data as
 * 
 * data[0].x, data[0].y, data[0].z, data[1].x... etc.
 * 
 * 
 * 
 * @param data
 * 
 *            array of Vector3f objects to place into a new FloatBuffer
 */

public static FloatBuffer createFloatBuffer(final Vector3f... data) {

	if (data == null) {

		return null;

	}

	final FloatBuffer buff = createFloatBuffer(3 * data.length);

	for (int x = 0; x &lt; data.length; x++) {

		if (data[x] != null) {

			buff.put(data[x].x).put(data[x].y).put(data[x].z);

		} else {

			buff.put(0).put(0).put(0);

		}

	}

	buff.flip();

	return buff;

}

/**
 * 
 * Generate a new FloatBuffer using the given array of Quaternion objects.
 * 
 * The FloatBuffer will be 4 * data.length long and contain the vector data.
 * 
 * 
 * 
 * @param data
 * 
 *            array of Quaternion objects to place into a new FloatBuffer
 */

public static FloatBuffer createFloatBuffer(final Quaternion... data) {

	if (data == null) {

		return null;

	}

	final FloatBuffer buff = createFloatBuffer(4 * data.length);

	for (int x = 0; x &lt; data.length; x++) {

		if (data[x] != null) {

			buff.put(data[x].getX()).put(data[x].getY()).put(data[x].getZ()).put(data[x].getW());

		} else {

			buff.put(0).put(0).put(0);

		}

	}

	buff.flip();

	return buff;

}

/**
 * 
 * Generate a new FloatBuffer using the given array of float primitives.
 * 
 * 
 * 
 * @param data
 * 
 *            array of float primitives to place into a new FloatBuffer
 */

public static FloatBuffer createFloatBuffer(final float... data) {

	if (data == null) {

		return null;

	}

	final FloatBuffer buff = createFloatBuffer(data.length);

	buff.clear();

	buff.put(data);

	buff.flip();

	return buff;

}

/**
 * 
 * Create a new FloatBuffer of an appropriate size to hold the specified
 * 
 * number of Vector3f object data.
 * 
 * 
 * 
 * @param vertices
 * 
 *            number of vertices that need to be held by the newly created
 * 
 *            buffer
 * 
 * @return the requested new FloatBuffer
 */

public static FloatBuffer createVector3Buffer(final int vertices) {

	final FloatBuffer vBuff = createFloatBuffer(3 * vertices);

	return vBuff;

}

/**
 * 
 * Create a new FloatBuffer of an appropriate size to hold the specified
 * 
 * number of Vector3f object data only if the given buffer if not already
 * 
 * the right size.
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to first check and rewind
 * 
 * @param vertices
 * 
 *            number of vertices that need to be held by the newly created
 * 
 *            buffer
 * 
 * @return the requested new FloatBuffer
 */

public static FloatBuffer createVector3Buffer(final FloatBuffer buf, final int vertices) {

	if (buf != null &amp;&amp; buf.limit() == 3 * vertices) {

		buf.rewind();

		return buf;

	}

	return createFloatBuffer(3 * vertices);

}

/**
 * 
 * Sets the data contained in the given color into the FloatBuffer at the
 * 
 * specified index.
 * 
 * 
 * 
 * @param color
 * 
 *            the data to insert
 * 
 * @param buf
 * 
 *            the buffer to insert into
 * 
 * @param index
 * 
 *            the postion to place the data; in terms of colors not floats
 */

public static void setInBuffer(final ColorRGBA color, final FloatBuffer buf, final int index) {

	buf.position(index * 4);

	buf.put(color.r);

	buf.put(color.g);

	buf.put(color.b);

	buf.put(color.a);

}

/**
 * 
 * Sets the data contained in the given quaternion into the FloatBuffer at
 * 
 * the specified index.
 * 
 * 
 * 
 * @param quat
 * 
 *            the {@link Quaternion} to insert
 * 
 * @param buf
 * 
 *            the buffer to insert into
 * 
 * @param index
 * 
 *            the postion to place the data; in terms of quaternions not
 * 
 *            floats
 */

public static void setInBuffer(final Quaternion quat, final FloatBuffer buf, final int index) {

	buf.position(index * 4);

	buf.put(quat.getX());

	buf.put(quat.getY());

	buf.put(quat.getZ());

	buf.put(quat.getW());

}

/**
 * 
 * Sets the data contained in the given Vector3F into the FloatBuffer at the
 * 
 * specified index.
 * 
 * 
 * 
 * @param vector
 * 
 *            the data to insert
 * 
 * @param buf
 * 
 *            the buffer to insert into
 * 
 * @param index
 * 
 *            the postion to place the data; in terms of vectors not floats
 */

public static void setInBuffer(final Vector3f vector, final FloatBuffer buf, final int index) {

	if (buf == null) {

		return;

	}

	if (vector == null) {

		buf.put(index * 3, 0);

		buf.put((index * 3) + 1, 0);

		buf.put((index * 3) + 2, 0);

	} else {

		buf.put(index * 3, vector.x);

		buf.put((index * 3) + 1, vector.y);

		buf.put((index * 3) + 2, vector.z);

	}

}

/**
 * 
 * Updates the values of the given vector from the specified buffer at the
 * 
 * index provided.
 * 
 * 
 * 
 * @param vector
 * 
 *            the vector to set data on
 * 
 * @param buf
 * 
 *            the buffer to read from
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) to read from
 * 
 *            the buf
 */

public static void populateFromBuffer(final Vector3f vector, final FloatBuffer buf, final int index) {

	vector.x = buf.get(index * 3);

	vector.y = buf.get(index * 3 + 1);

	vector.z = buf.get(index * 3 + 2);

}

/**
 * 
 * Generates a Vector3f array from the given FloatBuffer.
 * 
 * 
 * 
 * @param buff
 * 
 *            the FloatBuffer to read from
 * 
 * @return a newly generated array of Vector3f objects
 */

public static Vector3f[] getVector3Array(final FloatBuffer buff) {

	buff.clear();

	final Vector3f[] verts = new Vector3f[buff.limit() / 3];

	for (int x = 0; x &lt; verts.length; x++) {

		final Vector3f v = new Vector3f(buff.get(), buff.get(), buff.get());

		verts[x] = v;

	}

	return verts;

}

/**
 * 
 * Copies a Vector3f from one position in the buffer to another. The index
 * 
 * values are in terms of vector number (eg, vector number 0 is postions 0-2
 * 
 * in the FloatBuffer.)
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to copy from/to
 * 
 * @param fromPos
 * 
 *            the index of the vector to copy
 * 
 * @param toPos
 * 
 *            the index to copy the vector to
 */

public static void copyInternalVector3(final FloatBuffer buf, final int fromPos, final int toPos) {

	copyInternal(buf, fromPos * 3, toPos * 3, 3);

}

/**
 * 
 * Normalize a Vector3f in-buffer.
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to find the Vector3f within
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) of the vector
 * 
 *            to normalize
 */

public static void normalizeVector3(final FloatBuffer buf, final int index) {

	final TempVars vars = TempVars.get();

	final Vector3f tempVec3 = vars.vect1;

	populateFromBuffer(tempVec3, buf, index);

	tempVec3.normalizeLocal();

	setInBuffer(tempVec3, buf, index);

	vars.release();

}

/**
 * 
 * Add to a Vector3f in-buffer.
 * 
 * 
 * 
 * @param toAdd
 * 
 *            the vector to add from
 * 
 * @param buf
 * 
 *            the buffer to find the Vector3f within
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) of the vector
 * 
 *            to add to
 */

public static void addInBuffer(final Vector3f toAdd, final FloatBuffer buf, final int index) {

	final TempVars vars = TempVars.get();

	final Vector3f tempVec3 = vars.vect1;

	populateFromBuffer(tempVec3, buf, index);

	tempVec3.addLocal(toAdd);

	setInBuffer(tempVec3, buf, index);

	vars.release();

}

/**
 * 
 * Multiply and store a Vector3f in-buffer.
 * 
 * 
 * 
 * @param toMult
 * 
 *            the vector to multiply against
 * 
 * @param buf
 * 
 *            the buffer to find the Vector3f within
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) of the vector
 * 
 *            to multiply
 */

public static void multInBuffer(final Vector3f toMult, final FloatBuffer buf, final int index) {

	final TempVars vars = TempVars.get();

	final Vector3f tempVec3 = vars.vect1;

	populateFromBuffer(tempVec3, buf, index);

	tempVec3.multLocal(toMult);

	setInBuffer(tempVec3, buf, index);

	vars.release();

}

/**
 * 
 * Checks to see if the given Vector3f is equals to the data stored in the
 * 
 * buffer at the given data index.
 * 
 * 
 * 
 * @param check
 * 
 *            the vector to check against - null will return false.
 * 
 * @param buf
 * 
 *            the buffer to compare data with
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) of the vector
 * 
 *            in the buffer to check against
 * 
 * @return
 */

public static boolean equals(final Vector3f check, final FloatBuffer buf, final int index) {

	final TempVars vars = TempVars.get();

	final Vector3f tempVec3 = vars.vect1;

	populateFromBuffer(tempVec3, buf, index);

	final boolean eq = tempVec3.equals(check);

	vars.release();

	return eq;

}

// // -- VECTOR2F METHODS -- ////

/**
 * 
 * Generate a new FloatBuffer using the given array of Vector2f objects. The
 * 
 * FloatBuffer will be 2 * data.length long and contain the vector data as
 * 
 * data[0].x, data[0].y, data[1].x... etc.
 * 
 * 
 * 
 * @param data
 * 
 *            array of Vector2f objects to place into a new FloatBuffer
 */

public static FloatBuffer createFloatBuffer(final Vector2f... data) {

	if (data == null) {

		return null;

	}

	final FloatBuffer buff = createFloatBuffer(2 * data.length);

	for (int x = 0; x &lt; data.length; x++) {

		if (data[x] != null) {

			buff.put(data[x].x).put(data[x].y);

		} else {

			buff.put(0).put(0);

		}

	}

	buff.flip();

	return buff;

}

/**
 * 
 * Create a new FloatBuffer of an appropriate size to hold the specified
 * 
 * number of Vector2f object data.
 * 
 * 
 * 
 * @param vertices
 * 
 *            number of vertices that need to be held by the newly created
 * 
 *            buffer
 * 
 * @return the requested new FloatBuffer
 */

public static FloatBuffer createVector2Buffer(final int vertices) {

	final FloatBuffer vBuff = createFloatBuffer(2 * vertices);

	return vBuff;

}

/**
 * 
 * Create a new FloatBuffer of an appropriate size to hold the specified
 * 
 * number of Vector2f object data only if the given buffer if not already
 * 
 * the right size.
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to first check and rewind
 * 
 * @param vertices
 * 
 *            number of vertices that need to be held by the newly created
 * 
 *            buffer
 * 
 * @return the requested new FloatBuffer
 */

public static FloatBuffer createVector2Buffer(final FloatBuffer buf, final int vertices) {

	if (buf != null &amp;&amp; buf.limit() == 2 * vertices) {

		buf.rewind();

		return buf;

	}

	return createFloatBuffer(2 * vertices);

}

/**
 * 
 * Sets the data contained in the given Vector2F into the FloatBuffer at the
 * 
 * specified index.
 * 
 * 
 * 
 * @param vector
 * 
 *            the data to insert
 * 
 * @param buf
 * 
 *            the buffer to insert into
 * 
 * @param index
 * 
 *            the postion to place the data; in terms of vectors not floats
 */

public static void setInBuffer(final Vector2f vector, final FloatBuffer buf, final int index) {

	buf.put(index * 2, vector.x);

	buf.put((index * 2) + 1, vector.y);

}

/**
 * 
 * Updates the values of the given vector from the specified buffer at the
 * 
 * index provided.
 * 
 * 
 * 
 * @param vector
 * 
 *            the vector to set data on
 * 
 * @param buf
 * 
 *            the buffer to read from
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) to read from
 * 
 *            the buf
 */

public static void populateFromBuffer(final Vector2f vector, final FloatBuffer buf, final int index) {

	vector.x = buf.get(index * 2);

	vector.y = buf.get(index * 2 + 1);

}

/**
 * 
 * Generates a Vector2f array from the given FloatBuffer.
 * 
 * 
 * 
 * @param buff
 * 
 *            the FloatBuffer to read from
 * 
 * @return a newly generated array of Vector2f objects
 */

public static Vector2f[] getVector2Array(final FloatBuffer buff) {

	buff.clear();

	final Vector2f[] verts = new Vector2f[buff.limit() / 2];

	for (int x = 0; x &lt; verts.length; x++) {

		final Vector2f v = new Vector2f(buff.get(), buff.get());

		verts[x] = v;

	}

	return verts;

}

/**
 * 
 * Copies a Vector2f from one position in the buffer to another. The index
 * 
 * values are in terms of vector number (eg, vector number 0 is postions 0-1
 * 
 * in the FloatBuffer.)
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to copy from/to
 * 
 * @param fromPos
 * 
 *            the index of the vector to copy
 * 
 * @param toPos
 * 
 *            the index to copy the vector to
 */

public static void copyInternalVector2(final FloatBuffer buf, final int fromPos, final int toPos) {

	copyInternal(buf, fromPos * 2, toPos * 2, 2);

}

/**
 * 
 * Normalize a Vector2f in-buffer.
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to find the Vector2f within
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) of the vector
 * 
 *            to normalize
 */

public static void normalizeVector2(final FloatBuffer buf, final int index) {

	final TempVars vars = TempVars.get();

	final Vector2f tempVec2 = vars.vect2d;

	populateFromBuffer(tempVec2, buf, index);

	tempVec2.normalizeLocal();

	setInBuffer(tempVec2, buf, index);

	vars.release();

}

/**
 * 
 * Add to a Vector2f in-buffer.
 * 
 * 
 * 
 * @param toAdd
 * 
 *            the vector to add from
 * 
 * @param buf
 * 
 *            the buffer to find the Vector2f within
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) of the vector
 * 
 *            to add to
 */

public static void addInBuffer(final Vector2f toAdd, final FloatBuffer buf, final int index) {

	final TempVars vars = TempVars.get();

	final Vector2f tempVec2 = vars.vect2d;

	populateFromBuffer(tempVec2, buf, index);

	tempVec2.addLocal(toAdd);

	setInBuffer(tempVec2, buf, index);

	vars.release();

}

/**
 * 
 * Multiply and store a Vector2f in-buffer.
 * 
 * 
 * 
 * @param toMult
 * 
 *            the vector to multiply against
 * 
 * @param buf
 * 
 *            the buffer to find the Vector2f within
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) of the vector
 * 
 *            to multiply
 */

public static void multInBuffer(final Vector2f toMult, final FloatBuffer buf, final int index) {

	final TempVars vars = TempVars.get();

	final Vector2f tempVec2 = vars.vect2d;

	populateFromBuffer(tempVec2, buf, index);

	tempVec2.multLocal(toMult);

	setInBuffer(tempVec2, buf, index);

	vars.release();

}

/**
 * 
 * Checks to see if the given Vector2f is equals to the data stored in the
 * 
 * buffer at the given data index.
 * 
 * 
 * 
 * @param check
 * 
 *            the vector to check against - null will return false.
 * 
 * @param buf
 * 
 *            the buffer to compare data with
 * 
 * @param index
 * 
 *            the position (in terms of vectors, not floats) of the vector
 * 
 *            in the buffer to check against
 * 
 * @return
 */

public static boolean equals(final Vector2f check, final FloatBuffer buf, final int index) {

	final TempVars vars = TempVars.get();

	final Vector2f tempVec2 = vars.vect2d;

	populateFromBuffer(tempVec2, buf, index);

	final boolean eq = tempVec2.equals(check);

	vars.release();

	return eq;

}

// // -- INT METHODS -- ////

/**
 * 
 * Generate a new IntBuffer using the given array of ints. The IntBuffer
 * 
 * will be data.length long and contain the int data as data[0], data[1]...
 * 
 * etc.
 * 
 * 
 * 
 * @param data
 * 
 *            array of ints to place into a new IntBuffer
 */

public static IntBuffer createIntBuffer(final int... data) {

	if (data == null) {

		return null;

	}

	final IntBuffer buff = createIntBuffer(data.length);

	buff.clear();

	buff.put(data);

	buff.flip();

	return buff;

}

/**
 * 
 * Create a new int[] array and populate it with the given IntBuffer's
 * 
 * contents.
 * 
 * 
 * 
 * @param buff
 * 
 *            the IntBuffer to read from
 * 
 * @return a new int array populated from the IntBuffer
 */

public static int[] getIntArray(final IntBuffer buff) {

	if (buff == null) {

		return null;

	}

	buff.clear();

	final int[] inds = new int[buff.limit()];

	for (int x = 0; x &lt; inds.length; x++) {

		inds[x] = buff.get();

	}

	return inds;

}

/**
 * 
 * Create a new float[] array and populate it with the given FloatBuffer's
 * 
 * contents.
 * 
 * 
 * 
 * @param buff
 * 
 *            the FloatBuffer to read from
 * 
 * @return a new float array populated from the FloatBuffer
 */

public static float[] getFloatArray(final FloatBuffer buff) {

	if (buff == null) {

		return null;

	}

	buff.clear();

	final float[] inds = new float[buff.limit()];

	for (int x = 0; x &lt; inds.length; x++) {

		inds[x] = buff.get();

	}

	return inds;

}

// // -- GENERAL DOUBLE ROUTINES -- ////

/**
 * 
 * Create a new DoubleBuffer of the specified size.
 * 
 * 
 * 
 * @param size
 * 
 *            required number of double to store.
 * 
 * @return the new DoubleBuffer
 */

public static DoubleBuffer createDoubleBuffer(final int size) {

	final DoubleBuffer buf = malloc(8 * size).order(ByteOrder.nativeOrder()).asDoubleBuffer();
	buf.clear();
	return buf;

}

/**
 * 
 * Create a new DoubleBuffer of an appropriate size to hold the specified
 * 
 * number of doubles only if the given buffer if not already the right size.
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to first check and rewind
 * 
 * @param size
 * 
 *            number of doubles that need to be held by the newly created
 * 
 *            buffer
 * 
 * @return the requested new DoubleBuffer
 */

public static DoubleBuffer createDoubleBuffer(DoubleBuffer buf, final int size) {

	if (buf != null &amp;&amp; buf.limit() == size) {

		buf.rewind();

		return buf;

	}

	buf = createDoubleBuffer(size);

	return buf;

}

/**
 * 
 * Creates a new DoubleBuffer with the same contents as the given
 * 
 * DoubleBuffer. The new DoubleBuffer is seperate from the old one and
 * 
 * changes are not reflected across. If you want to reflect changes,
 * 
 * consider using Buffer.duplicate().
 * 
 * 
 * 
 * @param buf
 * 
 *            the DoubleBuffer to copy
 * 
 * @return the copy
 */

public static DoubleBuffer clone(final DoubleBuffer buf) {

	if (buf == null) {

		return null;

	}

	buf.rewind();

	DoubleBuffer copy;

	if (buf.isDirect()) {

		copy = createDoubleBuffer(buf.limit());

	} else {

		copy = DoubleBuffer.allocate(buf.limit());

	}

	copy.put(buf);

	return copy;

}

// // -- GENERAL FLOAT ROUTINES -- ////

/**
 * 
 * Create a new FloatBuffer of the specified size.
 * 
 * 
 * 
 * @param size
 * 
 *            required number of floats to store.
 * 
 * @return the new FloatBuffer
 */

public static FloatBuffer createFloatBuffer(final int size) {

	final FloatBuffer buf = malloc(4 * size).order(ByteOrder.nativeOrder()).asFloatBuffer();
	buf.clear();
	return buf;

}

private static ByteBuffer malloc(final int size) {
	return malloc(size, false);
}

private static ByteBuffer malloc(final int size, final boolean retry) {
	synchronized (referencetrash) {
		// find free space
		final int i = findFreeSpace(size);

		if (i + size &gt; backingBuffer.capacity() || i == -1) {
			if (retry) {
				printStackTraces();

				throw new RuntimeException(&quot;Out of native memory&quot;);
			} else {
				System.gc();
				System.gc();
				System.gc();
				System.gc();
				try {
					Thread.sleep(5000);
				} catch (final InterruptedException e) {
					e.printStackTrace();
				}
				return malloc(size, true);
			}

		}

		System.out.println(&quot;New Buffer with size &quot; + size + &quot; at &quot; + i + &quot; to &quot; + (i + size));

		backingBuffer.limit(i + size);

		backingBuffer.position(i);

		final ByteBuffer returnvalue = backingBuffer.slice();

		bytesAllocated.addAndGet(size);

		final BufferReference a = new BufferReference(returnvalue, i, i + size);

		referencetrash.add(a);
		return returnvalue;
	}
}

private static void printStackTraces() {
	for (final BufferReference a : referencetrash) {
		System.out.println(&quot;Buffer &quot; + a.getSize());
		for (final StackTraceElement b : a.stacktrace) {
			System.out.println(b.getClassName() + &quot; &quot; + b.getLineNumber());
		}
	}
}

private static int findFreeSpace(final int size) {

	System.out.println(&quot;Live buffers &quot; + referencetrash.size());

	Collections.sort(referencetrash, new Comparator() {

		@Override
		public int compare(final Object arg0, final Object arg1) {

			final BufferReference b1 = (BufferReference) arg0;

			final BufferReference b2 = (BufferReference) arg1;

			return b1.start - b2.start;

		}

	});

	int lastend = 0;

	for (final BufferReference b : referencetrash) {

		if (b.start - lastend &gt; size) {

			return lastend;

		}

		lastend = b.end;

	}

	if (lastend &lt; backingBuffer.capacity()) {

		return lastend;

	}

	return -1;
}

/**
 * 
 * Copies floats from one position in the buffer to another.
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to copy from/to
 * 
 * @param fromPos
 * 
 *            the starting point to copy from
 * 
 * @param toPos
 * 
 *            the starting point to copy to
 * 
 * @param length
 * 
 *            the number of floats to copy
 */

public static void copyInternal(final FloatBuffer buf, final int fromPos, final int toPos, final int length) {

	final float[] data = new float[length];

	buf.position(fromPos);

	buf.get(data);

	buf.position(toPos);

	buf.put(data);

}

/**
 * 
 * Creates a new FloatBuffer with the same contents as the given
 * 
 * FloatBuffer. The new FloatBuffer is seperate from the old one and changes
 * 
 * are not reflected across. If you want to reflect changes, consider using
 * 
 * Buffer.duplicate().
 * 
 * 
 * 
 * @param buf
 * 
 *            the FloatBuffer to copy
 * 
 * @return the copy
 */

public static FloatBuffer clone(final FloatBuffer buf) {

	if (buf == null) {

		return null;

	}

	buf.rewind();

	FloatBuffer copy;

	if (buf.isDirect()) {

		copy = createFloatBuffer(buf.limit());

	} else {

		copy = FloatBuffer.allocate(buf.limit());

	}

	copy.put(buf);

	return copy;

}

// // -- GENERAL INT ROUTINES -- ////

/**
 * 
 * Create a new IntBuffer of the specified size.
 * 
 * 
 * 
 * @param size
 * 
 *            required number of ints to store.
 * 
 * @return the new IntBuffer
 */

public static IntBuffer createIntBuffer(final int size) {

	final IntBuffer buf = malloc(4 * size).order(ByteOrder.nativeOrder()).asIntBuffer();

	buf.clear();
	return buf;

}

/**
 * 
 * Create a new IntBuffer of an appropriate size to hold the specified
 * 
 * number of ints only if the given buffer if not already the right size.
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to first check and rewind
 * 
 * @param size
 * 
 *            number of ints that need to be held by the newly created
 * 
 *            buffer
 * 
 * @return the requested new IntBuffer
 */

public static IntBuffer createIntBuffer(IntBuffer buf, final int size) {

	if (buf != null &amp;&amp; buf.limit() == size) {

		buf.rewind();

		return buf;

	}

	buf = createIntBuffer(size);

	return buf;

}

/**
 * 
 * Creates a new IntBuffer with the same contents as the given IntBuffer.
 * 
 * The new IntBuffer is seperate from the old one and changes are not
 * 
 * reflected across. If you want to reflect changes, consider using
 * 
 * Buffer.duplicate().
 * 
 * 
 * 
 * @param buf
 * 
 *            the IntBuffer to copy
 * 
 * @return the copy
 */

public static IntBuffer clone(final IntBuffer buf) {

	if (buf == null) {

		return null;

	}

	buf.rewind();

	IntBuffer copy;

	if (buf.isDirect()) {

		copy = createIntBuffer(buf.limit());

	} else {

		copy = IntBuffer.allocate(buf.limit());

	}

	copy.put(buf);

	return copy;

}

// // -- GENERAL BYTE ROUTINES -- ////

/**
 * 
 * Create a new ByteBuffer of the specified size.
 * 
 * 
 * 
 * @param size
 * 
 *            required number of ints to store.
 * 
 * @return the new IntBuffer
 */

public static ByteBuffer createByteBuffer(final int size) {

	final ByteBuffer buf = malloc(size).order(ByteOrder.nativeOrder());

	buf.clear();
	return buf;

}

/**
 * 
 * Create a new ByteBuffer of an appropriate size to hold the specified
 * 
 * number of ints only if the given buffer if not already the right size.
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to first check and rewind
 * 
 * @param size
 * 
 *            number of bytes that need to be held by the newly created
 * 
 *            buffer
 * 
 * @return the requested new IntBuffer
 */

public static ByteBuffer createByteBuffer(ByteBuffer buf, final int size) {

	if (buf != null &amp;&amp; buf.limit() == size) {

		buf.rewind();

		return buf;

	}

	buf = createByteBuffer(size);

	return buf;

}

public static ByteBuffer createByteBuffer(final byte... data) {

	final ByteBuffer bb = createByteBuffer(data.length);

	bb.put(data);

	bb.flip();

	return bb;

}

public static ByteBuffer createByteBuffer(final String data) {

	final byte[] bytes = data.getBytes();

	final ByteBuffer bb = createByteBuffer(bytes.length);

	bb.put(bytes);

	bb.flip();

	return bb;

}

/**
 * 
 * Creates a new ByteBuffer with the same contents as the given ByteBuffer.
 * 
 * The new ByteBuffer is seperate from the old one and changes are not
 * 
 * reflected across. If you want to reflect changes, consider using
 * 
 * Buffer.duplicate().
 * 
 * 
 * 
 * @param buf
 * 
 *            the ByteBuffer to copy
 * 
 * @return the copy
 */

public static ByteBuffer clone(final ByteBuffer buf) {

	if (buf == null) {

		return null;

	}

	buf.rewind();

	ByteBuffer copy;

	if (buf.isDirect()) {

		copy = createByteBuffer(buf.limit());

	} else {

		copy = ByteBuffer.allocate(buf.limit());

	}

	copy.put(buf);

	return copy;

}

// // -- GENERAL SHORT ROUTINES -- ////

/**
 * 
 * Create a new ShortBuffer of the specified size.
 * 
 * 
 * 
 * @param size
 * 
 *            required number of shorts to store.
 * 
 * @return the new ShortBuffer
 */

public static ShortBuffer createShortBuffer(final int size) {
	final ShortBuffer buf = malloc(2 * size).order(ByteOrder.nativeOrder()).asShortBuffer();
	buf.clear();
	return buf;

}

/**
 * 
 * Create a new ShortBuffer of an appropriate size to hold the specified
 * 
 * number of shorts only if the given buffer if not already the right size.
 * 
 * 
 * 
 * @param buf
 * 
 *            the buffer to first check and rewind
 * 
 * @param size
 * 
 *            number of shorts that need to be held by the newly created
 * 
 *            buffer
 * 
 * @return the requested new ShortBuffer
 */

public static ShortBuffer createShortBuffer(ShortBuffer buf, final int size) {

	if (buf != null &amp;&amp; buf.limit() == size) {

		buf.rewind();

		return buf;

	}

	buf = createShortBuffer(size);

	return buf;

}

public static ShortBuffer createShortBuffer(final short... data) {

	if (data == null) {

		return null;

	}

	final ShortBuffer buff = createShortBuffer(data.length);

	buff.clear();

	buff.put(data);

	buff.flip();

	return buff;

}

/**
 * 
 * Creates a new ShortBuffer with the same contents as the given
 * 
 * ShortBuffer. The new ShortBuffer is seperate from the old one and changes
 * 
 * are not reflected across. If you want to reflect changes, consider using
 * 
 * Buffer.duplicate().
 * 
 * 
 * 
 * @param buf
 * 
 *            the ShortBuffer to copy
 * 
 * @return the copy
 */

public static ShortBuffer clone(final ShortBuffer buf) {

	if (buf == null) {

		return null;

	}

	buf.rewind();

	ShortBuffer copy;

	if (buf.isDirect()) {

		copy = createShortBuffer(buf.limit());

	} else {

		copy = ShortBuffer.allocate(buf.limit());

	}

	copy.put(buf);

	return copy;

}

/**
 * 
 * Ensures there is at least the &lt;code&gt;required&lt;/code&gt; number of entries
 * 
 * left after the current position of the buffer. If the buffer is too small
 * 
 * a larger one is created and the old one copied to the new buffer.
 * 
 * 
 * 
 * @param buffer
 * 
 *            buffer that should be checked/copied (may be null)
 * 
 * @param required
 * 
 *            minimum number of elements that should be remaining in the
 * 
 *            returned buffer
 * 
 * @return a buffer large enough to receive at least the
 * 
 *         &lt;code&gt;required&lt;/code&gt; number of entries, same position as the
 * 
 *         input buffer, not null
 */

public static FloatBuffer ensureLargeEnough(FloatBuffer buffer, final int required) {

	if (buffer == null || (buffer.remaining() &lt; required)) {

		final int position = (buffer != null ? buffer.position() : 0);

		final FloatBuffer newVerts = createFloatBuffer(position + required);

		if (buffer != null) {

			buffer.rewind();

			newVerts.put(buffer);

			newVerts.position(position);

		}

		buffer = newVerts;

	}

	return buffer;

}

public static ShortBuffer ensureLargeEnough(ShortBuffer buffer, final int required) {

	if (buffer == null || (buffer.remaining() &lt; required)) {

		final int position = (buffer != null ? buffer.position() : 0);

		final ShortBuffer newVerts = createShortBuffer(position + required);

		if (buffer != null) {

			buffer.rewind();

			newVerts.put(buffer);

			newVerts.position(position);

		}

		buffer = newVerts;

	}

	return buffer;

}

public static ByteBuffer ensureLargeEnough(ByteBuffer buffer, final int required) {

	if (buffer == null || (buffer.remaining() &lt; required)) {

		final int position = (buffer != null ? buffer.position() : 0);

		final ByteBuffer newVerts = createByteBuffer(position + required);

		if (buffer != null) {

			buffer.rewind();

			newVerts.put(buffer);

			newVerts.position(position);

		}

		buffer = newVerts;

	}

	return buffer;

}

public static void printCurrentDirectMemory(StringBuilder store) {
	synchronized (referencetrash) {
		try {
			// make a new set to hold the keys to prevent concurrency
			// issues.

			final int totalHeld = bytesAllocated.get();
			final long heapMem = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();

			final boolean printStout = store == null;

			if (store == null) {

				store = new StringBuilder();

			}

			store.append(&quot;Total   heap memory held: &quot;).append(heapMem / 1024).append(&quot;kbn&quot;);

			store.append(&quot;Total direct memory held: &quot;).append(totalHeld / 1024 / 1024f).append(&quot;/&quot;).append(backingBuffer.capacity() / 1024 / 1024f).append(&quot;mbn&quot;);

			if (detailPrint) {
				final HashMap&lt;String, Integer&gt; bufferdata = new HashMap&lt;String, Integer&gt;();

				for (final BufferReference b : referencetrash) {

					Integer size = bufferdata.get(b.getType());

					if (size == null) {

						size = 0;

					}

					size += b.getSize();

					bufferdata.put(b.getType(), size);

				}

				for (final Entry&lt;String, Integer&gt; typedata : bufferdata.entrySet()) {

					store.append(typedata.getKey()).append(&quot; &quot;).append(typedata.getValue() / 1024 / 1024f).append(&quot;mbn&quot;);

				}
			}
			if (printStout) {

				System.out.println(store.toString());

			}

		} catch (final Exception e) {

			e.printStackTrace();

		}
	}
}

/**
 * 
 * DirectByteBuffers are garbage collected by using a phantom reference and
 * 
 * a reference queue. Every once a while, the JVM checks the reference queue
 * 
 * and cleans the DirectByteBuffers. However, as this doesn't happen
 * 
 * immediately after discarding all references to a DirectByteBuffer, it's
 * 
 * easy to OutOfMemoryError yourself using DirectByteBuffers. This function
 * 
 * explicitly calls the Cleaner method of a DirectByteBuffer.
 * 
 * 
 * 
 * @param toBeDestroyed
 * 
 *            The DirectByteBuffer that will be &quot;cleaned&quot;. Utilizes
 * 
 *            reflection.
 * 
 * 
 */

public static void destroyByteBuffer(final ByteBuffer toBeDestroyed) {

	try {

		final Method cleanerMethod = toBeDestroyed.getClass().getMethod(&quot;cleaner&quot;);

		cleanerMethod.setAccessible(true);

		final Object cleaner = cleanerMethod.invoke(toBeDestroyed);

		final Method cleanMethod = cleaner.getClass().getMethod(&quot;clean&quot;);

		cleanMethod.setAccessible(true);

		cleanMethod.invoke(cleaner);

	} catch (final IllegalAccessException ex) {

		Logger.getLogger(BufferUtils.class.getName()).log(Level.SEVERE, &quot;{0}&quot;, ex);

	} catch (final IllegalArgumentException ex) {

		Logger.getLogger(BufferUtils.class.getName()).log(Level.SEVERE, &quot;{0}&quot;, ex);

	} catch (final InvocationTargetException ex) {

		Logger.getLogger(BufferUtils.class.getName()).log(Level.SEVERE, &quot;{0}&quot;, ex);

	} catch (final NoSuchMethodException ex) {

		Logger.getLogger(BufferUtils.class.getName()).log(Level.SEVERE, &quot;{0}&quot;, ex);

	} catch (final SecurityException ex) {

		Logger.getLogger(BufferUtils.class.getName()).log(Level.SEVERE, &quot;{0}&quot;, ex);

	}

}

}

class BufferReference extends PhantomReference<Buffer> {

private final int size;

private final String type;

final int start;

final int end;

public final StackTraceElement[] stacktrace;

public BufferReference(final Buffer referent, final int i, final int j) {

	super(referent, BufferUtils.collectedBuffers);

	this.start = i;

	this.end = j;

	this.size = referent.capacity();

	this.type = referent.getClass().getSimpleName();
	this.stacktrace = Thread.currentThread().getStackTrace();
}

public int getSize() {

	return size;

}

public String getType() {

	return type;

}

}

class CleanupThread extends Thread {
@Override
public void run() {
this.setName("DirectMemory CleanupThread");
try {
while (true) {
final BufferReference top = (BufferReference) BufferUtils.collectedBuffers.remove();
synchronized (BufferUtils.referencetrash) {
BufferUtils.referencetrash.remove(top);
}
BufferUtils.bytesAllocated.addAndGet(-top.getSize());
}
} catch (final Exception e) {
e.printStackTrace();
System.exit(2);
}
}
}

1 Like

Can you explain what you changed?

Looks like he allocates one huge buffer and then just returns slices of it… so at least all of the core allocation routines have changed. In theory, fragmentation is the only danger. I assume the JNI layer is smart enough to efficiently pass direct slices around as easily as plain direct buffers. (Though I haven’t checked… I’m sure Empire would have noticed performance drop significantly if that was a worry.)



I’m sure Empire has tested it but in my experience, PhantomReferences can be as problematic as waiting for GC since an object has to be finalized, etc. before it is phantom reachable.



Soft or weak references might be better but sometimes there is a delay in reference queue processing that I’m not sure where comes from. I’ve always assumed it was part of the GC processing but maybe not.

Well yes, the main difference is, that the actuall Buffers used are just slices, so the allocation costs are near zero (even with that kinda inefficient loop I use currently, a binary tree would be more suited for this)

At least no feelable performance difference. The fragmentation problem also is with the default system, since the jvm gurantees that directbuffer memory adresses do not change.



The main advantage is the ability to deallocate buffers, you only need to remove the buffer from the referencetrash list, and it is instatnly aviable for new useage, (since it does not matter if the old directbuffer that is sliced from it is already collected or not)

So you get a garbagecollection around as good as the current one, and the ability to unload buffers with the gurantee that the space is imidialty free, and this without the use of some hacky undokumented methods.

Since you usually know what parts of the programm churn memory trough it is now possible to return the memory asap.



Also if the initialize methos return sucessfully, you now have a gurantee how much direct memory you are allowed to use, while else you could crash anytime when trying to allocate a new buffer.

(possible useage, load level, then spawn grassbatches as long as more than 100mb directmemory is still free)





(Actually I have the feeling that the gc is far faster with just collecting the sliced buffers, probably because in jdk7 the g1 balaces between amount of work for collecting vs amount of heapspace freed, and the sliced buffer has no native deallocation costs while the normal has a quite high cost to deallocate the native part.)