Before I potentially recreate the wheel, has anyone gone down the road of creating a disk cached URLLocator or HttpZipLocator? Disk cache refresh could be done based on the server’s file last modification date so that it’s self updating or even a generated MD5 hash saved on the server for more accurate comparisons. This way we don’t have to package our assets with the game.
1 Like
You mean like as a way to auto-update your game?
I already use getdown … I am trying to not have to repackage a release just because I changed an asset
I thought you only had to update the description file to note that a jar has changed.
Or are you trying to avoid even downloading a new assets jar?
…sucking assets over the web from a remote jar… you’re going to have a bad time.
I can’t use a jar for my assets because unfortunately I still have 32 bit users that are limited to about a 1.5 gig max addressable memory space and jars are memory mapped and reserve a huge chunk of addressable ram in windows. Therefor my assets are distributed as zip files through getdown. Zip or Jar is not the big issue. The real issue is that it’s a waste to send over a 40 meg file through getdown just because I modified one texture. If we had a Locator that can download the asset in a ziped format and cache it locally for future assess, and auto refresh it if it changed on the server it would be the most efficient was to deal with asset updates.
You have either just described redownloading the zip in a different way… or you are describing reading the individual assets from the zip over the web. In the latter case, you aren’t going to enjoy that very much.
Maybe you can just tightly control the order you register zips and distribute patch zips with just the latest changes… and make sure they are earliest in the asset manager.
Yup I was thinking of individual assets … but you are correct, the micro downloads will be very inefficient due to the HTTP header overhead and re-connection latency even if I try to enforce http keepalive. After incorporating your input into my thought process I agree it’s a packaging issue not a locator issue. I need to break down my assets into distinct smaller files so that a small change does not require a 40 meg zip file update.
This is why we have these forums. Sometimes posting a bad idea allows others to correct you and save you tons of wasted time 
EDIT: I will need a custom locator when I allow my users to define their own skins for ships though 
Or you send that data differently… like as game data.
Yeah, it’s always good to ask on the forum. Sometimes people don’t like the pushback when we suggest they might be heading down a slightly wrong path from the beginning. Often times it’s not even about suggesting that the path is wrong as much as throwing straw men up to help the OP specify requirements better. (Though just as often it really does turn out that the ‘why hasn’t anyone done this before?!?’ is for a reason. )
1 Like
Well, if someone would fix the limited buffer size in spidermonkey for reliable messages then I would be able to send them as components that are client cachable through the ES system
The limited buffer size is because of a bone-headed decision in the design of Serializer that it writes to Buffers. So the max size possible of buffer has to be created before an object is serialized.
The other bone-headed decision was on my part when I rewrote spider monkey and kept the original Serializer. I regret that almost every day.
I don’t remember if the RMI service will auto-stream large content but it’s also not particularly hard to write your own. So you could have the component supply a new “image ID” that you then look up through your own interfaces. (Or just use it as a URL, I guess.)
lol @pspeed I understand the root cause of the spidermonkey message size issue. I was just making a funny comment. The pattern of sending an MD5 hash of the image as a component and the client checking against it’s locally cached hash to determine a refresh is exactly what I was thinking of.
oh man… that’s a very convoluted way to say “thanks for the advice”.
1 Like
Maybe a stupid question but cant you use getdown in combination with an usual FileLocator so each j3o and Texture on its own then?
storing every item in the asset pack into a separate getdown entry is not only inefficient but a release management nightmare
I have done something like this in another way without recreate locators.
I put all my assets in a “data” folder and load them once game started. I learned it from the MMORPG “mabinogi”.
Once I modified or created new Models/Textures/Materials, I pack them into a new .pack ( .zip) file.
/**
* register add all packs to AssetManager
* @param rootPath
*/
private static void locateAsset(String rootPath) {
File root = new File(rootPath);
if (root.exists() && root.isDirectory()) {
String[] name = root.list(new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
return name.endsWith(".pack");
}
});
// TODO should do some sort before register them.
for(int i=name.length - 1; i>=0; i--) {
String pack = rootPath + name[i];
assetManager.registerLocator(pack, PackLocator.class);
}
}
}
Ignore the “PackLocator.class”, it is in fact a ZipLocator.
Here is an example.
There is a texture “human/female/texture/female_skin.dds” both in 172_to_173.pack and 181_to_182.pack. When AssetManager find it in 181_to_182.pack, it will ignore the 172_to_173.pack.
This is how the DesktopAssetManager works.
/**
* Attempts to locate the given resource name.
* @param key The full name of the resource.
* @return The AssetInfo containing resource information required for
* access, or null if not found.
*/
public AssetInfo tryLocate(AssetKey key){
if (locatorsList.isEmpty()){
logger.warning("There are no locators currently"+
" registered. Use AssetManager."+
"registerLocator() to register a"+
" locator.");
return null;
}
for (ImplThreadLocal<AssetLocator> local : locatorsList){
AssetInfo info = local.get().locate(assetManager, key);
if (info != null) {
return info;
}
}
return null;
}
In the MMORPG mabinogi, a .pack file is a encrypted zip file. Every time they update or add some new asset, client have to download a .pack file. And in the .pack file they write the version such as 172, 173, 174… Game client sort the asset with the version.
After spending a really large amount of time with this,
I went back to just having a good old folder o crap, that is transfered by a downloader file by file, allowing files to be compressed before and just checks the checksum of files. Then a good old Filelocator does the rest. As a asset.jar would require me to do extensive delta patching, or fiddle with its content directly.
1 Like
I am using getdown gradle plug in and publishing assets as separate jars by subprojecting them in gradle. and I am considering to use pack200 for compressing jars which is also supported by getdown.
I do the same but use .zip for assets so they can be uncompressed to disk when getdown downloads them. I just have to break up my assets into smaller chunks and all will be golden
In case this helps, here is my settings.gradle snippet grabbed from internet.
this will search all directories and when found a build.gradle file it will create a subproject for that. So you only need to put an empty build.gradle under your asset directories which you want be taken as subproject.
// Find the directories containing a "build.gradle" file in the root directory
// of the project. That is, every directory containing a "build.gradle" will
// be automatically the subproject of this project.
def File sourceRoot = new File("$rootDir")
FileTree tree = fileTree("$rootDir").include('**/*.gradle')
tree.visit(new FileVisitor()
{
@Override
public void visitDir(FileVisitDetails dirDetails)
{
def subDirs = dirDetails.getFile().listFiles(new FileFilter() {
public boolean accept(File file) {
if (!file.isDirectory()) {
return false
}
if (file.name == 'buildSrc') {
return false
}
return new File(file, 'build.gradle').isFile()
}
});
subDirs.each { File dir ->
String relPath = sourceRoot.toURI().relativize(dir.toURI())
String []path=relPath.split('/')
String includePath="";
path.each{
includePath+=":$it"
}
include includePath
}
}
@Override
public void visitFile(FileVisitDetails fileDetails)
{
}
});
1 Like