[Suggestion] Shader debuging

When a shader compile/link failure occurs the source will be output on STDERR.



Often the exception will mention on which line there was an error, problem is this isn’t very usable when using the #import statement… since the line number in my editor and and the exception won’t match.



Suggestions:

  1. let GLSLLoader.java inline comments like this in shader source:
Code:
//BEGIN-------------------- Common/ShaderLib/Tangent.glsllib -------------------- some library shader code...

//END-------------------- Common/ShaderLib/Tangent.glsllib --------------------

Here is the rest of my fancy shader code


2. insert line numbers before output of shader source on exception

I can provide the code for it if wanted :)
1 Like

We already dump the code out to the log so you can copy n paste it into notepad to view where the error happened

@Momoko_Fan said:
We already dump the code out to the log so you can copy n paste it into notepad to view where the error happened


I think what he's saying is that the code we dump is not expanded... ie: it still has the #includes in it, etc... where as the error line numbers are for the fully expanded, #ifdef'd stuff removed, version.

...but I don't think we have access to that source without generating it ourselves. Or are we doing the preproc ourselves?
@pspeed said:
I think what he's saying is that the code we dump is not expanded... ie: it still has the #includes in it, etc... where as the error line numbers are for the fully expanded, #ifdef'd stuff removed, version.

...but I don't think we have access to that source without generating it ourselves. Or are we doing the preproc ourselves?

We are obviously not doing the preprocessing ourselves, its done by the driver. The exception to this is the "#include" which is handled in GLSLLoader.
The dumped source code is the exact source code that was given to OpenGL, so the line numbers should match.

@pspeed, @Momoko_Fan I will give you an example:



A. Library.glsllib has 43 rows

B. MyFancyShader.frag is #import(ing) Library.glsllib



Then lets say I have made a misstake in MyFancyShader.frag on line 14

Error from jME says, “you have an error on line 57”



line 57 means nothing to me since the error I’ve made is on line 14.



If I then copy all the error to notepad like @Momoko_Fan I can find the line where the error occured but now I don’t know from which file it origin from…

He’s got a point IMO.

And common Kirill copying the output in notepad is that your solution?

Shader are already a pain to debug…could be kinda easy to output the line number with the code when there is an error…

No… pointing to the line number implies parsing the output from the driver.



But… providing line numbers in the output would be super-awesome-cool. I don’t know what’s required to add a “being this file” style comment but I know exactly where to add line numbers to the source dump.



If no one gets to it first, I will do it. (Man, I’m imagining all the time this will save already. :))

1 Like

Yeah, “tracing back” the line number in the actual file would probably be too much overhead, but I give +1 for the line numbers in output :slight_smile:

@nehon said:
Shader are already a pain to debug...

So true, sigh. ^^

@pspeed said:
No... pointing to the line number implies parsing the output from the driver.


Nope, don't think so (If I'm not missin' something). line 1000-1005 in LwjglRenderer is outputting the error and it just concats the shader source to the output.. just split the string by newline and then rejoin it with a loop..


Adding the filename-comments should be done in GLSLLoader#loadNode .. will be like 2-3 rows of code :)

@pspeed said:
If no one gets to it first, I will do it. (Man, I'm imagining all the time this will save already. :))


Already waiting :P
@kwando said:
Nope, don't think so (If I'm not missin' something). line 1000-1005 in LwjglRenderer is outputting the error and it just concats the shader source to the output.. just split the string by newline and then rejoin it with a loop..


I think you misunderstood. I can put a hundred million line numbers in the source... but knowing WHICH line the bug happened on requires parsing the string that the driver returns. There's not a nice: int getErrorLine() as far as I know.

...and given how much the quality of error messages varies from vendor to vendor, I wouldn't want to try to universally guess what the line number is from a string.

@normen, @pspeed



I think you are one step ahead of me :slight_smile: I will try to explain what I mean again:



fancy.glsllib

Code:
float magic_float(){ return 42; }

ubershader.frag
Code:
#import "fancy.glsllib"

void main(){
gl_FragCoord = magic_number();
}



The string sent to the driver would be:
Code:
float magic_float(){ return 42; }

void main(){
gl_FragCoord = magic_number();
}


This will of course complain on line 6 since magic_number returns() a float and gl_FragCoord is vec4, and remember no implicit conversion in GLSL version blahblah..

Current error message will be like:
Code:
float magic_float(){ return 42; }

void main(){
gl_FragCoord = magic_number();
}

I throw an exception on you because you messed up something on line 6 + stacktrace


If one send this to the driver instead:
Code:
//BEGIN-----------------------fancy.glsllib------------------------------ float magic_float(){ return 42; }

//BEGIN-----------------------ubershader.frag------------------------------

void main(){
gl_FragCoord = magic_number();
}


The driver now fails compile on line 9 instead. Then we could print something like this:
Code:
1: /BEGIN-----------------------fancy.glsllib------------------------------ 2: float magic_float(){ 3: return 42; 4: } 5: 6: //BEGIN-----------------------ubershader.frag------------------------------ 7: 8: void main(){ 9: gl_FragCoord = magic_number(); 10: }

I throw an exception on you because you messed up something on line 9 + stacktrace


This would make it so much easier to find the error :)


Ofcourse it would be so much nicer if it told me in which file and exact line number, but I don't think thats really viable.

As said, its hard to trace back that line of a single file, the final shader is compiled based on many things, saying which line is from which file is not so easily possible w/o asking OpenGL how it compiled it. If its juts about the line that is being reported, it fits to the shader as it is being printed in the log.

Your last listing was precisely what I proposed doing. Adding line numbers. Though I will not much with the // begin lines because I’m unfamiliar with that code. I know the code that outputs the shader source on error because I added that. :slight_smile: Besides, tracing back to the original file is not so tough if you already know what the line looks like.



What I said was not feasible was trying to guess what line the error was on from whatever garbage the driver decides to spit out. After all, one of ATI’s favorite message is the equivalent of “it’s broken”… with 0 additional information. :wink:

@normen, I realize it’s not really possible to find the correct line in the correct file. But if it’s done like I explained it above one can read the error number manually, then find the corresponding row in the log and just look at the comment above it and then you know which file the error was in (not the correct row).



The correct row could be found like this (not automatically): (I’m not sure I like this, but it could be done like this)



split the shader source by the //BEGIN marker, add a new line counter for every new //BEGIN - block

Code:
1: //BEGIN-----------------------fancy.glsllib------------------------------ 2: float magic_float(){ 3: return 42; 4: } 5: 6: //BEGIN-----------------------ubershader.frag------------------------------ 7: 1: 8: 2: void main(){ 9: 3: gl_FragCoord = magic_number(); 10: 4: }

I throw an exception on you because you messed up something on line 9 + stacktrace


Now I can find line 9 and I immediately see that line 9 in "preprocessed shader" corrseponds to line 3 in ubershader.frag

But like I said, not sure if I like this :P

Just linenumbers would be a huge help :)

Since theres plans for combining shader code in even other ways I don’t know how feasible it is to add these “name marker lines” now but the line numbers can surely be added.

@normen I see, thanks for your time anyway :slight_smile:



Is there a good way to find out whats currently being developed? How can one find out things like “theres plans for combining shader code in even other ways” for instance?

Remote viewing, divination, sacrifices to the gods, stuff like that :wink:

@normen ah, I was hoping more for lika a mailing list, hidden IRC or something along that :stuck_out_tongue: seems like I’ts time for to learn some new skills, besides bugging you guys with silly questions and such.



BTW is it much job to wrap up some code and send it to the user contributions repo? Seems like you’re the guy to ask :slight_smile:



Sorry for thread hijack myself :stuck_out_tongue:

Uh, its not much work, basically just a commit :slight_smile: If you want to contribute you can do that yourself, I can give you commit access. The libraries have to be wrapped as a plugin for the SDK though, but wrapping a jar library is easy and explained in the wiki. What would that code be?

Edit: for the other question, this site here is the best chance to follow jme most closely really (as you have found out though this “interwoven” information ^^).

Edit2: and for “plans” theres ofc the googlecode tracker but some features just “drop in” through some information, contribution or plain interest by some core member :wink:

@normen My first thought was a ImgurScreenshotAppState, upload a screenshot directly to imgur and hand you the url to it :slight_smile:

wrap it in a plugin can’t be that hard, but I vaguely remember it’s messy to send HTTP requests in java… but google might now about it :stuck_out_tongue: