Warning for shader makers using #import with other glsllib

As my internal GLSL noise library is getting bigger I decided to start segregating general functions used by the different noise shaders into a NoiseUtils.glsllib file. That works fine but there’s something I didn’t know (and I guess most people don’t) is that the import order is important.



One would imagine that a glsllib containing utility functions with no dependence should be imported first, but it’s the contrary. Import that file last.



In short, if you have dependencies on a file, the one that many has no dependent has to be last.



For example this would be the way to import the utility lib:


#import "Shaders/Noise2D.glsllib"
#import "Shaders/NoiseUtils.glsllib"


and not


#import "Shaders/NoiseUtils.glsllib"
#import "Shaders/Noise2D.glsllib"


That last one won't work; you'll get an error.

I'm wondering if that's the intended usage because when you think about it, it's contrary to pretty much everything I've seen up until now. Not that it matters though (once you've found the "problem").

Anyway, just wanted to share that. ;)

IMPORTANT
It's been found--in later posts in this thread--that the preprocessor importing method uses a hashmap for those included files. Because of that, the order in which the files are thrown at the GPU might be in the wrong order.

Since the ordering depends on the hashmap's mood (yes, I am joking) it's possible that even with the above you might get an error, but my experiments showed me the above was the safest.

Anyway, until this is fixed, that's the best way to go at it.

Thank you for your time, you can now go back to your scheduled programming. ;)
7 Likes

cool thanks :slight_smile: but i got 1 question:



hows that novel you been working on?

Which one? I’ve got 1 in French, 1 in English, countless ideas and some story ideas jotted down (ok, more than that on the website) for Disenthral. So… which one? :wink:

http://www.youtube.com/watch?v=w9rv1oJ4Res

3 Likes
@madjack said:
As my internal GLSL noise library is getting bigger I decided to start segregating general functions used by the different noise shaders into a NoiseUtils.glsllib file. That works fine but there's something I didn't know (and I guess most people don't) is that the import order is important.

One would imagine that a glsllib containing utility functions with no dependence should be imported first, but it's the contrary. Import that file last.

In short, if you have dependencies on a file, the one that many has no dependent has to be last.

For example this would be the way to import the utility lib:


and not



That last one won't work; you'll get an error.

I'm wondering if that's the intended usage because when you think about it, it's contrary to pretty much everything I've seen up until now. Not that it matters though (once you've found the "problem").

Anyway, just wanted to share that. ;)


What error did you get? It shouldn't matter which order unless one library depends on the other being included.

#include should be _literally_ like cutting and pasting the contents of the file in right at that location.

GLSL is like C you can not refer to a function that is declared later in the file.

But like in C you can go

[java]

//declaration

returntype myfunc;





do stuff using myfunc





//implementation

returntype myfunc(){



myfunc content;



}

[/java]

You could declare your utility function in the first glslib.

Also why not merge the 2 libs? too big? In the end all is pasted into the shader as Paul said.

It seems there’s some weird shit going on.



Whereas yesterday I was having this behavior all the time now I only get it intermittently. But, there’s one material in which it always happen.



The error message is this:



[java]

Jul 18, 2012 12:32:33 PM com.jme3.app.Application handleError

SEVERE: Uncaught exception thrown in Thread[LWJGL Renderer Thread,5,main]

com.jme3.renderer.RendererException: compile error in:ShaderSource[name=CustomShader/Stars/sun.frag, defines, type=Fragment, language=GLSL130] error:0(57) : error C1008: undefined variable “mod289”

0(58) : error C1008: undefined variable “permute”

0(58) : error C1008: undefined variable “permute”

0(58) : error C1008: undefined variable “permute”

0(58) : error C1008: undefined variable “permute”

0(63) : error C1008: undefined variable “permute”

0(63) : error C1008: undefined variable “permute”

0(63) : error C1008: undefined variable “permute”

0(63) : error C1008: undefined variable “permute”

0(69) : error C1008: undefined variable “grad4”

0(70) : error C1008: undefined variable “grad4”

0(71) : error C1008: undefined variable “grad4”

0(72) : error C1008: undefined variable “grad4”

0(73) : error C1008: undefined variable “grad4”

0(76) : error C1008: undefined variable “taylorInvSqrt”

0(81) : error C1008: undefined variable “taylorInvSqrt”

[/java]



If I move the imports around that error disappears completely and everything renders fine.



As to why I split things up, it’s easier for me to work with and not doing this would mean I’d have to make a procedural lighting shader for each type of noise (2D/3D/4D).

If you are getting that error then some part of the shader is referring to a variable before it is defined. If you take your outer shader and cut and paste the contents of the #include lines exactly in place of the #include lines then your error should be clearly apparent.



I’d have to have access to the shaders to comment further but there really shouldn’t be any magic here.

Noise4D.glsllib contains 1 method and that’s it. (As do Noise3D and Noise2D)



So unless I misunderstand something, that shouldn’t make things crap out.



For the NoiseUtils.glsllib, it contains methods like this:


vec2 mod289(vec2 x) {
return x - floor(x * (1.0 / 289.0)) * 289.0;
}


except there are several variations of the same method, vec2, vec3, vec4... Nothing complex here.

What does the beginning of CustomShader/Stars/sun.frag look like? How many lines is Noise4D.glsllib.



Or if you want to make it easier without posting all of the source…

open sun.frag

open the file referenced in the first #include line

select all, copy

go back and paste that in where the #include line is.

repeat for every #include.

post that source here… at least up to line 57



BTW, if you get inconsistent results with something like this then make sure to do a clean build every time just to make sure the assets get rebuilt.

…and actually, I guess the full log would have included the source already broken out this way… so you could just paste that here.

Weird…

For your viewing pleasure here’s the whole dump when an error pops up.



[java]

Jul 18, 2012 2:42:20 PM com.jme3.renderer.lwjgl.LwjglRenderer updateShaderSourceData

WARNING: Bad compile of:

1 #version 130

//


Noise4D.glsllib starts here
2 //
3 // Description : Array and textureless GLSL 2D/3D/4D simplex
4 // noise functions.
5 // Author : Ian McEwan, Ashima Arts.
6 // Maintainer : ijm
7 // Lastmod : 20110822 (ijm)
8 // License : Copyright (C) 2011 Ashima Arts. All rights reserved.
9 // Distributed under the MIT License. See LICENSE file.
10 // https://github.com/ashima/webgl-noise
11 //
12
13 float noise4d(vec4 v) {
14 const vec4 C = vec4( 0.138196601125011, // (5 - sqrt(5))/20 G4
15 0.276393202250021, // 2 * G4
16 0.414589803375032, // 3 * G4
17 -0.447213595499958); // -1 + 4 * G4
18
19 // (sqrt(5) - 1)/4 = F4, used once below
20 #define F4 0.309016994374947451
21
22 // First corner
23 vec4 i = floor(v + dot(v, vec4(F4)) );
24 vec4 x0 = v - i + dot(i, C.xxxx);
25
26 // Other corners
27
28 // Rank sorting originally contributed by Bill Licea-Kane, AMD (formerly ATI)
29 vec4 i0;
30 vec3 isX = step( x0.yzw, x0.xxx );
31 vec3 isYZ = step( x0.zww, x0.yyz );
32 // i0.x = dot( isX, vec3( 1.0 ) );
33 i0.x = isX.x + isX.y + isX.z;
34 i0.yzw = 1.0 - isX;
35 // i0.y += dot( isYZ.xy, vec2( 1.0 ) );
36 i0.y += isYZ.x + isYZ.y;
37 i0.zw += 1.0 - isYZ.xy;
38 i0.z += isYZ.z;
39 i0.w += 1.0 - isYZ.z;
40
41 // i0 now contains the unique values 0,1,2,3 in each channel
42 vec4 i3 = clamp( i0, 0.0, 1.0 );
43 vec4 i2 = clamp( i0-1.0, 0.0, 1.0 );
44 vec4 i1 = clamp( i0-2.0, 0.0, 1.0 );
45
46 // x0 = x0 - 0.0 + 0.0 * C.xxxx
47 // x1 = x0 - i1 + 1.0 * C.xxxx
48 // x2 = x0 - i2 + 2.0 * C.xxxx
49 // x3 = x0 - i3 + 3.0 * C.xxxx
50 // x4 = x0 - 1.0 + 4.0 * C.xxxx
51 vec4 x1 = x0 - i1 + C.xxxx;
52 vec4 x2 = x0 - i2 + C.yyyy;
53 vec4 x3 = x0 - i3 + C.zzzz;
54 vec4 x4 = x0 + C.wwww;
55
56 // Permutations
57 i = mod289(i);
58 float j0 = permute( permute( permute( permute(i.w) + i.z) + i.y) + i.x);
59 vec4 j1 = permute( permute( permute( permute (
60 i.w + vec4(i1.w, i2.w, i3.w, 1.0 ))
61 + i.z + vec4(i1.z, i2.z, i3.z, 1.0 ))
62 + i.y + vec4(i1.y, i2.y, i3.y, 1.0 ))
63 + i.x + vec4(i1.x, i2.x, i3.x, 1.0 ));
64
65 // Gradients: 7x7x6 points over a cube, mapped onto a 4-cross polytope
66 // 7*7*6 = 294, which is close to the ring size 17*17 = 289.
67 vec4 ip = vec4(1.0/294.0, 1.0/49.0, 1.0/7.0, 0.0) ;
68
69 vec4 p0 = grad4(j0, ip);
70 vec4 p1 = grad4(j1.x, ip);
71 vec4 p2 = grad4(j1.y, ip);
72 vec4 p3 = grad4(j1.z, ip);
73 vec4 p4 = grad4(j1.w, ip);
74
75 // Normalise gradients
76 vec4 norm = taylorInvSqrt(vec4(dot(p0,p0), dot(p1,p1), dot(p2, p2), dot(p3,p3)));
77 p0 *= norm.x;
78 p1 *= norm.y;
79 p2 *= norm.z;
80 p3 *= norm.w;
81 p4 *= taylorInvSqrt(dot(p4,p4));
82
83 // Mix contributions from the five corners
84 vec3 m0 = max(0.6 - vec3(dot(x0,x0), dot(x1,x1), dot(x2,x2)), 0.0);
85 vec2 m1 = max(0.6 - vec2(dot(x3,x3), dot(x4,x4) ), 0.0);
86 m0 = m0 * m0;
87 m1 = m1 * m1;
88 return 49.0 * ( dot(m0*m0, vec3( dot( p0, x0 ), dot( p1, x1 ), dot( p2, x2 )))
89 + dot(m1*m1, vec2( dot( p3, x3 ), dot( p4, x4 ) ) ) ) ;
90
91 }
92
93 //
NoiseUtils.glsllib starts here
94 // Noise Functions used in Noise2D, Noise3D and Noise4D glsllib
95
96 vec2 mod289(vec2 x) {
97 return x - floor(x * (1.0 / 289.0)) * 289.0;
98 }
99
100 vec3 mod289(vec3 x) {
101 return x - floor(x * (1.0 / 289.0)) * 289.0;
102 }
103
104 vec4 mod289(vec4 x) {
105 return x - floor(x * (1.0 / 289.0)) * 289.0;
106 }
107
108 float mod289(float x) {
109 return x - floor(x * (1.0 / 289.0)) * 289.0;
110 }
111
112 vec3 permute(vec3 x) {
113 return mod289(((x*34.0)+1.0)*x);
114 }
115
116 vec4 permute(vec4 x) {
117 return mod289(((x*34.0)+1.0)*x);
118 }
119
120 float permute(float x) {
121 return mod289(((x*34.0)+1.0)*x);
122 }
123
124 vec4 taylorInvSqrt(vec4 r) {
125 return 1.79284291400159 - 0.85373472095314 * r;
126 }
127
128 float taylorInvSqrt(float r) {
129 return 1.79284291400159 - 0.85373472095314 * r;
130 }
131
132 vec4 grad4(float j, vec4 ip) {
133 const vec4 ones = vec4(1.0, 1.0, 1.0, -1.0);
134 vec4 p,s;
135
136 p.xyz = floor( fract (vec3(j) * ip.xyz) * 7.0) * ip.z - 1.0;
137 p.w = 1.5 - dot(abs(p.xyz), ones.xyz);
138 s = vec4(lessThan(p, vec4(0.0)));
139 p.xyz = p.xyz + (s.xyz*2.0 - 1.0) * s.www;
140
141 return p;
142 }
143
144
//
sun.frag starts here
145 uniform vec4 m_Color;
146 uniform float g_Time;
147
148 in vec4 texCoord4D;
149
150 out vec4 outTex;
151
152 void main( void ) {
153 // Make texel turbulence.
154 float n = abs(noise4d(vec4(texCoord4D.xyz, g_Time * 0.15)) - 0.5) +
155 abs(noise4d(vec4(texCoord4D.xyz * 2.0, g_Time * 0.15)) - 0.25) +
156 abs(noise4d(vec4(texCoord4D.xyz * 4.0, g_Time * 0.15)) - 0.125) +
157 abs(noise4d(vec4(texCoord4D.xyz * 8.0, g_Time * 0.15)) - 0.0625) +
158 abs(noise4d(vec4(texCoord4D.xyz * 16.0, g_Time * 0.15)) - 0.03125);
159
160 // colorize to sun's color then return.
161 outTex = vec4(n * m_Color.rgb * vec3(0.50), m_Color.a);
162 }

Jul 18, 2012 2:42:20 PM com.jme3.app.Application handleError
SEVERE: Uncaught exception thrown in Thread[LWJGL Renderer Thread,5,main]
com.jme3.renderer.RendererException: compile error in:ShaderSource[name=CustomShader/Stars/sun.frag, defines, type=Fragment, language=GLSL130] error:0(57) : error C1008: undefined variable "mod289"
0(58) : error C1008: undefined variable "permute"
0(58) : error C1008: undefined variable "permute"
0(58) : error C1008: undefined variable "permute"
0(58) : error C1008: undefined variable "permute"
0(63) : error C1008: undefined variable "permute"
0(63) : error C1008: undefined variable "permute"
0(63) : error C1008: undefined variable "permute"
0(63) : error C1008: undefined variable "permute"
0(69) : error C1008: undefined variable "grad4"
0(70) : error C1008: undefined variable "grad4"
0(71) : error C1008: undefined variable "grad4"
0(72) : error C1008: undefined variable "grad4"
0(73) : error C1008: undefined variable "grad4"
0(76) : error C1008: undefined variable "taylorInvSqrt"
0(81) : error C1008: undefined variable "taylorInvSqrt"
[/java]

Relevant line numbers are: 1, 57 (where error occurs), 93, 145 are commented with ownership.

The sun.frag is simple.

[java]
#import "Shaders/NoiseUtils.glsllib"
#import "Shaders/Noise4D.glsllib"

uniform vec4 m_Color;
uniform float g_Time;

in vec4 texCoord4D;

out vec4 outTex;

void main( void ) {
// [noise generation goes here]
}
[/java]
1 Like

I do think I’ve rebuilt but I’ll do it again just to make sure.

Line 57 calls: mod289 before it is declared.



Are you sure that sun.frag is the same one that errors… because the error log implies that the include lines are the other way around.



Did you do a clean build before running?

@pspeed said:
Line 57 calls: mod289 before it is declared.

That's what I've been saying.


Are you sure that sun.frag is the same one that errors... because the error log implies that the include lines are the other way around.

Positive.


Did you do a clean build before running?

It just finished. I'll reply in a minute with the result.

Exact same result.

I looked in the source. This is a bug in JME.



It doesn’t handle these preprocessor directives like every other preprocessor on the planet and instead shoves them all into a hashset to spit out after the fact. So they potentially get reordered.

3 Likes

Feels good to know I’m not crazy. I’m just mad. :wink:

Also looking at the source, I think you can get around this by making sure to import your dependencies everywhere that is needed.



JME #import doesn’t work like #include by design (I still think it’s broken for other reasons) in that it will only import the file once… I don’t know for sure if that extends to the outer file as well but it’s worth a try. The part I think is broken is that ordering is messed up and the file is included at the beginning of the file even if you import later… but I will defer on everything but the ordering.



Now I want to add proper #include support. :wink: